Unmanned aerial vehicle positioning method and system fused with relative absolute vision positioning technology

Abstract

The invention provides an unmanned aerial vehicle positioning method and system fused with a relative absolute vision positioning technology, and the method comprises the following steps: carrying out the feature extraction and matching of image frames continuously collected by a vision sensor, obtaining the corresponding relation between adjacent image frames according to the feature extraction and matching results, and combining with the motion information measured by an inertial measurement unit, thereby obtaining the relative absolute vision positioning information. Obtaining relative positioning information of the unmanned aerial vehicle; image information collected by the visual sensor in real time is matched and registered with remote sensing images in a pre-constructed image database, so that absolute position information of the unmanned aerial vehicle in a global coordinate system is obtained; fusing the relative positioning information and the absolute position information, and optimizing a global path of the unmanned aerial vehicle according to a fusion result; the method has the beneficial effects that by fusing the advantages of a relative vision positioning technology and an absolute vision positioning technology, the error accumulation of relative positioning and the matching error of absolute positioning are effectively reduced, and the positioning precision of the unmanned aerial vehicle in a complex environment is improved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) positioning technology, and in particular to a UAV positioning method and system integrating relative absolute visual positioning technology. Background Art

[0002] In drone applications, accurate positioning is crucial for ensuring safe flight and mission completion. Traditional drone positioning methods rely primarily on the Global Navigation Satellite System (GNSS). However, GNSS has significant limitations in certain complex environments. For example, in urban canyons, indoor areas, and dense forests, GNSS signals can be obstructed, causing signal strength to weaken or even be lost, making it impossible to provide accurate positioning information for drones. GNSS signals are also susceptible to electromagnetic interference and multipath effects, further reducing positioning accuracy and reliability.

[0003] Relative visual positioning technology (RVL) determines the relative position and attitude of the drone by analyzing the feature changes between the previous and next frame images of the drone. This method does not rely on external signal sources and can make up for the shortcomings of GNSS to a certain extent. However, RVL has the problem of error accumulation. As the flight time increases, the positioning error will gradually increase, affecting the accuracy of long-term positioning. Absolute visual positioning (AVL) technology obtains the absolute position of the drone by matching the images taken by the drone with a pre-stored map or image database. However, due to the differences in perspective and scale between drone and satellite images, the feature matching efficiency is low during large-scale searches, making it difficult to meet real-time positioning needs. In addition, the matching accuracy will be affected when the environment changes significantly or the image features are not obvious.

[0004] Therefore, a single positioning technology is difficult to meet the positioning needs of UAVs in complex environments. A method that integrates the advantages of multiple positioning technologies is needed to improve the accuracy, reliability and real-time performance of UAV positioning. Summary of the Invention

[0005] In view of the shortcomings of the existing technology mentioned above, the purpose of the present invention is to provide a UAV positioning method and system that integrates relative and absolute visual positioning technology to solve the problems that the existing global navigation satellite system (GNSS) is prone to signal attenuation and loss and susceptibility to interference in complex environments, and that traditional relative and absolute visual positioning technologies have error accumulation and are difficult to meet real-time positioning requirements.

[0006] To achieve the above-mentioned and other related purposes, the present invention provides the following technical solutions:

[0007] A method for positioning a drone that integrates relative absolute visual positioning technology includes the following steps: obtaining image information of the drone's surrounding environment collected in real time by a visual sensor installed on the drone and motion information of the drone measured in real time by an inertial measurement unit, and preprocessing the image information and motion information; performing feature extraction and matching on image frames continuously collected by the preprocessed visual sensor, obtaining a correspondence between adjacent image frames based on the feature extraction and matching results, fusing the correspondence with the motion information measured by the inertial measurement unit, and obtaining relative positioning information of the drone based on the fusion result; matching and registering the image information collected in real time by the preprocessed visual sensor with remote sensing images in a pre-constructed image database, and determining the absolute position information of the drone in a global coordinate system based on the matching and registration results; fusing the relative positioning information and absolute position information of the drone, and optimizing the global path of the drone based on the fusion result, thereby achieving precise positioning of the drone.

[0008] A UAV positioning system that integrates relative absolute visual positioning technology includes: a data acquisition module for acquiring image information of the UAV's surrounding environment collected in real time by a visual sensor installed on the UAV and motion information of the UAV measured in real time by an inertial measurement unit; a relative visual positioning module for performing feature extraction and matching on pre-processed image frames continuously collected by the visual sensor, obtaining a correspondence between adjacent image frames based on the feature extraction and matching results, fusing the correspondence with the motion information measured by the inertial measurement unit, and obtaining relative positioning information of the UAV based on the fusion result; an absolute visual positioning module for matching and registering the pre-processed image information collected in real time by the visual sensor with remote sensing images in a pre-built image database, determining the absolute position information of the UAV in a global coordinate system based on the matching and registration results; and a data fusion module for fusing the relative positioning information and absolute position information of the UAV, optimizing the global path of the UAV based on the fusion result, thereby achieving precise positioning of the UAV.

[0009] An electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the drone positioning method integrating relative absolute visual positioning technology as described above.

[0010] In one embodiment of the present invention, feature extraction and matching are performed on the image frames continuously collected by the preprocessed visual sensor, a correspondence between adjacent image frames is obtained based on the feature extraction and matching results, the correspondence is fused with the motion information measured by the inertial measurement unit, and relative positioning information of the drone is obtained based on the fusion result, including: extracting corner features in the image frames continuously collected by the preprocessed visual sensor and generating descriptors, using inter-frame feature tracking technology to determine the correspondence between adjacent image frames based on the corner features; using visual inertial navigation technology to fuse the correspondence between adjacent image frames with the motion information measured by the inertial measurement unit, and obtaining the relative positioning information of the drone based on the fusion result.

[0011] In one embodiment of the present invention, when using visual inertial navigation technology to fuse the correspondence between adjacent image frames and the motion information measured by the inertial measurement unit, in order to suppress error accumulation, a sliding window nonlinear optimization framework is used to tightly couple the visual reprojection error and the IMU pre-integration residual to perform nonlinear optimization, construct a state vector containing the key frame pose, the three-dimensional coordinates of the feature points and the IMU deviation, and achieve dynamic optimization of the relative pose by minimizing the joint cost function.

[0012] In one embodiment of the present invention, the image information collected in real time by the preprocessed visual sensor is matched and registered with the remote sensing images in a pre-built image database, and the absolute position information of the drone in the global coordinate system is determined based on the matching and registration results, including: constructing a neural network based on representation learning, and inputting the image information collected in real time by the preprocessed visual sensor and the remote sensing images in the pre-built image database into the neural network; performing feature extraction on the real-time collected image information and the remote sensing images in the image database in the neural network to obtain feature vectors of each image, calculating the similarity between the feature vectors, and obtaining a remote sensing image of the area where the drone is located based on the calculated similarity result; screening out a high-definition remote sensing image of the area where the drone is located from the image database, using an image registration algorithm to accurately match feature points, and estimating a homography matrix to describe the geometric transformation from the aerial image taken by the visual sensor to the high-definition remote sensing image, and obtaining the absolute position information of the drone in the global coordinate system based on the geometric transformation result.

[0013] In one embodiment of the present invention, the relative positioning information and absolute position information of the drone are fused, and the global path of the drone is optimized according to the fusion result, so as to achieve precise positioning of the drone, including: constructing a tightly coupled factor graph optimization framework according to the relative positioning information, absolute position information, visual reprojection error and IMU pre-integration residual, fusing the relative positioning information and absolute position information of the drone according to the tightly coupled factor graph optimization framework, and optimizing the global path of the drone according to the fusion result, so as to achieve precise positioning of the drone.

[0014] In one embodiment of the present invention, when the relative positioning information and the absolute position information of the drone are fused according to the tightly coupled factor graph optimization framework, the weight parameters of the relative positioning information and the absolute position information of the drone in the fusion are dynamically adjusted by minimizing the joint cost function containing relative constraints and absolute constraints.

[0015] As described above, the present invention provides a method and system for positioning a drone that integrates relative absolute visual positioning technology, which has the following beneficial effects: the present invention integrates the correspondence between adjacent image frames and the motion information measured by the inertial measurement unit through visual inertial navigation technology, thereby obtaining the relative positioning information of the drone; the present invention also obtains a remote sensing image of the area where the drone is located based on a neural network based on representation learning, and obtains the absolute position information of the drone in the global coordinate system based on the remote sensing image of the area where the drone is located based on an image registration algorithm; then the relative positioning information and the absolute position information of the drone are integrated, and the weights of the relative positioning and the absolute positioning are dynamically adjusted by minimizing the joint cost function containing relative constraints and absolute constraints, thereby achieving precise positioning of the drone; the present invention effectively reduces the error accumulation of relative positioning and the matching error of absolute positioning by integrating the advantages of relative visual positioning technology and absolute visual positioning technology, thereby improving the positioning accuracy of the drone in complex environments. In complex environments such as GNSS denial, the positioning system of the present invention can work independently without relying on external signal sources, thereby ensuring the reliability of drone positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Shown is an overall flow chart of the UAV positioning method integrating relative absolute visual positioning technology disclosed in the first embodiment of the present invention;

[0017] Figure 2 Shown is a diagram of the overall algorithm design framework of the UAV positioning method integrating relative absolute visual positioning technology disclosed in the first embodiment of the present invention;

[0018] Figure 3Shown is a workflow diagram of the absolute visual positioning module in the UAV positioning method integrating relative absolute visual positioning technology disclosed in the first embodiment of the present invention;

[0019] Figure 4 Shown is a structural diagram of the image matching network in the absolute visual positioning module of the UAV positioning method integrating the relative absolute visual positioning technology disclosed in the first embodiment of the present invention;

[0020] Figure 5 Shown is a schematic diagram of a UAV positioning system integrating relative absolute visual positioning technology disclosed in a second embodiment of the present invention;

[0021] Figure 6 Schematic diagram of an electronic device disclosed in a third embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless there is a conflict.

[0023] The first embodiment of the present invention relates to a UAV positioning method integrating relative absolute visual positioning technology, the process is as follows: Figure 1 and Figure 2 As shown, the details are as follows:

[0024] Step 101: obtain image information of the drone's surrounding environment collected in real time by a visual sensor installed on the drone and motion information of the drone measured in real time by an inertial measurement unit, and preprocess the image information and motion information.

[0025] Specifically, the visual sensors and inertial measurement units on the drone are activated to collect image information and motion information of the drone's surrounding environment in real time, and the collected image information and motion information are preprocessed, including image denoising, normalization, and motion data filtering, to improve data quality; among which, the drone's motion information includes the drone's acceleration and angular velocity.

[0026] More specifically, the drone in the present invention adopts a four-rotor drone, equipped with a gimbal-mounted camera (resolution 1280×720, frame rate 90FPS) to collect images, and the edge computing device processes data in real time; the drone is equipped with a visual sensor (such as a camera) and an inertial measurement unit (IMU), and the data acquisition module consists of a visual sensor (such as a camera) and an inertial measurement unit (IMU). The visual sensor is used to collect image information of the drone's surrounding environment in real time, and the IMU is used to measure the drone's acceleration, angular velocity and other motion information. The image data collected by the visual sensor will be used as the input of the subsequent relative visual positioning technology (RVL) and absolute visual positioning technology (AVL) algorithms for feature extraction and matching; the motion data collected by the IMU will be used to assist the visual positioning technology to improve the accuracy and real-time performance of positioning.

[0027] Furthermore, the present invention selects a suitable UAV platform, equipped with a gimbal-mounted camera and an IMU. The gimbal-mounted camera collects downward images in real time, and the inertial sensor outputs acceleration and angular velocity data at a high frequency of 200Hz. At the same time, the edge computing device is used to process the collected images and motion information in real time to provide support for subsequent positioning calculations. The present invention adopts a four-degree-of-freedom positioning model (three-dimensional position + yaw angle) and uses the gravity vector measured by the IMU to directly determine the roll and pitch angles, thereby simplifying the positioning calculation. During the data acquisition process, the camera obtains the relative displacement in the plane through feature point extraction and matching algorithms, and the acceleration data of the IMU is combined with the gravity vector to solve the roll and pitch angles to ensure the stability of the attitude reference. For example, when the UAV yaws, The IMU can reflect heading changes in real time, and the visual sensor assists in verifying the yaw angle increment through continuous frame feature tracking to form complementary constraints; within the framework of the visual inertial navigation algorithm, the sliding window optimization is only performed on the four-degree-of-freedom state variables, and by minimizing the visual reprojection error and the IMU pre-integration residual, a tightly coupled estimation of the four-degree-of-freedom state is achieved, effectively avoiding the redundant calculation of the traditional six-degree-of-freedom model and improving the real-time positioning; in addition, the sensor calibration link accurately obtains the camera's intrinsic parameters, IMU noise parameters and extrinsic parameter matrix to ensure the spatiotemporal alignment of visual and inertial data, providing a high-precision foundation for four-degree-of-freedom fusion, reducing the computational load, and is suitable for UAV edge computing platforms, thereby laying a good foundation for state estimation for subsequent AVL and RVL data fusion.

[0028] Step 102: perform feature extraction and matching on the image frames continuously collected by the preprocessed visual sensor, obtain the correspondence between adjacent image frames based on the feature extraction and matching results, fuse the correspondence with the motion information measured by the inertial measurement unit, and obtain the relative positioning information of the drone based on the fusion result.

[0029] Specifically, the corner features in the image frames continuously collected by the preprocessed visual sensor are first extracted and descriptors are generated. The inter-frame feature tracking technology is used to determine the correspondence between adjacent image frames based on the corner features. Then, the visual inertial navigation technology is used to fuse the correspondence between adjacent image frames with the motion information measured by the inertial measurement unit. The relative positioning information of the drone is obtained based on the fusion result. Among them, the correspondence between adjacent image frames is the relative displacement and rotation angle between adjacent image frames.

[0030] More specifically, feature extraction and matching are performed on the image frames continuously collected by the preprocessed visual sensor, and the relative displacement and rotation angle between adjacent frames are calculated. Combined with the motion information provided by the IMU, visual inertial navigation technology is used for data fusion. By constructing relevant state vectors and adopting a sliding window nonlinear optimization framework, dynamic optimization of the relative position of the drone is achieved, error accumulation is suppressed, and relative positioning results are obtained, providing real-time relative position change information for positioning. Among them, the motion information provided by the IMU is filtered by algorithms such as Kalman filtering or extended Kalman filtering.

[0031] Furthermore, the content in step 102 corresponds to the relative visual positioning technology (RVL) module, which builds a relative positioning unit based on the visual inertial navigation algorithm. For the continuous image frames collected by the visual sensor, high-precision relative pose estimation is achieved through the following steps: first, a lightweight and efficient feature extraction algorithm is used to extract the corner features in the image and generate a descriptor, and the correspondence between the feature points of adjacent image frames is obtained through the inter-frame feature tracking technology; then, the acceleration and angular velocity data collected in real time by the IMU are combined, and the IMU measurement values ​​are pre-integrated to compensate for sensor noise and zero bias, and the relative motion approximation between the two frames is generated. To suppress error accumulation, the module adopts a sliding window nonlinear optimization framework to tightly couple the visual reprojection error with the IMU pre-integration residual for nonlinear optimization, constructing a state vector containing the key frame pose, three-dimensional coordinates of feature points and IMU deviation, and realizing dynamic optimization of the relative pose (position, velocity, attitude angle) over a period of time by minimizing the joint cost function; this scheme outputs high-precision relative positioning results in real time through the complementary fusion of the geometric constraints of visual features and the high-frequency measurement data of IMU, effectively solving the error accumulation problem of traditional visual odometry and providing continuous and reliable relative motion state estimation for drones.

[0032] Step 103 , matching and registering the pre-processed image information collected in real time by the visual sensor with the remote sensing images in the pre-built image database, and determining the absolute position information of the UAV in the global coordinate system based on the matching and registration results.

[0033] Specifically, a database containing images of the target area is pre-built, and the image information collected in real time by the current visual sensor is matched with the remote sensing images in the pre-built image database. An image matching algorithm based on deep learning, that is, a neural network based on representation learning, is used to find the most matching database image; then, the absolute position information of the drone in the global coordinate system is determined according to the image registration algorithm, providing a global coordinate reference for subsequent positioning; among them, the neural network algorithm based on representation learning uses a neural network to extract image features, outputs feature vectors through a fully connected layer, and calculates cosine similarity; multi-scale matching and feature fusion methods are also used to improve the robustness and accuracy of matching.

[0034] More specifically, a neural network based on representation learning is first constructed, and the pre-processed image information collected in real time by the visual sensor and the remote sensing images in the pre-built image database are input into the neural network; then, feature extraction is performed on the real-time collected image information and the remote sensing images in the image database in the neural network to obtain the feature vectors of each image, and the similarity between the feature vectors is calculated. Based on the similarity results, the remote sensing image of the area where the UAV is located is obtained; finally, the high-definition remote sensing image of the area where the UAV is located is screened out from the image database, and the image registration algorithm is used to accurately match the feature points, and the homography matrix is ​​estimated to describe the geometric transformation from the aerial image taken by the visual sensor to the high-definition remote sensing image. Based on the geometric transformation results, the absolute position information of the UAV in the global coordinate system is obtained.

[0035] Specifically, the content in step 103 corresponds to the Absolute Visual Positioning (AVL) module: this module pre-builds an image database that stores high-precision map images of the target area. The preliminary processing steps are as follows: during the flight of the drone, the module will match the currently acquired image with the images in the database. To improve the matching efficiency and accuracy, it uses deep learning-based image matching technology to first extract features from the acquired image and the database image to obtain the image's feature vectors. Then, the cosine similarity between these feature vectors is calculated to find the database image that best matches the current image. The remote sensing map of the drone's area is then determined based on the matching results. For details, please refer to Figure 4 ;

[0036] The post-processing steps are as follows: High-definition remote sensing images of the area where the drone is located are selected from the remote sensing image library. An image registration algorithm is used to accurately match feature points and estimate a homography matrix to describe the geometric transformation from the aerial image to the high-resolution remote sensing image. The homography matrix is ​​then used to transform the center point of the aerial image to the pixel coordinate system of the remote sensing image and normalize the coordinates. This is ultimately converted into the absolute position of the drone in the global coordinate system. To account for potential environmental changes and the lack of significant image features, the Absolute Visual Localization (AVL) module employs a multi-scale matching and feature fusion strategy. Multi-scale matching involves analyzing and matching images at different scales to capture features of varying sizes. Feature fusion integrates multiple types of features to enhance their expressiveness, thereby improving matching robustness and ensuring accurate positioning even in complex environments. For details, please refer to [ 15 ] for more information. Figure 3 .

[0037] In step 104, the relative positioning information and the absolute position information of the UAV are fused, and the global path of the UAV is optimized according to the fusion result, thereby achieving accurate positioning of the UAV.

[0038] Specifically, a tightly coupled factor graph optimization framework is first constructed based on the relative positioning information, absolute position information, visual reprojection error and IMU pre-integration residual; then the relative positioning information and absolute position information of the UAV are fused according to the tightly coupled factor graph optimization framework, and the global path of the UAV is optimized according to the fusion result, thereby achieving precise positioning of the UAV.

[0039] More specifically, the content in step 104 corresponds to the data fusion module, the main function of which is to build a tightly coupled factor graph optimization framework. Based on the relative pose sequence output by the RVL module, the absolute position constraint of the AVL module (converting longitude and latitude to ENU coordinates) is used as a global factor, and is uniformly incorporated into the factor graph modeling together with the visual reprojection error and the IMU pre-integration residual. The weight parameters are dynamically adjusted based on the error covariance matrix, environmental feature evaluation, and flight status (such as according to the environmental texture density and the UAV motion state). By minimizing the joint cost function containing relative constraints and absolute constraints, high-precision fusion positioning with reduced positioning error and drift resistance in GNSS-denied environments is achieved. In other words, a global optimization effect of reducing positioning error by 13.16% in GNSS-denied environments is achieved.

[0040] During the fusion process, the module has the ability to dynamically adjust weights, and it will operate flexibly according to the flight status of the UAV and the environmental conditions in which it is located. Specifically, when the UAV is in an area with obvious environmental features and a high absolute positioning accuracy of the AVL module, the module will increase the weight of the absolute positioning result in the fusion in order to more accurately determine the position of the UAV in the global coordinate system. When the UAV flies to an area with unclear environmental features and the AVL module has difficulty in image matching, the module will increase the weight of the relative positioning result of the RVL module, relying on the continuity and stability of relative positioning to maintain the reliability of positioning. By dynamically adjusting weights in this way, the data fusion module can provide the UAV with high-precision and stable positioning information in different environments, ensuring that the UAV can complete its tasks safely and efficiently in complex environments.

[0041] It should also be explained that in response to GNSS signal denial: when the UAV is in an area with high-rise buildings in the city, an indoor environment, or is interfered with by malicious signals, resulting in problems with the GNSS signal, the present invention adopts a method that integrates relative visual positioning technology (RVL) and absolute visual positioning technology (AVL). The RVL module uses the feature relationship between the previous and next frame images to calculate the relative displacement and posture change of the UAV in real time, to compensate for the positioning delay and error that may occur at the moment of signal denial, and to ensure the continuity and accuracy of positioning; at the same time, the AVL module matches the image currently taken by the UAV with a pre-built global reference image database. Taking the urban environment as an example, the database stores high-precision remote sensing images of various areas of the city. After the UAV takes the image of the urban building, the AVL module uses feature extraction and matching algorithms to find the image area that matches it in the database, and then determines the absolute position of the UAV.

[0042] It is necessary to further explain that Figure 2 This is the overall design framework diagram of the algorithm: it shows the overall design framework of the vision-based UAV GNSS denial positioning algorithm, including multimodal sensor data acquisition, dual-channel data processing link (AVL and RVL computing nodes), data fusion and global path optimization. The multimodal sensor data acquisition module is responsible for collecting data from visual sensors, IMUs and barometers, etc. These data enter the AVL and RVL computing nodes for processing respectively. The AVL computing node obtains the absolute position information of the UAV based on the collected image data through image matching and feature analysis. The RVL computing node uses vision and IMU data to calculate the relative position of the UAV through algorithms such as visual inertial navigation. The data fusion module fuses the results of AVL and RVL, and dynamically adjusts the weights of the two according to different scenarios and data reliability. Finally, the global path optimization module plans the optimal flight path based on the fused positioning results, combined with the UAV's mission objectives and environmental constraints.

[0043] Figure 3 This is the workflow diagram of the Absolute Visual Positioning Technology (AVL) module: It shows the process of the AVL module from image acquisition and feature extraction to global coordinate calculation, and realizes the registration of remote sensing images and aerial photos through the image registration algorithm network. It appears when introducing the principles of absolute visual positioning technology, and clearly presents the specific steps of AVL technology from obtaining the current view, feature extraction, to matching with the global reference image, and finally calculating the drone's position.

[0044] Figure 4 This is the structure diagram of the image matching network: This figure shows the structure of the image matching network. In the design and experiment of the drone positioning method based on relative absolute visual positioning technology, it is located in the part that introduces the image matching steps in the AVL node. It presents in detail the process from input image to output matching results, including key links such as feature extraction through the backbone network, pooling layer processing, full connection layer calculation, and similarity measurement using cosine distance. In the image matching step of the AVL node, the network plays a core role. The input image first passes through the backbone network for feature extraction. The backbone network is usually composed of multiple convolutional layers and pooling layers, which can extract high-level semantic features of the image; then the features are processed through the pooling layer. Dimensionality reduction and compression are performed to reduce the amount of data while retaining important feature information. The fully connected layer further processes and maps the pooled features and converts them into feature vectors suitable for similarity calculation. Finally, similarity measurement methods such as cosine distance are used to calculate the similarity between the feature vectors of the input image and the global reference image to find the most matching image area. In one embodiment, in order to improve the performance of the image matching network, the network will be trained and optimized using a large amount of image data, and the network parameters will be adjusted to enable it to better adapt to the matching requirements of different scenes and image features. At the same time, technologies such as data enhancement will be used to expand the diversity of training data and improve the generalization ability of the network.

[0045] The second embodiment of the present invention relates to a UAV positioning system that integrates relative absolute visual positioning technology, see Figure 5 ,include:

[0046] The data acquisition module is used to obtain the image information of the drone's surrounding environment collected in real time by the visual sensor installed on the drone and the motion information of the drone measured in real time by the inertial measurement unit;

[0047] The relative visual positioning module is used to extract and match features from the pre-processed image frames continuously collected by the visual sensor. Based on the feature extraction and matching results, the corresponding relationship between adjacent image frames is obtained. The corresponding relationship is then fused with the motion information measured by the inertial measurement unit to obtain the relative positioning information of the UAV based on the fusion result.

[0048] The absolute visual positioning module is used to match and register the pre-processed image information collected by the visual sensor in real time with the remote sensing images in the pre-built image database, and determine the absolute position information of the UAV in the global coordinate system based on the matching and registration results;

[0049] The data fusion module is used to fuse the relative positioning information and absolute position information of the UAV, and optimize the global path of the UAV based on the fusion results, so as to achieve accurate positioning of the UAV.

[0050] It is not difficult to find that this embodiment is a system embodiment corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment.

[0051] It is worth noting that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovations of the present invention, this embodiment does not include units that are not closely related to solving the technical problems proposed by the present invention. However, this does not mean that other units do not exist in this embodiment.

[0052] The third embodiment of the present invention relates to an electronic device, see Figure 6 ,include:

[0053] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned drone positioning method integrating relative absolute visual positioning technology.

[0054] The memory and processor are connected using a bus, which can include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and memories. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and are therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits it to the processor.

[0055] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0056] A fourth embodiment of the present invention relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0057] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps in the various embodiments of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0058] In summary, the relative visual positioning module in the present invention constructs a tightly coupled visual inertial navigation system based on the visual inertial navigation algorithm to suppress short-term error accumulation; the absolute visual positioning module narrows the matching range of remote sensing images through a lightweight representation learning neural network, combines the image registration algorithm to achieve efficient feature point matching, and uses the homography matrix to calculate the global coordinates of the drone; the two are dynamically fused through a sliding window factor graph optimization framework and combined with the solution of the joint cost function to achieve high-precision positioning in a GNSS-denied environment. Compared with the traditional visual inertial navigation algorithm, the maximum positioning error is reduced by 13.16%, and the accuracy of the scene with high-speed yaw angle changes is significantly improved; the present invention has low computing power overhead, high real-time performance and robustness, and is suitable for operations in complex urban environments, detection in extreme conditions, etc. It can support drones to complete tasks such as autonomous navigation, obstacle avoidance and target detection, and has significant engineering application value and commercial prospects.

[0059] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Any equivalent modifications or variations made by persons skilled in the art without departing from the spirit and technical concepts disclosed herein shall be encompassed by the claims of the present invention.

Claims

1. A UAV positioning method integrating relative absolute visual positioning technology, characterized in that: The following steps are involved: Acquire image information of the drone's surroundings collected in real time by a visual sensor installed on the drone and motion information of the drone measured in real time by an inertial measurement unit, and preprocess the image information and motion information; Perform feature extraction and matching on the pre-processed image frames continuously collected by the visual sensor, obtain the correspondence between adjacent image frames based on the feature extraction and matching results, fuse the correspondence with the motion information measured by the inertial measurement unit, and obtain the relative positioning information of the UAV based on the fusion result; Match and register the pre-processed image information collected in real time by the visual sensor with the remote sensing images in the pre-built image database, and determine the absolute position information of the UAV in the global coordinate system based on the matching and registration results; The relative positioning information and absolute position information of the UAV are fused, and the global path of the UAV is optimized according to the fusion result, thereby achieving accurate positioning of the UAV.

2. The method for positioning an unmanned aerial vehicle (UAV) integrating relative and absolute visual positioning technology according to claim 1, characterized in that: The method includes performing feature extraction and matching on the image frames continuously collected by the preprocessed visual sensor, obtaining a correspondence between adjacent image frames based on the feature extraction and matching results, fusing the correspondence with the motion information measured by the inertial measurement unit, and obtaining relative positioning information of the UAV based on the fusion result, including: Extracting corner features from the pre-processed image frames continuously acquired by the visual sensor and generating descriptors, and using inter-frame feature tracking technology to determine the correspondence between adjacent image frames based on the corner features; Visual inertial navigation technology is used to fuse the correspondence between adjacent image frames and the motion information measured by the inertial measurement unit, and the relative positioning information of the drone is obtained based on the fusion results.

3. The method for positioning an unmanned aerial vehicle (UAV) incorporating relative absolute visual positioning technology according to claim 2, characterized in that: When using visual inertial navigation technology to fuse the correspondence between adjacent image frames and the motion information measured by the inertial measurement unit, in order to suppress error accumulation, a sliding window nonlinear optimization framework is used to tightly couple the visual reprojection error and the IMU pre-integration residual to perform nonlinear optimization. A state vector containing the key frame pose, the three-dimensional coordinates of the feature points and the IMU deviation is constructed, and the dynamic optimization of the relative pose is achieved by minimizing the joint cost function.

4. The method for positioning an unmanned aerial vehicle (UAV) incorporating relative absolute visual positioning technology according to claim 1, characterized in that: The method of matching and registering the pre-processed image information collected in real time by the visual sensor with the remote sensing images in the pre-built image database, and determining the absolute position information of the UAV in the global coordinate system according to the matching and registration results, includes: Constructing a neural network based on representation learning, and inputting the pre-processed image information collected in real time by the visual sensor and the remote sensing images in the pre-constructed image database into the neural network; Performing feature extraction on the real-time collected image information and the remote sensing images in the image database in the neural network to obtain feature vectors of each image, calculating the similarity between the feature vectors, and obtaining a remote sensing image of the area where the drone is located based on the calculated similarity results; High-definition remote sensing images of the area where the drone is located are screened out from the image database, and an image registration algorithm is used to accurately match feature points and estimate the homography matrix to describe the geometric transformation from the aerial image taken by the visual sensor to the high-definition remote sensing image. Based on the geometric transformation results, the absolute position information of the drone in the global coordinate system is obtained.

5. The method for positioning an unmanned aerial vehicle (UAV) integrating relative and absolute visual positioning technology according to claim 1, characterized in that: The method of fusing the relative positioning information and the absolute position information of the UAV and optimizing the global path of the UAV according to the fusion result, thereby achieving accurate positioning of the UAV, includes: A tightly coupled factor graph optimization framework is constructed based on the relative positioning information, absolute position information, visual reprojection error and IMU pre-integration residual. The relative positioning information and absolute position information of the UAV are fused according to the tightly coupled factor graph optimization framework. The global path of the UAV is optimized based on the fusion result, thereby achieving precise positioning of the UAV.

6. The method for positioning an unmanned aerial vehicle (UAV) incorporating relative absolute visual positioning technology according to claim 5, characterized in that: When the relative positioning information and the absolute position information of the UAV are fused according to the tightly coupled factor graph optimization framework, the weight parameters of the relative positioning information and the absolute position information of the UAV in the fusion are dynamically adjusted by minimizing the joint cost function containing relative constraints and absolute constraints.

7. A UAV positioning system integrating relative absolute visual positioning technology, characterized by: include: The data acquisition module is used to obtain the image information of the drone's surrounding environment collected in real time by the visual sensor installed on the drone and the motion information of the drone measured in real time by the inertial measurement unit; The relative visual positioning module is used to extract and match features from the pre-processed image frames continuously collected by the visual sensor, obtain the correspondence between adjacent image frames based on the feature extraction and matching results, fuse the correspondence with the motion information measured by the inertial measurement unit, and obtain the relative positioning information of the UAV based on the fusion result; The absolute visual positioning module is used to match and register the pre-processed image information collected by the visual sensor in real time with the remote sensing images in the pre-built image database, and determine the absolute position information of the UAV in the global coordinate system based on the matching and registration results; The data fusion module is used to fuse the relative positioning information and absolute position information of the UAV, and optimize the global path of the UAV according to the fusion result, so as to achieve accurate positioning of the UAV.

8. An electronic device, characterized in that: include: at least one processor; as well as, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the UAV positioning method integrating relative absolute visual positioning technology as described in any one of claims 1 to 6.

Images