Unmanned aerial vehicle multi-mode visual positioning method and system in GNSS denial environment

Abstract

The invention relates to an unmanned aerial vehicle multi-modal visual positioning method and system in a GNSS denial environment, and the method comprises the steps: constructing a visual inertial mileage calculation method based on deep learning feature extraction and matching, calculating the relative positioning of an unmanned aerial vehicle, and outputting to obtain high-frequency 6-DoF pose information; performing scene matching of infrared RGB image multi-modal fusion based on the Officient-LoFTR, and obtaining absolute positioning in a world system by solving a relative pose between a current key frame picture and a matched satellite image; a factor graph optimization mode is adopted to carry out self-adaptive weight information fusion according to the confidence coefficient of absolute positioning and relative positioning under a world system, and 6-DoF pose information with high frequency and higher precision is output; based on high-frequency and higher-precision 6-DoF pose information, after the unmanned aerial vehicle is recovered, all aerial images and satellite images are fused and updated. The system comprises a calculation module and a fusion updating module. According to the invention, more accurate and high-frequency pose output can be provided.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned aerial vehicle navigation, and in particular relates to a multi-modal visual positioning method and system for unmanned aerial vehicle in a GNSS-denied environment. Background Art

[0002] In recent years, the rapid development and widespread application of drone technology has attracted widespread attention. Drones have shown great potential in aerial photography, agricultural plant protection, logistics distribution, disaster monitoring and other fields. However, the precise positioning of drones has always been a challenging problem. Traditional positioning methods mainly rely on the Global Navigation Satellite System (GNSS), but in complex environments such as urban canyons, densely forested areas or indoor environments, GNSS signals may be affected by blocking or multipath effects, resulting in reduced positioning accuracy or even loss of positioning capabilities. Therefore, the development of a reliable and accurate drone positioning perception technology has become an urgent research need.

[0003] In this context, UAV visual perception positioning has become one of the research hotspots. By using the visual sensors carried by UAVs, such as cameras, lidar, etc., rich environmental information can be obtained and analyzed and processed through image processing and computer vision algorithms. This positioning method based on visual perception is low-cost and has high flexibility and adaptability, and can achieve accurate positioning and navigation under various environmental conditions. In addition, UAV visual perception positioning can also be combined with other sensor data, such as inertial measurement unit (IMU) and ground feature extraction, to further improve the accuracy and robustness of positioning. By integrating multiple perception data and adopting advanced algorithms and technologies, UAVs can achieve functions such as high-precision autonomous navigation, obstacle avoidance and precise positioning, thus bringing more possibilities for UAV applications. Summary of the invention

[0004] The present invention focuses on the problem of autonomous positioning of UAVs in GNSS-denied environments, and proposes a multimodal visual positioning method and system for UAVs in GNSS-denied environments. The present invention proposes a scene matching fusion strategy for RGB images and infrared images, which complements the matching results of RGB and infrared images to obtain more accurate absolute positioning. The present invention also proposes a factor graph optimization fusion strategy with adaptive weights based on absolute positioning confidence. In addition, the present invention proposes a satellite map update mechanism, which performs affine transformation of the aerial image to the satellite image plane based on the final posture obtained by fusion, and updates the satellite map.

[0005] In order to achieve the above object, the present invention adopts the following technical solution: The multi-modal visual positioning method of UAV in GNSS-denied environment includes the following steps: 1) Construct a visual inertial odometry calculation method based on deep learning feature extraction and matching to calculate the relative positioning of the drone and output high-frequency 6-DoF pose information; 2) Based on the high-frequency 6-DoF pose information, the satellite map is searched, and scene matching based on multimodal fusion of infrared RGB images is performed based on Efficient-LoFTR. The absolute positioning in the world system is obtained by solving the relative pose between the current key frame image and the matched satellite image. 3) Adopting factor graph optimization to perform adaptive weighted information fusion based on the confidence of absolute positioning and relative positioning in the world system, output a high-frequency and more accurate 6-DoF pose information; 4) Based on the high-frequency and more accurate 6-DoF pose information, after the UAV is recovered, all aerial images and satellite images are fused and updated to provide new satellite images for the next UAV positioning.

[0006] A further improvement of the present invention is that in step 1), a visual inertial odometer calculation method based on deep learning feature extraction and matching is constructed to calculate the relative positioning of the drone and output high-frequency 6-DoF pose information, including: ORB-SLAM3, which is improved based on deep learning features and matching, is used as the visual inertial mileage calculation method. SuperPoint is used for feature extraction. The SuperPoint network architecture consists of three components: a shared encoder, a feature detection decoder, and a descriptor decoder. The network then outputs a semi-dense descriptor, and then a bicubic interpolation algorithm is applied to obtain the complete descriptor. After obtaining the feature point tensor and descriptor tensor output by the network, an adaptive threshold selection strategy is used to filter out the feature points and perform post-processing operations to obtain the feature points and their descriptors. In order to achieve efficient feature matching, the LightGlue model is adopted, which contains multiple identical layers to jointly process two feature sets; each layer contains self-attention and cross-attention units to update point representations; the classifier in each layer determines the reasoning stop; finally, the lightweight head calculates the partial matching score; the network depth is dynamically adjusted according to the complexity of the input image; if the image pair is easy to match, it is terminated early due to the high token confidence; Given two sets of feature points extracted from images A and B, where image A has M feature points and image B has N feature points, the feature points of image A are represented as , the feature points of image B are represented as Each feature point needs to be Normalized to the range of 0 to 1; the description of the feature point is expressed as ; After inputting them into the LightGlue network, the distribution matrix will be output Its elements Indicates the possibility of matching the i-th feature point of image A with the j-th feature point of image B; In order to obtain high-frequency 6-DoF pose information, the inference code of CNN and GNN is converted from Python to C++ and deployed using ONNX and NVIDIA TensorRT, where 16-bit floating-point operations replace 32-bit floating-point operations.

[0007] A further improvement of the present invention is that, in step 2), according to the high-frequency 6-DoF pose information, the satellite map is searched, and scene matching based on the multimodal fusion of infrared RGB images is performed based on Efficient-LoFTR, and the absolute positioning in the world system is obtained by solving the relative pose between the current key frame image and the matched satellite image, including: The global coordinates of the UAV in the geodetic coordinate system are obtained by matching the aerial image with the satellite image; Satellite images and aerial images are divided into two stages: the initialization matching stage and the motion process matching stage. The initialization matching stage takes into account the matching accuracy. The field of view of the drone's aerial images will increase as the drone's altitude increases. As the drone continues to rise, the matching error will continue to decrease. In the motion process matching stage, the drone's field of view changes less, and the drone's motion constraints are considered to obtain a faster matching speed. First, the satellite image is divided into blocks, and the W_S×H_S satellite image is divided into n blocks of satellite sub-images of W_S / √n×H_S / √n; Then, feature matching is performed on the collected image and the satellite image through Efficient-LoFTR, and matching pairs are retained according to the matching confidence threshold. In the motion matching stage, the reference search area is provided by the global pose optimized by the backend, and the matching of the next frame is achieved by searching in the satellite sub-image block of the current frame along the direction of the UAV's motion, so as to achieve fast search and matching during the motion process. Assuming that the size of the satellite sub-image block of the current frame is W_SC×H_SC, a mask of (W_SC+S_x)×(H_SC+S_y) is created as the satellite sub-image of the next frame, where the size and positive and negative of S_x and S_y are positively correlated with the speed of the UAV. Based on the obtained matching pairs, the estimated homography matrix H between the UAV image and the local satellite map is obtained through OpenCV. Then the coordinates of the UAV in the global map are Estimated to be:

[0008] in and Represent the center coordinates and the upper left corner coordinates of the satellite image respectively.

[0009] A further improvement of the present invention is that, in step 3), adaptive weighted information fusion is performed according to the confidence of absolute positioning and relative positioning in the world system by using factor graph optimization, and a high-frequency and more accurate 6-DoF pose information is output, including: In the scene matching process, the matching pair between the current key frame image and the satellite image has been obtained through Efficient-LoFTR, and the 2-DoF pose estimation of the current UAV in the world system has been solved based on the matching pair research; Define the state parameters for optimization for

[0010] in, is the inverse depth of the feature point; and Represents the starting index of the feature point and the number of feature points observed in all key frames in the sliding window; and Represents the translation and rotation matrix from the camera to the IMU; and Represents the translation and rotation matrix from the world coordinate system to the camera coordinate system; The measurement residual model is constructed according to the current factor graph framework and optimized by minimizing the residual function:

[0011] in, represents the covariance factor, which is related to the weighting factor of the measurement; It is the prior information that can be calculated after the frames in the sliding window are marginalized; is the prior Jacobian matrix of the Hessian matrix generated after the last optimization; Represents the observation residual of IMU; Represents the feature point reprojection residual; Represents the position and velocity residuals of scene matching.

[0012] A further improvement of the present invention is that it also includes: 1) Visual feature factor node The residual of the visual observation value is the reprojection error of the camera feature point. The error between the estimated value and the observed value of the projection position in the normalized camera coordinate system is the reprojection error. The visual observation residual is established:

[0013]

[0014]

[0015] in, is the camera intrinsic parameter matrix; and Representing feature points In the and Coordinates of the frame image plane; and They are the camera coordinate system to IMU Coordinate System The translation and rotation matrices of For feature points The inverse depth of and Respectively frame IMU Translation and rotation matrices from coordinate system to world coordinate system; and Respectively Frame world coordinate system to IMU Translation and rotation matrices of the coordinate system; 2) IMU pre-integration factor node In Section B, the IMU has been pre-integrated to obtain the residuals at two adjacent moments; 3) Scene matching factor node The scene matching is interpolated to reduce the impact of time asynchrony; thus, the scene matching is The interpolation of the moment is shown in the formula:

[0016] in, and After scene matching, the camera is and The coordinates of the moment, using the camera as a sensor for scene matching; ; By combining IMU, the position error equation of scene matching is obtained: .

[0017] Multi-modal visual positioning system for UAVs in GNSS-denied environments, including: The first calculation module builds a visual inertial odometer calculation method based on deep learning feature extraction and matching, calculates the relative positioning of the drone, and outputs high-frequency 6-DoF pose information; The second computing module searches the satellite map based on the high-frequency 6-DoF pose information, uses scene matching based on multimodal fusion of infrared RGB images based on Efficient-LoFTR, and obtains the absolute positioning in the world system by solving the relative pose between the current key frame image and the matched satellite image; The third calculation module uses factor graph optimization to perform adaptive weighted information fusion based on the confidence of absolute positioning and relative positioning in the world system, and outputs a high-frequency and more accurate 6-DoF pose information; The fusion update module, based on the high-frequency and more accurate 6-DoF pose information, fuses and updates all aerial images with satellite images after the drone is recovered, providing new satellite images for the next drone positioning.

[0018] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the multi-modal visual positioning method for a drone in a GNSS-denied environment.

[0019] Compared with the prior art, the present invention has at least the following beneficial technical effects: The multimodal visual positioning method and system for UAV in GNSS-denied environment provided by the present invention is based on the framework of scene matching and visual odometer fusion. Compared with positioning only by visual SLAM or only by scene matching, the present invention can provide more accurate and high-frequency posture output.

[0020] Furthermore, a scene matching algorithm using Efficient LoFTR as feature matching is constructed, which can not only maintain high output efficiency but also handle large viewing angle changes and obtain dense matching results in scenes with poor textures.

[0021] Furthermore, factor graph optimization is introduced, and the scene matching results are used as constraint factors to construct a factor graph with the visual inertial odometry factor. The confidence of the scene matching results is considered for adaptive weighted fusion, and finally a more accurate 6-DoF pose result is obtained.

[0022] Currently, this method can enable the drone to achieve a positioning accuracy of 10m based on visible light images at an altitude of 500m, and the algorithm frame rate is 25 FPS. The positioning accuracy based on infrared images can reach 30m, and the algorithm frame rate is 25 FPS. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A simplified diagram of the UAV visual localization process.

[0024] Figure 2Schematic diagram of constructing the factor graph framework.

[0025] Figure 3 It is a complete aerial image of the drone during flight.

[0026] Figure 4 This is a block diagram of the multimodal visual positioning system for UAVs in GNSS-denied environments. DETAILED DESCRIPTION

[0027] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0028] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0029] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0030] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0031] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0032] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0033] Example 1 like Figure 1 As shown, the multimodal visual positioning method of a UAV in a GNSS-denied environment provided by the present invention comprises the following steps: 1) Construct a visual-inertial odometry method based on deep learning feature extraction and matching to calculate the relative positioning of the drone and output high-frequency 6-DoF pose information.

[0034] 2) According to the high-frequency 6-DoF pose information, the satellite map is searched, and scene matching based on multimodal fusion of infrared RGB images is performed based on Efficient-LoFTR. The absolute positioning in the world system is obtained by solving the relative pose between the current key frame image and the matched satellite image.

[0035] 3) The factor graph optimization method is used to perform adaptive weighted information fusion based on the confidence of absolute positioning and relative positioning in the world system, and output a high-frequency and more accurate 6-DoF pose information.

[0036] 4) After the drone is recovered, all aerial images are fused and updated with satellite images to provide new satellite images for the next drone positioning.

[0037] In this embodiment, the specific implementation method of step 1) is as follows: The present invention adopts ORB-SLAM3 based on deep learning features and matching improvement as the visual inertial mileage calculation method. Compared with its traditional ORB feature descriptor, the deep learning feature points can have stronger matching ability in the weak texture area of ​​the aerial picture, and the efficiency is higher. The present invention adopts SuperPoint for feature extraction. The SuperPoint network architecture mainly consists of three components: a shared encoder, a feature detection decoder and a descriptor decoder. The encoder is a VGG-style network that can reduce the image dimension and extract features. The task of the feature detection decoder is to calculate the probability of each pixel of the image to determine its possibility as a feature point. The descriptor decoding network uses sub-pixel convolution to reduce the computational complexity of the decoding process. Then the network outputs a semi-dense descriptor, and then a bicubic interpolation algorithm is applied to obtain a complete descriptor. After obtaining the feature point tensor and descriptor tensor output by the network, in order to improve the robustness of feature extraction, an adaptive threshold selection strategy is used to filter out feature points and perform post-processing operations to obtain feature points and their descriptors.

[0038] The LightGlue model contains multiple identical layers that process two feature sets together. Each layer contains self-attention and cross-attention units to update point representations. Classifiers in each layer determine the inference stop, avoiding unnecessary computations. Finally, the lightweight head computes the partial matching score. The network depth is dynamically adjusted based on the complexity of the input image. If the image pair is easy to match, it can be terminated early due to high token confidence. As a result, LightGlue has a shorter runtime and lower memory footprint, making it suitable for integration into tasks that require real-time performance.

[0039] Given two sets of feature points extracted from images A and B, where image A has M feature points and image B has N feature points, the feature points of image A are represented as , the feature points of image B are represented as Each feature point needs to be Normalized to the range of 0 to 1. The description of the feature point is expressed as After feeding them into the LightGlue network, the output distribution matrix Its elements Indicates the possibility that the i-th feature point of image A matches the j-th feature point of image B.

[0040] Since the time interval between adjacent frames is very short, usually only tens of milliseconds, ORB-SLAM3 assumes that the camera moves at a uniform speed during this short period of time. It uses the pose and velocity of the previous frame to estimate the pose of the current frame, and performs projection matching using the estimated pose. It then searches for matching points within a certain range and refines the pose accordingly. However, in reality, the movement of the camera may not always be uniform. Sudden acceleration, deceleration, or rotation can have an adverse effect on the effectiveness of this method. LightGlue can effectively solve this problem by directly matching features between the current frame and the previous frame. It then uses these matched features to refine the initial pose estimate, thereby reducing the negative impact of sudden acceleration or rotation.

[0041] The inference code of CNN and GNN was converted from Python to C++ and deployed using ONNX and NVIDIA TensorRT, where 16-bit floating-point operations replaced 32-bit floating-point operations. This greatly accelerated the efficiency of feature matching.

[0042] In this embodiment, the specific implementation method of step 2) is as follows: The present invention studies the positioning of UAVs in GNSS-denied conditions. However, there may be cumulative errors when relying solely on visual SLAM for positioning. At the same time, due to the lack of GNSS signals, the coordinates of the UAV in the geodetic coordinate system cannot be obtained, which is not conducive to global positioning. Therefore, the global coordinates of the UAV in the geodetic coordinate system are obtained by matching aerial images with satellite images.

[0043] Satellite images and aerial images can be divided into two stages: the initialization matching stage and the motion matching stage. The initialization matching stage mainly considers the matching accuracy. The field of view of the drone's aerial image will increase as the drone's altitude increases. As the drone continues to rise, the matching error will continue to decrease. In the motion matching stage, the drone's field of view changes less, and the drone's motion constraints can be considered to obtain a faster matching speed.

[0044] Since the heights of satellites and drones are very different, the size and resolution of satellite images and aerial images are also very different. Directly matching the two will generate a huge amount of calculation and consume a lot of time. In order to improve the matching speed and ensure the matching accuracy of the two images, the present invention first divides the satellite image into blocks, dividing the W_S×H_S satellite image into n blocks of W_S / √n×H_S / √n satellite sub-images.

[0045] Then, feature matching is performed on the collected image and the satellite image using Efficient-LoFTR, and matching pairs are retained based on the matching confidence threshold. Based on the matching pairs obtained, the estimated homography matrix H between the drone image and the local satellite map is obtained using OpenCV. Then the coordinates of the drone in the global map are It can be estimated as:

[0046] in and Represent the center coordinates and the upper left corner coordinates of the satellite image respectively.

[0047] In the motion matching stage, considering that the motion of the UAV is coherent, when the matching error is low, the global pose optimized by the backend can be used to provide a reference search area, and the next frame can be matched by searching the satellite sub-image block of the current frame along the motion direction of the UAV to achieve fast search and matching during the motion process. Assuming that the size of the satellite sub-image block of the current frame is W_SC×H_SC, create a mask of (W_SC+S_x)×(H_SC+S_y) as the satellite sub-image of the next frame, where the size and positive and negative of S_x and S_y are positively correlated with the speed of the UAV.

[0048] In this embodiment, the specific implementation method of step 3) is as follows: Since the confidence of the output of scene matching is not high at the beginning of the ascent of the drone, and scene matching fails in some rapidly developing areas, a tightly coupled optimization method is used in the back-end optimization. However, scene matching only uses the output global coordinates for back-end optimization, so even if scene matching fails, it will not affect vision and IMU.

[0049] In the scene matching process, the matching pair between the current key frame image and the satellite image has been obtained through Efficient-LoFTR, and the 2-DoF pose estimation of the current UAV in the world system is solved based on the matching pair. Therefore, the factor graph framework is constructed as follows Figure 2 shown.

[0050] Define the state parameters that need to be optimized for

[0051] in, is the inverse depth of the feature point; and Represents the starting index of the feature point and the number of feature points observed in all key frames in the sliding window; and Represents the translation and rotation matrix from the camera to the IMU; and Represents the translation and rotation matrix from the world coordinate system to the camera coordinate system.

[0052] The measurement residual model is constructed according to the current factor graph framework and optimized by minimizing the residual function:

[0053] in, represents the covariance factor, which is related to the weighting factor of the measurement; It is the prior information that can be calculated after the frames in the sliding window are marginalized; is the prior Jacobian matrix of the Hessian matrix generated after the last optimization; Represents the observation residual of IMU; Represents the feature point reprojection residual; Represents the position and velocity residuals of scene matching.

[0054] 1) Visual feature factor node The residual of the visual observation value is the reprojection error of the camera feature point. The error between the estimated value and the observed value of the projection position in the normalized camera coordinate system is the reprojection error. The visual observation residual is established:

[0055]

[0056]

[0057] in, is the camera intrinsic parameter matrix; and Representing feature points In the and Coordinates of the frame image plane; and They are the camera coordinate system to IMU Coordinate System The translation and rotation matrices of For feature points The inverse depth of and Respectively frame IMU Coordinate System Translation and rotation matrices to world coordinates; and Respectively Frame world coordinate system to IMU Translation and rotation matrices of the coordinate system.

[0058] 2) IMU pre-integration factor node In Section B, the IMU has been pre-integrated to obtain the residuals at two adjacent moments.

[0059] 3) Scene matching factor node Since the scene matching frequency is low, directly fusing the scene matching results with the visual inertial odometry will produce errors. Therefore, the scene matching is interpolated to reduce the impact of time asynchrony. The interpolation of the moment is shown in the formula:

[0060] in, and After scene matching, the camera is and The coordinates of the time, the present invention uses the camera as a sensor for scene matching; .

[0061] By combining IMU, the position error equation for scene matching can be obtained:

[0062] 4) Scene matching confidence calculation The confidence estimation of scene matching is mainly determined by evaluating whether the warped image is close to a rectangle. It determines how close its shape is to a rectangle by calculating the internal angles at the four vertices of the quadrilateral and comparing these angles with the internal angles of an ideal rectangle (90 degrees). For each vertex, the internal angle of the vertex is obtained by calculating the angle between the two sides adjacent to the vertex. The angle calculation uses the dot product formula between two vectors, and the dot product value is converted into an angle value by the arc cosine function (acos). Since each vertex of the quadrilateral is connected by two sides, the angles of the four vertices are calculated in turn. After the internal angle of each vertex is calculated, the program compares it with the ideal rectangular angle of 90 degrees and calculates the angle deviation (that is, the difference between the internal angle and 90 degrees). This deviation is expressed in absolute value to ensure that a positive deviation value can be obtained whether the angle is greater than or less than 90 degrees. The angle deviations of the four vertices will be accumulated one by one. The smaller the total deviation value, the closer the quadrilateral is to a rectangle. On the contrary, if the deviation is large, it means that the shape of the quadrilateral is far from a rectangle. The sum of the internal angle deviations can be used to further deduce how close the quadrilateral is to the rectangle. A smaller angle deviation means that the shape of the quadrilateral is closer to the rectangle, so the confidence level will be higher. When the total deviation is larger, it means that the shape deviates further from the rectangle, and the confidence level will decrease.

[0063] Example 2 The present invention provides a multi-modal information fusion positioning architecture for unmanned aerial vehicles. Its main innovation is to use RGB images, infrared images combined with visual inertial odometers to perform scene matching from coarse to fine in large-scale satellite maps to solve absolute positioning and propose a mechanism for adaptive fusion weight correction based on absolute posture confidence. The key frame images of aerial photography are fused with satellite images to update the reference satellite image library: First, collect the RGB image, infrared image and inertial navigation information of the drone. The RGB image and inertial navigation information are used to obtain relative and high-frequency poses through the ORB-SLAM3 visual inertial odometer based on deep learning feature matching. According to the pose information of the odometer, the satellite map of the corresponding position in the satellite image library is retrieved, and the RGB image and infrared image are matched with the satellite image respectively through Efficient-LoFTR, and the matching is filtered according to the confidence of the matching pair. The homography matrix between the drone image and the satellite image is solved through the matching results to obtain the 2-DoF absolute positioning of the drone. Then the low-frequency absolute pose is used to align the coordinate system of the visual inertial odometer with the coordinate system of the satellite map. Then, the confidence of the absolute pose is combined with the factor graph optimization fusion of the visual inertial odometer to obtain a more accurate, stable and smooth 6-DoF pose output.

[0064] In actual projects, it is found that the time of constructing satellite images is usually far away from the current time, and scene matching may fail in some rapidly developing areas. Therefore, it is necessary to use aerial images to build prefabricated maps. After the drone is recovered, all aerial images and satellite images are fused and updated to provide new satellite images for the next drone positioning.

[0065] The aerial images are fused using a weighted average. The pixel value in the overlapping area is related to the distance between the pixel point and the original image. The farther the distance, the The smaller the point value on the image, the larger it is. Similarly, for the original image It is the same, as shown in the formula.

[0066]

[0067] in, and represents the weight value, and , , .

[0068] like Figure 3 As shown, a complete aerial image of the entire UAV flight process is obtained, and after overlaying the satellite image, an updated satellite image can be obtained.

[0069] Example 3 like Figure 4 As shown, the multimodal visual positioning system for unmanned aerial vehicles in a GNSS-denied environment provided by the present invention includes: The first calculation module builds a visual inertial odometer calculation method based on deep learning feature extraction and matching, calculates the relative positioning of the drone, and outputs high-frequency 6-DoF pose information; The second computing module searches the satellite map based on the high-frequency 6-DoF pose information, uses scene matching based on multimodal fusion of infrared RGB images based on Efficient-LoFTR, and obtains the absolute positioning in the world system by solving the relative pose between the current key frame image and the matched satellite image; The third calculation module uses factor graph optimization to perform adaptive weighted information fusion based on the confidence of absolute positioning and relative positioning in the world system, and outputs a high-frequency and more accurate 6-DoF pose information; The fusion update module, based on the high-frequency and more accurate 6-DoF pose information, fuses and updates all aerial images with satellite images after the drone is recovered, providing new satellite images for the next drone positioning.

[0070] Example 4 The present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the multi-modal visual positioning method for unmanned aerial vehicles in a GNSS-denied environment are implemented.

[0071] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0072] The present application is described with reference to the flowcharts and / or block diagrams of the methods, systems and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.

[0073] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0074] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0075] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the attached claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved.

[0076] In addition, it should be understood that although this specification is described in accordance with the implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation modes that can be understood by those skilled in the art. The above content is only to illustrate the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution according to the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A multi-modal visual positioning method for UAVs in a GNSS-denied environment, characterized in that: The following steps are involved: 1) Construct a visual inertial odometry calculation method based on deep learning feature extraction and matching to calculate the relative positioning of the drone and output high-frequency 6-DoF pose information; 2) Based on the high-frequency 6-DoF pose information, the satellite map is searched, and scene matching based on multimodal fusion of infrared RGB images is performed based on Efficient-LoFTR. The absolute positioning in the world system is obtained by solving the relative pose between the current key frame image and the matched satellite image. 3) Adopting factor graph optimization to perform adaptive weighted information fusion based on the confidence of absolute positioning and relative positioning in the world system, output a high-frequency and more accurate 6-DoF pose information; 4) Based on the high-frequency and more accurate 6-DoF pose information, after the UAV is recovered, all aerial images and satellite images are fused and updated to provide new satellite images for the next UAV positioning.

2. The multimodal visual positioning method for UAV in a GNSS-denied environment according to claim 1, characterized in that: In step 1), a visual inertial odometry calculation method based on deep learning feature extraction and matching is constructed to calculate the relative positioning of the drone and output high-frequency 6-DoF pose information, including: ORB-SLAM3, which is improved based on deep learning features and matching, is used as the visual inertial mileage calculation method. SuperPoint is used for feature extraction. The SuperPoint network architecture consists of three components: a shared encoder, a feature detection decoder, and a descriptor decoder. The network then outputs a semi-dense descriptor, and then a bicubic interpolation algorithm is applied to obtain the complete descriptor. After obtaining the feature point tensor and descriptor tensor output by the network, an adaptive threshold selection strategy is used to filter out the feature points and perform post-processing operations to obtain the feature points and their descriptors. In order to achieve efficient feature matching, the LightGlue model is adopted, which contains multiple identical layers to jointly process two feature sets; each layer contains self-attention and cross-attention units to update point representations; the classifier in each layer determines the reasoning stop; finally, the lightweight head calculates the partial matching score; the network depth is dynamically adjusted according to the complexity of the input image; if the image pair is easy to match, it is terminated early due to the high token confidence; Given two sets of feature points extracted from images A and B, where image A has M feature points and image B has N feature points, the feature points of image A are represented as , the feature points of image B are represented as Each feature point needs to be Normalized to the range of 0 to 1; the description of the feature point is expressed as ; After inputting them into the LightGlue network, the distribution matrix will be output Its elements Indicates the possibility of matching the i-th feature point of image A with the j-th feature point of image B; In order to obtain high-frequency 6-DoF pose information, the inference code of CNN and GNN is converted from Python to C++ and deployed using ONNX and NVIDIA TensorRT, where 16-bit floating-point operations replace 32-bit floating-point operations.

3. The multimodal visual positioning method for UAV in a GNSS-denied environment according to claim 2, characterized in that: In step 2), the satellite map is searched based on the high-frequency 6-DoF pose information, and scene matching based on multimodal fusion of infrared RGB images is performed based on Efficient-LoFTR. The absolute positioning in the world system is obtained by solving the relative pose between the current key frame image and the matched satellite image, including: The global coordinates of the UAV in the geodetic coordinate system are obtained by matching the aerial image with the satellite image; Satellite images and aerial images are divided into two stages: the initialization matching stage and the motion process matching stage. The initialization matching stage takes into account the matching accuracy. The field of view of the drone's aerial images will increase as the drone's altitude increases. As the drone continues to rise, the matching error will continue to decrease. In the motion process matching stage, the drone's field of view changes less, and the drone's motion constraints are considered to obtain a faster matching speed. First, the satellite image is divided into blocks, and the W_S×H_S satellite image is divided into n blocks of satellite sub-images of W_S / √n×H_S / √n; Then, feature matching is performed on the collected image and the satellite image through Efficient-LoFTR, and matching pairs are retained according to the matching confidence threshold. In the motion matching stage, the reference search area is provided by the global pose optimized by the backend, and the matching of the next frame is achieved by searching in the satellite sub-image block of the current frame along the direction of the UAV's motion, so as to achieve fast search and matching during the motion process. Assuming that the size of the satellite sub-image block of the current frame is W_SC×H_SC, a mask of (W_SC+S_x)×(H_SC+S_y) is created as the satellite sub-image of the next frame, where the size and positive and negative of S_x and S_y are positively correlated with the speed of the UAV. Based on the obtained matching pairs, the estimated homography matrix H between the UAV image and the local satellite map is obtained through OpenCV. Then the coordinates of the UAV in the global map are Estimated to be: in and Represent the center coordinates and the upper left corner coordinates of the satellite image respectively.

4. The multimodal visual positioning method for UAV in a GNSS-denied environment according to claim 3, characterized in that: In step 3), factor graph optimization is used to perform adaptive weighted information fusion based on the confidence of absolute positioning and relative positioning in the world system, and output a high-frequency and more accurate 6-DoF pose information, including: In the scene matching process, the matching pair between the current key frame image and the satellite image has been obtained through Efficient-LoFTR, and the 2-DoF pose estimation of the current UAV in the world system has been solved based on the matching pair research; Define the state parameters for optimization for in, is the inverse depth of the feature point; and Represents the starting index of the feature point and the number of feature points observed in all key frames in the sliding window; and Represents the translation and rotation matrix from the camera to the IMU; and Represents the translation and rotation matrix from the world coordinate system to the camera coordinate system; The measurement residual model is constructed according to the current factor graph framework and optimized by minimizing the residual function: in, represents the covariance factor, which is related to the weighting factor of the measurement; It is the prior information that can be calculated after the frames in the sliding window are marginalized; is the prior Jacobian matrix of the Hessian matrix generated after the last optimization; Represents the observation residual of IMU; Represents the feature point reprojection residual; Represents the position and velocity residuals of scene matching.

5. The multimodal visual positioning method for UAV in a GNSS-denied environment according to claim 4, characterized in that: Also includes: 1) Visual feature factor node The residual of the visual observation value is the reprojection error of the camera feature point. The error between the estimated value and the observed value of the projection position in the normalized camera coordinate system is the reprojection error. The visual observation residual is established: in, is the camera intrinsic parameter matrix; and Representing feature points In the and Coordinates of the frame image plane; and They are the camera coordinate system to IMU Coordinate System The translation and rotation matrices of For feature points The inverse depth of and Respectively frame IMU Translation and rotation matrices from coordinate system to world coordinate system; and Respectively Frame world coordinate system to IMU Translation and rotation matrices of the coordinate system; 2) IMU pre-integration factor node In Section B, the IMU has been pre-integrated to obtain the residuals at two adjacent moments; 3) Scene matching factor node The scene matching is interpolated to reduce the impact of time asynchrony; thus, the scene matching is The interpolation of the moment is shown in the formula: in, and After scene matching, the camera is and The coordinates of the moment, using the camera as a sensor for scene matching; ; By combining IMU, the position error equation of scene matching is obtained: 。 6. A multi-modal visual positioning system for UAVs in GNSS-denied environments, characterized by: include: The first calculation module builds a visual inertial odometer calculation method based on deep learning feature extraction and matching, calculates the relative positioning of the drone, and outputs high-frequency 6-DoF pose information; The second computing module searches the satellite map based on the high-frequency 6-DoF pose information, uses scene matching based on multimodal fusion of infrared RGB images based on Efficient-LoFTR, and obtains the absolute positioning in the world system by solving the relative pose between the current key frame image and the matched satellite image; The third calculation module uses factor graph optimization to perform adaptive weighted information fusion based on the confidence of absolute positioning and relative positioning in the world system, and outputs a high-frequency and more accurate 6-DoF pose information; The fusion update module, based on the high-frequency and more accurate 6-DoF pose information, fuses and updates all aerial images with satellite images after the drone is recovered, providing new satellite images for the next drone positioning.

7. The multi-modal visual positioning system for unmanned aerial vehicles in a GNSS-denied environment according to claim 6, characterized in that: In the first calculation module, a visual inertial odometer calculation method based on deep learning feature extraction and matching is constructed to calculate the relative positioning of the drone and output high-frequency 6-DoF pose information, including: ORB-SLAM3, which is improved based on deep learning features and matching, is used as the visual inertial mileage calculation method. SuperPoint is used for feature extraction. The SuperPoint network architecture consists of three components: a shared encoder, a feature detection decoder, and a descriptor decoder. The network then outputs a semi-dense descriptor, and then a bicubic interpolation algorithm is applied to obtain the complete descriptor. After obtaining the feature point tensor and descriptor tensor output by the network, an adaptive threshold selection strategy is used to filter out the feature points and perform post-processing operations to obtain the feature points and their descriptors. In order to achieve efficient feature matching, the LightGlue model is adopted, which contains multiple identical layers to jointly process two feature sets; each layer contains self-attention and cross-attention units to update point representations; the classifier in each layer determines the reasoning stop; finally, the lightweight head calculates the partial matching score; the network depth is dynamically adjusted according to the complexity of the input image; if the image pair is easy to match, it is terminated early due to the high token confidence; Given two sets of feature points extracted from images A and B, where image A has M feature points and image B has N feature points, the feature points of image A are represented as , the feature points of image B are represented as Each feature point needs to be Normalized to the range of 0 to 1; the description of the feature point is expressed as ; After inputting them into the LightGlue network, the distribution matrix will be output Its elements Indicates the possibility of matching the i-th feature point of image A with the j-th feature point of image B; In order to obtain high-frequency 6-DoF pose information, the inference code of CNN and GNN is converted from Python to C++ and deployed using ONNX and NVIDIA TensorRT, where 16-bit floating-point operations replace 32-bit floating-point operations.

8. The multi-modal visual positioning system for unmanned aerial vehicles in a GNSS-denied environment according to claim 7, characterized in that: In the second calculation module, the satellite map is searched according to the high-frequency 6-DoF pose information, and the scene matching based on the multimodal fusion of infrared RGB images is performed based on Efficient-LoFTR. The absolute positioning in the world system is obtained by solving the relative pose between the current key frame image and the matched satellite image, including: The global coordinates of the UAV in the geodetic coordinate system are obtained by matching the aerial image with the satellite image; Satellite images and aerial images are divided into two stages: the initialization matching stage and the motion process matching stage. The initialization matching stage takes into account the matching accuracy. The field of view of the drone's aerial images will increase as the drone's altitude increases. As the drone continues to rise, the matching error will continue to decrease. In the motion process matching stage, the drone's field of view changes less, and the drone's motion constraints are considered to obtain a faster matching speed. First, the satellite image is divided into blocks, and the W_S×H_S satellite image is divided into n blocks of satellite sub-images of W_S / √n×H_S / √n; Then, feature matching is performed on the collected image and the satellite image through Efficient-LoFTR, and matching pairs are retained according to the matching confidence threshold. In the motion matching stage, the reference search area is provided by the global pose optimized by the backend, and the matching of the next frame is achieved by searching in the satellite sub-image block of the current frame along the direction of the UAV's motion, so as to achieve fast search and matching during the motion process. Assuming that the size of the satellite sub-image block of the current frame is W_SC×H_SC, a mask of (W_SC+S_x)×(H_SC+S_y) is created as the satellite sub-image of the next frame, where the size and positive and negative of S_x and S_y are positively correlated with the speed of the UAV. Based on the obtained matching pairs, the estimated homography matrix H between the UAV image and the local satellite map is obtained through OpenCV. Then the coordinates of the UAV in the global map are Estimated to be: in and Represent the center coordinates and the upper left corner coordinates of the satellite image respectively.

9. The multi-modal visual positioning system for unmanned aerial vehicles in a GNSS-denied environment according to claim 8, characterized in that: In the third calculation module, factor graph optimization is used to perform adaptive weighted information fusion based on the confidence of absolute positioning and relative positioning in the world system, and output a high-frequency and more accurate 6-DoF pose information, including: In the scene matching process, the matching pair between the current key frame image and the satellite image has been obtained through Efficient-LoFTR, and the 2-DoF pose estimation of the current UAV in the world system has been solved based on the matching pair research; Define the state parameters for optimization for in, is the inverse depth of the feature point; and Represents the starting index of the feature point and the number of feature points observed in all key frames in the sliding window; and Represents the translation and rotation matrix from the camera to the IMU; and Represents the translation and rotation matrix from the world coordinate system to the camera coordinate system; The measurement residual model is constructed according to the current factor graph framework and optimized by minimizing the residual function: in, represents the covariance factor, which is related to the weighting factor of the measurement; It is the prior information that can be calculated after the frames in the sliding window are marginalized; is the prior Jacobian matrix of the Hessian matrix generated after the last optimization; Represents the observation residual of IMU; Represents the feature point reprojection residual; Represents the position and velocity residuals of scene matching.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the multimodal visual positioning method for a drone in a GNSS-denied environment according to any one of claims 1 to 5.

Images