Unmanned aerial vehicle visual navigation method based on infrared semantic feature matching and inter-frame anchoring

Abstract

The invention discloses an unmanned aerial vehicle visual navigation method based on infrared semantic feature matching and inter-frame anchoring, and aims at the core requirement of unmanned aerial vehicle autonomous navigation in a GNSS denial environment, the stability of infrared imaging and the robustness of high-level semantic features are fully combined, and the high-precision, continuous and stable visual autonomous navigation capability is realized. The method is especially suitable for complex scenes such as low-illumination, weak-texture and large-area homogenized areas at night. Through innovation in the aspects of infrared semantic map construction, deep semantic feature extraction, multi-scale semantic matching, semantic anchoring key frame selection, inter-frame recursion positioning and closed-loop correction, Bayesian fusion estimation longitude and latitude solution and the like, the positioning precision and robustness of the unmanned aerial vehicle in GNSS denial, low-visibility and large-area homogeneous scenes are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of unmanned aerial vehicle navigation, and particularly relates to a visual navigation method for unmanned aerial vehicles based on infrared semantic feature matching and inter-frame anchoring. BACKGROUND

[0002] At present, the autonomous navigation of unmanned aerial vehicles in conventional scenes generally relies on the Global Navigation Satellite System (GNSS) to obtain continuous global position information. However, in complex environments such as urban canyons, forest areas, and battlefield electromagnetic countermeasures, GNSS signals are often interfered or deceived, resulting in a decrease in positioning accuracy or even failure. Therefore, autonomous navigation relying on on-board visual sensors has received extensive research and attention.

[0003] At present, the visual positioning method based on scene matching has become an important technical means for unmanned aerial vehicles to realize autonomous navigation in GNSS denial environments. The core idea is to match the real-time images collected by the on-board camera with the prior reference images to estimate the absolute position of the unmanned aerial vehicle. From the perspective of specific implementation, the existing visual scene matching navigation methods can be roughly divided into three categories: the first category is the matching method based on grayscale or correlation, such as Normalized Cross-Correlation (NCC), which is simple to calculate but extremely sensitive to changes in light and texture, and has poor robustness in low light and weak texture scenes; the second category is the matching method based on local feature points and descriptors, which has strong adaptability to rotation and scale changes, but the matching stability significantly decreases in low contrast, weak texture, and infrared imaging environments; the third category is the semantic feature matching method based on deep learning, which extracts high-level features through Convolutional Neural Network (CNN) or Vision Transformer (ViT) models, and has high accuracy in visible light scenes, but there is a feature domain shift in infrared image and infrared-visible light cross-modal matching, and the semantic consistency is insufficient. Although existing research has made certain progress in visible light conditions, there are still the following outstanding problems.

[0004] (1) Infrared modal matching is lacking. Most existing visual navigation methods are based on visible light cameras, and in night, low light, smoke, dust, and other special environments, the image signal-to-noise ratio is low, and the texture information is severely missing, resulting in the almost failure of traditional visible light visual matching algorithms (see patent CN 118506031 B).

[0005] (2) Cross-modal consistency is insufficient. The infrared and visible light (satellite) images have large differences in imaging mechanism, and the existing method lacks a unified semantic consistency correction mechanism, so the cross-modal matching accuracy is low.

[0006] (3) Homogeneous region drift is serious. In large-area homogeneous scenes such as farmland, desert, water surface, snowfield, etc., the matching method based on low-level features is prone to large-scale mismatch and cumulative drift due to the lack of semantic structure constraint.

[0007] (4) Continuity and robustness are lacking. Most of the existing scene matching methods highly depend on global map frame-by-frame matching, which requires high independent matching accuracy for each frame of unmanned aerial vehicle image. When a frame is mismatched in a low-texture area or under strong interference noise, a large local deviation is introduced, reducing the reliability of the overall positioning (see patent CN120426992 A).

[0008] Therefore, how to fully utilize high-level semantic features for stable matching in an infrared imaging environment, and achieve high-precision and continuous and stable unmanned aerial vehicle autonomous navigation in low-illumination and GNSS denial scenarios, has become a core technical problem that needs to be broken through. SUMMARY

[0009] In order to overcome the shortcomings of the prior art, the present application provides a kind of unmanned aerial vehicle visual navigation method based on infrared semantic feature matching and interframe anchoring, for the core demand of unmanned aerial vehicle autonomous navigation in GNSS denial environment, fully combine the stability of infrared imaging and the robustness of high-level semantic features, realize high-precision and continuous and stable visual autonomous navigation capability, especially suitable for night low illumination, weak texture and large-area homogeneous region etc. Complex scene. Through innovation in infrared semantic map construction, deep semantic feature extraction, multi-scale semantic matching, semantic anchor key frame selection, interframe recursive positioning and loop correction, Bayesian fusion estimation latitude and longitude solution, etc., the positioning accuracy and robustness of unmanned aerial vehicle in GNSS denial, low visibility, large-area homogeneous scene are significantly improved.

[0010] The purposes of the present application are as follows: (1) Improve anti-interference ability and all-weather working ability. Infrared imaging and deep semantic feature extraction technology are used, so that the navigation system can still obtain robust semantic observation under low-visibility conditions such as night, fog, smoke, etc., improving the positioning availability and robustness in GNSS strong interference scene.

[0011] (2) Ensure the continuity and robustness of positioning. By constructing semantic anchor key frames and using interframe recursive positioning, the global infrared semantic map matching results are fused to ensure the continuous positioning capability of long-time tasks and suppress cumulative drift.

[0012] (3) Inhibit the same area of the false match and drift. For the large area of the same structure scene such as farmland and industrial park, the structure numbering and topological constraint strategy based on semantics are proposed as the additional semantic constraint of matching to prevent cross-domain false matching and improve the global positioning robustness.

[0013] (4) Good engineering implementability and scalability. The function boundary of the sub-modules such as offline map construction, online semantic extraction, scene matching, key frame anchoring, topological numbering, loop detection and Bayesian fusion is defined according to the principle of modular design, and the real-time inference scheme which can be realized on the edge computing platform is provided.

[0014] The technical scheme adopted by the application to solve the technical problems is as follows: Step 1: offline high-precision infrared semantic map construction; Step 2: real-time infrared image acquisition and semantic feature extraction; Step 3: multi-scale semantic feature matching and initial positioning; Step 4: inter-frame recursive positioning solution based on semantic anchoring key frame; Step 5: scene consistency enhancement mechanism based on semantic topological constraint; Step 6: loop correction and global optimization mechanism; Step 7: multi-source information Bayesian fusion estimation of latitude and longitude solution.

[0015] Preferably, the step 1 is specifically: Step 1: offline high-precision infrared semantic map construction; Step 1-1: in the offline stage of infrared semantic feature scene matching positioning, for the visible light satellite tile image of the target task area, the cross-modal style transfer technology based on generative adversarial network (GAN) is used to convert the original high-resolution visible light satellite tile image to the infrared feature field, and generate an infrared prior map which is highly consistent with the airborne infrared image in terms of thermal radiation mode, texture distribution and semantic structure ; the optimization objective of style transfer is represented as:

[0016] wherein, represents the adversarial loss, by minimizing the loss, it is ensured that the generated infrared image is highly consistent with the real infrared image in terms of thermal radiation features; represents the generator for mapping the visible light image to the infrared field; represents the discriminator for distinguishing the real infrared image and the generated image; represents the real infrared image sample; represents a visible light image sample; represents a real infrared image according to its data distribution take expectation; represents a real visible light image according to its data distribution take expectation; Step 1-2: Combine the deep semantic segmentation network to extract the high-level semantic information of road, building, water body and vegetation , and the segmentation process adopts cross entropy loss optimization:

[0017] wherein, is the cross entropy loss, which is used to measure the difference between the network prediction and the real label; represents the total number of pixels; represents the pixel class label; represents the pixel class probability predicted by the network; Step 1-3: Realize the accurate alignment of map coordinates and global coordinates, and complete the construction of high-resolution infrared semantic map of task area .

[0018] Preferably, the step 2 is specifically: In the online stage, the unmanned aerial vehicle on-board infrared imaging system acquires the nadir thermal radiation image in real time , adopts a multi-scale infrared semantic feature extraction network to explicitly model the scene, and obtains semantic features ; the feature extraction network adopts a multi-scale convolution structure, and the generated multi-layer feature vector is represented as:

[0019] wherein, represents a multi-scale semantic feature extraction network, represents the infrared image input at the current moment, represents the infrared semantic map constructed offline, represents the global semantic features extracted from the infrared semantic map.

[0020] Preferably, the step 3 is specifically: Step 3-1: Coarse scale matching stage; In the low-resolution global feature space, the global similarity based on semantic saliency weight is used to screen the candidate region, and the map sub-region with high similarity to the semantic structure of the current frame is preferentially locked; Step 3-2: Fine scale matching stage; In the rough screening of candidate regions, further extraction of high-resolution local semantic descriptor and combined with geometric consistency constraints and mutual information maximization criterion, eliminate false matching pairs, and keep high confidence matching results; Step 3-3: matching strategy based on semantic saliency weight:

[0021] wherein, The semantic feature similarity is represented as:

[0022] The geographic consistency constraint is represented as: The fusion weights of semantic similarity and geographic constraint are represented as: When , it is determined that the matching is successful, wherein The semantic matching confidence threshold is represented as: Homography matrix based on infrared semantic map Calculate the longitude and latitude coordinates of the unmanned aerial vehicle:

[0023] wherein, The longitude and latitude obtained by matching the infrared semantic map are represented as: The matching feature point pixel coordinates of the current image frame are represented as: The homography matrix of the infrared semantic map is represented as.

[0024] Preferably, the step 4 is specifically: An inter-frame recursive positioning mechanism based on semantic anchor key frame SAK is adopted, and in each frame After completing high-precision matching with the infrared semantic map, if its matching confidence exceeds the threshold , it is selected as the semantic anchor key frame , and its corresponding longitude and latitude are ; For subsequent ordinary frames , the nearest semantic anchor key frame is used as the local reference coordinate system, the relative translation is estimated based on the inter-frame feature point matching result, and the longitude and latitude of the current frame are recursively calculated through the geographic projection relationship:

[0025] wherein, The longitude and latitude of the current frame are represented as: The relative displacement of the key frame and the current frame in the ground coordinate system is represented as: This represents the projection scale matrix from local plane coordinates to latitude and longitude coordinates.

[0026] Preferably, step 5 specifically comprises: Using semantic topology numbering The anti-confusion strategy involves assigning a unique number to each structural unit by performing topological modeling of the regular structures of farmland plots, industrial parks, and factory buildings. Furthermore, semantic numbering consistency constraints are introduced during the matching process:

[0027] in, The semantic matching score represents the confidence level of the match between the current frame image and the infrared semantic map. This indicates the corrected matching score; A unique identifier representing the semantic topological region to which the current frame image belongs; Indicates the semantic topology number to which the anchor keyframe in the current recursive window belongs; This represents the penalty coefficient, used to control the strength of the penalty when there is topological inconsistency; Indicates the indicator function, when the semantic topology number of the current frame... Semantic topology number of anchor keyframe When inconsistent, The value is 1 if it is set to 1, and 0 otherwise. After assigning a unique topology identifier to each region, the identifier system is updated through semantic anchor keyframes to achieve cross-frame topology consistency matching.

[0028] Preferably, step 6 specifically comprises: When the relative displacement or viewpoint change between the drone and the current keyframe exceeds a set threshold, a new SAK is automatically selected and a precise match with the global infrared semantic map is triggered to achieve closed-loop correction.

[0029] in, Represents the optimal coordinate transformation matrix; and These represent the semantic feature vectors of the current frame's infrared image and the infrared semantic map, respectively. and These represent the corresponding sets of geographic coordinate points; This represents a balance factor used to adjust the weight ratio between semantic feature consistency terms and geographic coordinate constraint terms. This represents the perspective transformation matrix from the map coordinate system to the current image coordinate system.

[0030] Preferably, step 7 specifically comprises: A Bayesian fusion strategy is adopted for the absolute localization results of infrared semantic map matching and the inter-frame recursive prediction based on semantically anchored keyframes, with time t set as follows: The target state vector is:

[0031] Leveraging recent semantic anchoring keywords latitude and longitude Inter-frame relative ground translation Through local geographic projection matrix Converting the translation into latitude and longitude increments yields the prior position:

[0032] The linear approximation is as follows:

[0033] in, This represents the Earth's average radius, taken as 6,378,137 meters. This represents the relative displacement increment obtained by inter-frame image matching in the local coordinate system; This indicates the geographic reference coordinates corresponding to the semantic anchoring keyframe; The prior covariance, i.e., the inter-frame recursive uncertainty, is derived from the propagation of keyframe uncertainty and inter-frame motion uncertainty:

[0034] in, Keyframe latitude and longitude covariance, Indicates translation amount The covariance in meters is obtained by estimating the feature matching residuals of two frames and taking into account scale uncertainty. Indicates process noise; Infrared semantic map matching provides latitude and longitude observations based on homography mapping. From pixel coordinates homography mapping matrix get:

[0035] Linearizing this nonlinear mapping, the measurement covariance is determined by the uncertainty of the pixel coordinates. Received:

[0036] in, Represents the Jacobian matrix. Indicates pixel matching covariance. This represents the registration / calibration error covariance of the offline map; The measurement model is represented as:

[0037] Under the priori and Gaussian measurement assumption, the posterior distribution is Gaussian, and the posterior mean and covariance give closed-form solutions; let the priori be , and the measurement be , then the posterior mean and covariance are respectively:

[0038] wherein, represents the uncertainty covariance matrix of the fused state estimation, represents the optimal estimation state after fusion, represents the inverse of the priori estimation covariance matrix obtained by inter-frame recursive positioning, represents the inverse of the measurement covariance matrix obtained by infrared semantic map matching, represents the state value from the current unmanned aerial vehicle sensor observation, represents the state value from the reference map; If the latitude and longitude are regarded as approximately independent, then the scalar Gaussian fusion is adopted for each dimension:

[0039] wherein, and respectively represent the latitude and longitude estimation results of the unmanned aerial vehicle after fusion, and respectively represent the confidence weights of the infrared semantic map matching measurement in the latitude and longitude directions, and respectively represent the confidence weights of the inter-frame recursive positioning result in the latitude and longitude directions.

[0040] The beneficial effects of the present application are as follows: The present application proposes a kind of unmanned aerial vehicle visual navigation method based on infrared semantic feature matching and inter-frame anchoring, through the innovation in the following aspects such as infrared semantic map construction, depth semantic feature extraction, multi-scale semantic matching, semantic anchoring key frame selection, inter-frame recursive positioning and closed loop correction, bayesian fusion estimation latitude and longitude solution, the positioning precision and robustness of unmanned aerial vehicle under GNSS denial, low visibility, large area homogeneous scene can be significantly improved.Specific effects are embodied in the following aspects: (1) Anti-interference and cross-modal robustness are significantly improved. The application utilizes infrared imaging combined with deep semantic feature extraction and cross-modal GAN style transfer technology, and through a multi-scale semantic feature extraction method based on a lightweight CNN, it can still stably extract significant semantic targets such as roads, buildings, and vegetation in low-visibility night environments, effectively overcoming the feature degradation problem of traditional visible light images in weak texture, strong noise, and low light conditions.

[0041] (2) Continuity and robustness are greatly enhanced. The application proposes a Bayesian fusion estimation strategy combined with satellite map direct matching and inter-frame semantic anchoring. This fusion strategy fully utilizes the high-precision characteristics of global semantic map matching, while taking into account the continuity advantage of inter-frame semantics in a local range, which can effectively suppress the cascading drift caused by single frame mismatching, significantly improving the stability and anti-interference ability in long-time tasks. At the same time, combined with the semantic structure number anti-mixing strategy, it can effectively maintain the consistency of positioning in large-area homogeneous regions such as farmland and desert environments.

[0042] (3) Good implementability and platform adaptability. The application adopts modular design, integrating infrared semantic map construction, real-time semantic feature extraction, multi-scale semantic matching, SAK recursive positioning, closed-loop correction, and navigation fusion modules, which is convenient for deployment on various unmanned aerial vehicle on-board computing platforms. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a method flowchart of the application; Figure 2 is an unmanned aerial vehicle mounting platform of the application; Figure 3 is a matching schematic diagram of the aerial image on board (left) and the infrared satellite image after style transfer (right) of the application; Figure 4 is a matching schematic diagram of the homogenization region in inter-frame semantic anchoring of the application; Figure 5 is a comparison diagram of the actual flight trajectory of the unmanned aerial vehicle (blue is the GPS trajectory, green is the visual navigation matching trajectory, L is the starting point); Figure 6 is an infrared semantic feature matching and semantic anchoring key frame recursive flowchart. DETAILED DESCRIPTION

[0044] The application will be further described below in conjunction with the drawings and examples.

[0045] This invention proposes a UAV visual navigation method based on infrared semantic feature matching and inter-frame anchoring. Addressing the core requirement of autonomous UAV navigation in GNSS-denied environments, it fully leverages the stability of infrared imaging and the robustness of high-level semantic features. A novel visual navigation technology integrating global semantic constraints and local recursive optimization is proposed, achieving high-precision and continuously stable autonomous visual navigation capabilities, particularly suitable for complex scenarios such as low-light conditions at night, weak textures, and large areas of homogeneous terrain. The specific technical solution of this invention is as follows: (1) Offline high-precision infrared semantic map construction; First, in the offline stage of infrared semantic feature scene matching and localization, this invention targets the visible light satellite tile imagery of the target task area and uses cross-modal style transfer technology based on generative adversarial networks (GANs) to transform the original high-resolution visible light satellite tile imagery into a more suitable localized image. Transformed to the infrared feature domain, an infrared prior map is generated that is highly consistent with the airborne infrared image in terms of thermal radiation pattern, texture distribution, and semantic structure. The optimization objective for style transfer is expressed as:

[0046] in, This represents the adversarial loss, which is minimized to ensure that the generated infrared image is highly consistent with the real infrared image in terms of thermal radiation characteristics; This represents a generator used to map visible light images to the infrared domain; This represents a discriminator used to distinguish between real infrared images and generated images; Represents a real infrared image sample; Represents a visible light image sample; Indicates the true infrared image According to its data distribution Take the expected value; Represents a true visible light image According to its data distribution Take the expected value.

[0047] Subsequently, a deep semantic segmentation network is used to extract high-level semantic information such as roads, buildings, water bodies, and vegetation. The segmentation process is optimized using cross-entropy loss:

[0048] in, Cross-entropy loss is used to measure the difference between the network's prediction and the true label. Indicates the total number of pixels; Indicates pixel category label; This represents the probability of the pixel category predicted by the network.

[0049] Finally, the accurate alignment of the map coordinates and the global coordinates is realized, and the high-resolution infrared semantic map of the task area is completed .

[0050] (2) Real-time infrared image acquisition and semantic feature extraction; In the online phase, the unmanned aerial vehicle on-board infrared imaging system acquires the downward-looking thermal radiation image in real time The multi-scale infrared semantic feature extraction network is used to explicitly model the key scene, and the semantic feature is obtained The feature extraction network uses a multi-scale convolution structure, and the generated multi-layer feature vector can be expressed as:

[0051] Among them, indicates the multi-scale semantic feature extraction network, indicates the infrared image input at the current moment, indicates the infrared semantic map constructed offline, indicates the global semantic feature extracted from the infrared semantic map.

[0052] (3) Multi-scale semantic feature matching and initial positioning; In order to realize high-precision matching, the present application proposes a multi-scale semantic feature matching strategy between real-time infrared images and offline infrared semantic maps. Firstly, the coarse scale matching stage, in the low-resolution global feature space, the global similarity based on semantic saliency weight is used to quickly screen the candidate area, and the map sub-area with high similarity to the current frame semantic structure is preferentially locked. Then, the fine scale matching stage, in the candidate area screened out, further extract high-resolution local semantic descriptor and combine geometric consistency constraint and mutual information maximization criterion, eliminate false matching pairs, and retain high confidence matching results.

[0053] On this basis, the present application proposes a matching strategy based on semantic saliency weight:

[0054] Among them, is the semantic feature similarity:

[0055] indicates the geographical consistency constraint; respectively indicate the fusion weight of semantic similarity and geographical constraint; When , it is determined that the matching is successful, wherein indicates the semantic matching confidence threshold; the homography matrix based on the infrared semantic map Calculate the latitude and longitude coordinates of the UAV:

[0056] in, This indicates the latitude and longitude obtained through infrared semantic map matching. Indicates the current image frame The pixel coordinates of the matching feature points. The homography matrix represents the infrared semantic map.

[0057] (4) Inter-frame recursive localization solution based on semantically anchored keyframes; To improve the continuity and robustness of localization, this invention proposes an inter-frame recursive localization mechanism based on semantic-anchored keyframes (SAKs). In each frame... After achieving high-precision matching with the infrared semantic map, if the matching confidence exceeds a threshold... Then select it as the semantic anchoring keyframe. Its corresponding latitude and longitude are . For subsequent normal frames Utilize the most recent semantic anchoring keyframes As a local reference coordinate system, the relative translation is estimated based on the inter-frame feature point matching results. And recursively deduce the latitude and longitude of the current frame through geographic projection relationships:

[0058] in, Indicates the latitude and longitude of the current frame. This represents the relative displacement between the keyframe and the current frame in the ground coordinate system, calculated through feature point matching between the two frames and the geometric relationship of camera imaging. This represents the projection scale matrix from local plane coordinates to latitude and longitude coordinates.

[0059] (5) A scene consistency enhancement mechanism that integrates semantic topological constraints; To address the mismatch problem caused by highly similar local features in large homogeneous regions, this invention proposes a semantic topology numbering method. The anti-confusion strategy assigns a unique number to each structural unit by performing topological modeling of regular structures such as farmland plots, industrial parks, and factory buildings. Furthermore, semantic numbering consistency constraints are introduced during the matching process:

[0060] in, The semantic matching score represents the confidence level of the match between the current frame image and the infrared semantic map. represents the modified matching score; represents the unique identifier of the semantic topological region to which the current frame image belongs; represents the semantic topological number to which the anchor key frame in the current recursive window belongs; represents the penalty coefficient for controlling the punishment intensity when the topologies are inconsistent; represents the indicator function, when the semantic topological number of the current frame is inconsistent with the semantic topological number of the anchor key frame , is 1, otherwise 0. After assigning a unique topological identification number to each region, the number system is updated through the semantic anchor key frame, realizing the topological consistency matching across frames.

[0061] (6) Closed-loop correction and global optimization mechanism; When the relative displacement or perspective change of the unmanned aerial vehicle and the current key frame exceeds the set threshold, the application will automatically select a new SAK and trigger accurate matching with the global infrared semantic map, realizing closed-loop correction:

[0062] Among them, represents the optimal coordinate transformation matrix; and respectively represent the semantic feature vectors of the current frame infrared image and the infrared semantic map; and respectively represent the corresponding geographic coordinate point sets; represents the balance factor for adjusting the weight ratio of the semantic feature consistency term and the geographic coordinate constraint term; represents the perspective transformation matrix from the map coordinate system to the current image coordinate system.

[0063] (7) Multi-source information Bayesian fusion estimation of longitude and latitude solution; In order to obtain continuous and robust longitude and latitude positioning coordinates, the application adopts a Bayesian fusion strategy for the absolute positioning results of infrared semantic map matching and inter-frame recursive prediction based on semantic anchor key frames. Let the target state vector at time be

[0064] The longitude and latitude of the nearest semantic anchor key frame and the inter-frame relative ground translation (unit: meter) are used to convert the translation into longitude and latitude increments through the local geographic projection matrix to obtain the prior position:

[0065] The linear approximation form commonly used can be taken as:

[0066] where, denotes the average radius of the earth, taken as 6378137 meters; denotes the relative displacement increment obtained by inter-frame image matching in the local coordinate system; denotes the georeferenced coordinates corresponding to the semantic anchor key frame.

[0067] The prior covariance, i.e., the inter-frame recursive uncertainty, is obtained by the propagation of the key frame uncertainty and the uncertainty of inter-frame movement:

[0068] where, denotes the longitude and latitude covariance of the key frame , and denotes the covariance of the translation in meters, which can be estimated by the two-frame feature matching residual and considering the scale uncertainty, denotes the process noise.

[0069] The infrared semantic map matching gives the longitude and latitude observation based on the homography mapping. The pixel coordinates are calculated by the homography mapping matrix :

[0070] Linearizing the non-linear mapping, the measurement covariance is obtained by the uncertainty of the pixel coordinates :

[0071] where, denotes the Jacobian matrix, denotes the pixel matching covariance, denotes the registration / calibration error covariance of the offline map.

[0072] The measurement model can be expressed as:

[0073] Under the prior and Gaussian measurement assumptions, the posterior distribution is Gaussian, and the posterior mean and covariance give a closed-form solution. Let the prior be and the measurement be . Then:

[0074] where, This represents the uncertainty covariance matrix of the fused state estimate. This represents the optimal estimated state after fusion. This represents the inverse of the prior estimate covariance matrix obtained from inter-frame recursive positioning. This represents the inverse of the measurement covariance matrix obtained from infrared semantic map matching. This represents the state value observed from the current drone's sensors. This indicates the status value from the reference map.

[0075] If latitude and longitude are considered approximately independent, then scalar Gaussian fusion can be applied to each dimension:

[0076] in, and These represent the latitude and longitude estimates of the UAV obtained after fusion, respectively. and These represent the confidence weights of the infrared semantic map matching measurements in the latitude and longitude directions, respectively. and These represent the confidence weights of the inter-frame recursive positioning results in the latitude and longitude directions, respectively.

[0077] Example: Step 1: Construction of a high-precision infrared semantic map; Before the drone takes off, offline mapping is performed on the target flight area. This invention introduces a cross-modal style transfer technique based on Generative Adversarial Networks (GANs) to convert high-resolution visible light satellite tile imagery into infrared style pseudo-imagery, obtaining image data that is highly consistent with airborne infrared imagery in terms of texture and semantic structure. Based on this, a deep semantic segmentation network is used to extract salient semantic targets such as roads, buildings, and vegetation, and geographic coordinates are calibrated to construct a high-precision infrared semantic map, providing a unified reference for subsequent real-time positioning and matching.

[0078] Step 2: Real-time infrared image acquisition and semantic feature extraction; During flight, the drone's onboard infrared camera continuously captures real-time images of the ground's thermal radiation from a direct downward view (sampling frequency 30fps, resolution ). For the acquired images, this invention uses a deep semantic feature extraction method based on a lightweight convolutional neural network to perform multi-scale feature pyramid processing on the input images. By dilated convolution, local and global contextual features are captured to obtain the high-dimensional semantic feature vector of the current frame image. Furthermore, through a cross-modal feature alignment mechanism, the consistency of feature distribution between real-time infrared images and offline infrared semantic maps is ensured.

[0079] Step 3: Multi-scale semantic matching and initial localization; Between real-time infrared image and offline infrared semantic map, the application adopts multi-scale semantic matching strategy. Firstly, candidate regions are quickly screened based on low-resolution global feature vector in coarse scale, then false matching points are removed by using local geometric consistency constraint and mutual information maximization criterion in fine scale, and high confidence matching relationship is obtained. Finally, the longitude and latitude coordinates of the current frame in the map coordinate system are solved through the homography matrix of the infrared semantic map according to the matching result.

[0080] Step four: semantic anchor key frame selection and recursive positioning; When a real-time infrared image frame is high-precision aligned with the infrared semantic map and successfully solves its longitude and latitude coordinates, and the semantic saliency and map alignment confidence are comprehensively considered, the application marks the frame as a semantic anchor key frame (SAK), which is used as a local coordinate system reference in the subsequent inter-frame positioning process. For non-key frames, the application adopts a recursive positioning strategy based on frame-to-SAK, calculates the ground translation of the current frame relative to SAK by extracting the salient semantic structure of the current frame and matching it with the corresponding features in SAK. Combined with the longitude and latitude coordinates of SAK in the world coordinate system, the longitude and latitude coordinates of the current frame are recursively obtained.

[0081] Step five: semantic topology constraint; In a large area of homogeneous scene, a semantic topology structure model and TopoID unique numbering mechanism are applied to assign a unique topological identification number to each region, and the number system is updated by the semantic anchor key frame to avoid feature confusion and ensure the continuity and stability of recursive positioning.

[0082] Step six: key frame update and loop correction; During flight, when the angle of view of the unmanned aerial vehicle and the current SAK changes more than a threshold or the relative displacement exceeds a certain range, the application promotes the current frame to a new SAK, calculates the cross-time semantic correlation, and determines based on the geometric constraint of key features and the local attitude regression residual error to trigger a high-precision matching with the global infrared semantic map, realizing loop correction.

[0083] Step seven: Bayesian fusion of longitude and latitude estimation; After obtaining the SAK recursive positioning result, the application introduces a Bayesian fusion module to jointly estimate the longitude and latitude calculation results based on SAK recursive positioning and the absolute positioning results obtained by directly matching real-time infrared image, and finally outputs the longitude and latitude coordinate results of the unmanned aerial vehicle.

Claims

1. A UAV visual navigation method based on infrared semantic feature matching and inter-frame anchoring, characterized in that, Includes the following steps: Step 1: Offline high-precision infrared semantic map construction; Step 2: Real-time infrared image acquisition and semantic feature extraction; Step 3: Multi-scale semantic feature matching and initial localization; Step 4: Inter-frame recursive localization solution based on semantically anchored keyframes; Step 5: Enhance scene consistency by incorporating semantic topological constraints; Step 6: Closed-loop correction and global optimization mechanism; Step 7: Bayesian fusion estimation of latitude and longitude from multi-source information.

2. The UAV visual navigation method based on infrared semantic feature matching and inter-frame anchoring according to claim 1, characterized in that, Step 1 specifically involves: Step 1: Offline high-precision infrared semantic map construction; Step 1-1: In the offline stage of infrared semantic feature scene matching and localization, for the visible light satellite tile image of the target task area, the original high-resolution visible light satellite tile image is transformed into a cross-modal style transfer technique based on Generative Adversarial Network (GAN) using the visible light satellite tile image. Transformed to the infrared feature domain, an infrared prior map is generated that is highly consistent with the airborne infrared image in terms of thermal radiation pattern, texture distribution, and semantic structure. The optimization objective for style transfer is expressed as: in, This represents the adversarial loss, which is minimized to ensure that the generated infrared image is highly consistent with the real infrared image in terms of thermal radiation characteristics; This represents a generator used to map visible light images to the infrared domain; This represents a discriminator used to distinguish between real infrared images and generated images; Represents a real infrared image sample; Represents a visible light image sample; Indicates the true infrared image According to its data distribution Take the expected value; Represents a true visible light image According to its data distribution Take the expected value; Steps 1-2: Extract high-level semantic information of roads, buildings, water bodies, and vegetation using a deep semantic segmentation network. The segmentation process is optimized using cross-entropy loss: in, Cross-entropy loss is used to measure the difference between the network's prediction and the true label. Indicates the total number of pixels; Indicates pixel category label; This represents the probability of the pixel category predicted by the network. Steps 1-3: Achieve precise alignment between map coordinates and global coordinates to complete the construction of a high-resolution infrared semantic map of the task area. .

3. The UAV visual navigation method based on infrared semantic feature matching and inter-frame anchoring according to claim 2, characterized in that, Step 2 specifically involves: During the online phase, the UAV's onboard infrared imaging system acquires real-time images of the downward-facing thermal radiation. A multi-scale infrared semantic feature extraction network is used to explicitly model the scene and obtain semantic features. The feature extraction network employs a multi-scale convolutional structure, and the generated multi-layer feature vectors are represented as follows: in, This represents a multi-scale semantic feature extraction network. This indicates the current infrared image input. This represents an offline constructed infrared semantic map. This represents the global semantic features extracted from the infrared semantic map.

4. The UAV visual navigation method based on infrared semantic feature matching and inter-frame anchoring according to claim 3, characterized in that, Step 3 specifically involves: Step 3-1: Coarse-scale matching stage; In a low-resolution global feature space, candidate regions are selected using global similarity based on semantic saliency weights, prioritizing map sub-regions that are highly similar to the semantic structure of the current frame. Step 3-2: Fine-scale matching stage; Within the candidate regions selected by the initial screening, high-resolution local semantic descriptors are further extracted and combined with geometric consistency constraints and mutual information maximization criteria to eliminate incorrect matching pairs and retain high-confidence matching results. Step 3-3: Matching strategy based on semantic saliency weights: in, Indicates semantic feature similarity: Indicates geographical consistency constraints; These represent the fusion weights for semantic similarity and geographical constraints, respectively. when When the match is successful, the match is considered to be successful. Represents the semantic matching confidence threshold; homography matrix based on infrared semantic map. Calculate the latitude and longitude coordinates of the UAV: in, This indicates the latitude and longitude obtained through infrared semantic map matching. Indicates the current image frame The pixel coordinates of the matching feature points The homography matrix represents the infrared semantic map.

5. The UAV visual navigation method based on infrared semantic feature matching and inter-frame anchoring according to claim 4, characterized in that, Step 4 specifically involves: An inter-frame recursive localization mechanism based on semantic anchor keyframe (SAK) is adopted in each frame. After achieving high-precision matching with the infrared semantic map, if the matching confidence exceeds a threshold... Then select it as the semantic anchoring keyframe. Its corresponding latitude and longitude are ; For subsequent normal frames Utilize the most recent semantic anchoring keyframes As a local reference coordinate system, the relative translation is estimated based on the inter-frame feature point matching results. And recursively deduce the latitude and longitude of the current frame through geographic projection relationships: in, Indicates the latitude and longitude of the current frame. This represents the relative displacement between the keyframe and the current frame in the ground coordinate system, calculated through feature point matching between the two frames and the camera imaging geometry. This represents the projection scale matrix from local plane coordinates to latitude and longitude coordinates.

6. The UAV visual navigation method based on infrared semantic feature matching and inter-frame anchoring according to claim 5, characterized in that, Step 5 specifically involves: Using semantic topology numbering The anti-confusion strategy involves assigning a unique number to each structural unit by performing topological modeling of the regular structures of farmland plots, industrial parks, and factory buildings. Furthermore, semantic numbering consistency constraints are introduced during the matching process: in, The semantic matching score represents the confidence level of the match between the current frame image and the infrared semantic map. This indicates the corrected matching score; A unique identifier representing the semantic topological region to which the current frame image belongs; Indicates the semantic topology number to which the anchor keyframe in the current recursive window belongs; This represents the penalty coefficient, used to control the strength of the penalty when there is topological inconsistency; Indicates the indicator function, when the semantic topology number of the current frame... Semantic topology number of anchor keyframe When inconsistent, The value is 1 if it is set to 1, and 0 otherwise. After assigning a unique topology identifier to each region, the identifier system is updated through semantic anchor keyframes to achieve cross-frame topology consistency matching.

7. The UAV visual navigation method based on infrared semantic feature matching and inter-frame anchoring according to claim 6, characterized in that, Step 6 specifically involves: When the relative displacement or viewpoint change between the drone and the current keyframe exceeds a set threshold, a new SAK is automatically selected and a precise match with the global infrared semantic map is triggered to achieve closed-loop correction. in, Represents the optimal coordinate transformation matrix; and These represent the semantic feature vectors of the current frame's infrared image and the infrared semantic map, respectively. and These represent the corresponding sets of geographic coordinate points; This represents a balance factor used to adjust the weight ratio between semantic feature consistency terms and geographic coordinate constraint terms. This represents the perspective transformation matrix from the map coordinate system to the current image coordinate system.

8. The UAV visual navigation method based on infrared semantic feature matching and inter-frame anchoring according to claim 7, characterized in that, Step 7 specifically involves: A Bayesian fusion strategy is adopted for the absolute localization results of infrared semantic map matching and the inter-frame recursive prediction based on semantically anchored keyframes, with time t set as follows: The target state vector is: Leveraging recent semantic anchoring keywords latitude and longitude Inter-frame relative ground translation Through local geographic projection matrix Converting the translation into latitude and longitude increments yields the prior position: The linear approximation is as follows: in, This represents the Earth's average radius, taken as 6,378,137 meters. This represents the relative displacement increment obtained by inter-frame image matching in the local coordinate system; This indicates the geographic reference coordinates corresponding to the semantic anchoring keyframe; The prior covariance, i.e., the inter-frame recursive uncertainty, is derived from the propagation of keyframe uncertainty and inter-frame motion uncertainty: in, Keyframe latitude and longitude covariance, Indicates translation amount The covariance in meters is obtained by estimating the feature matching residuals of two frames and taking into account scale uncertainty. Indicates process noise; Infrared semantic map matching provides latitude and longitude observations based on homography mapping. From pixel coordinates homography mapping matrix get: Linearizing this nonlinear mapping, the measurement covariance is determined by the uncertainty of the pixel coordinates. Received: in, Represents the Jacobian matrix. Indicates pixel matching covariance. This represents the registration / calibration error covariance of the offline map; The measurement model is represented as follows: Under the prior and Gaussian measurement assumptions, the posterior distribution is Gaussian, and the posterior mean and covariance give closed-form solutions; let the prior be... Measurement is Then the posterior mean and covariance are respectively: in, This represents the uncertainty covariance matrix of the fused state estimate. This represents the optimal estimated state after fusion. This represents the inverse of the prior estimate covariance matrix obtained from inter-frame recursive positioning. This represents the inverse of the measurement covariance matrix obtained from infrared semantic map matching. This represents the state value observed from the current drone's sensors. This indicates a state value derived from a reference map; If latitude and longitude are considered approximately independent, then scalar Gaussian fusion is applied to each dimension: in, and These represent the latitude and longitude estimates of the UAV obtained after fusion, respectively. and These represent the confidence weights of the infrared semantic map matching measurements in the latitude and longitude directions, respectively. and These represent the confidence weights of the inter-frame recursive positioning results in the latitude and longitude directions, respectively.

Images