Passive geographic information overlay system and method based on cross-modal features

Abstract

The application belongs to the technical field of unmanned aerial vehicle navigation and augmented reality, and relates to a passive geographic information superimposition system and method based on cross-modal features, which comprises: an airborne visual acquisition module installed on an unmanned aerial vehicle platform, used for real-time acquisition of video frames and flight attitude data; an edge computing processing module used for receiving video frames and performing geographic information calculation, calculating a confidence level, and outputting geographic feature information meeting a set confidence threshold as an optimal data set; and an augmented reality display module used for superimposing geographic feature information in the optimal data set in the video screen in an augmented reality manner. The system has the beneficial effects of positioning matching and augmented display without dependence on satellite signals in a GNSS denial environment, has the advantages of strong anti-interference capability, high positioning accuracy, low superimposition delay, and the like, can realize superimposition and presentation of geographic features in the unmanned aerial vehicle video, and is convenient for users to quickly identify and understand the geographic position and environmental elements corresponding to the screen.

Description

Technical Field

[0001] This invention relates to the fields of unmanned aerial vehicle (UAV) navigation and augmented reality technology, and in particular to a passive geographic information overlay system and method based on cross-modal features. Background Technology

[0002] Existing UAV geographic information augmentation display technologies typically rely on Global Navigation Satellite Systems (GNSS) to obtain absolute coordinates for overlaying map features. However, in GNSS-denied environments such as urban high-rise buildings or complex electromagnetic countermeasures, satellite signals are easily interfered with or lost, rendering traditional overlay methods relying on absolute coordinates completely ineffective. Furthermore, while existing visual-assisted positioning methods (such as visual odometry) can calculate relative displacement, they accumulate drift over time and struggle to overcome the significant feature differences between aerial video (visual modality) and vector maps (GIS modality), exhibiting poor robustness in scenarios with changing viewpoints or repetitive textures. Simultaneously, during high-speed UAV maneuvers, the lack of effective temporal compensation mechanisms often leads to "tag separation" between geographic labels and video footage due to computational delays in visual algorithms, severely impacting commanders' real-time judgment.

[0003] Therefore, there is an urgent need for a passive geographic information overlay method for UAV videos that does not rely on satellite signals, can achieve accurate positioning based on cross-modal features, and has low-latency real-time overlay capability. Summary of the Invention

[0004] Technical problems to be solved

[0005] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a passive geographic information overlay system and method based on cross-modal features, which solves the technical problems of existing UAVs being unable to obtain geographic location by relying on satellite signals in GNSS denied environments, poor real-time performance of video stream and geographic information overlay, and label drift caused by visual interference.

[0006] Technical solution

[0007] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0008] In a first aspect, the present invention provides a passive geographic information overlay system based on cross-modal features, comprising: an airborne visual acquisition module, an edge computing processing module, and an augmented reality display module;

[0009] The airborne vision acquisition module is installed on the UAV platform and is used to acquire video frames and flight attitude data in real time.

[0010] The edge computing processing module is used to receive video frames and perform geographic information calculations, calculate confidence levels, and output geographic feature information that meets the set confidence threshold as the preferred dataset.

[0011] The augmented reality display module is used to overlay geographic feature information from the preferred dataset onto the video screen in an augmented reality manner.

[0012] As a further improvement to the system of the present invention, the airborne vision acquisition module includes an industrial camera, a six-axis inertial measurement unit (IMU), and a time synchronization unit;

[0013] The industrial camera is used to acquire continuous video frames containing street and ground features under GNSS rejection conditions; it is connected to the edge computing processing module through a data interface to transmit video streams.

[0014] The six-axis inertial measurement unit is used to collect the angular velocity and acceleration information of the UAV body;

[0015] The time synchronization unit is used to perform hardware-level alignment of video frames and inertial data.

[0016] As a further improvement to the system of the present invention, the edge computing processing module adopts a high-performance embedded AI computing platform, including a local map sample construction unit, a cross-modal feature retrieval unit, and a localization and confidence evaluation unit.

[0017] The local map sample construction unit is used to store mapping data from the global vector map to local image features;

[0018] The cross-modal feature retrieval unit, based on a visual Transformer network, performs global semantic encoding on the input video frames to obtain visual features, maps the visual features to the vector map feature space, and outputs retrieval results as a candidate dataset. The retrieval results include candidate street segment identifiers and key element coordinates.

[0019] The location and confidence assessment unit combines road topology constraints and temporal smoothing strategies to locate the search results, calculate the confidence level of the output results, and filter the search results based on the confidence level. It then selects the geographic feature information that meets the set confidence threshold as the preferred dataset and assigns it the corresponding display priority or rendering strategy.

[0020] As a further improvement to the system of the present invention, the augmented reality display module runs on a ground control station or an airborne head-mounted display, adopts a multi-threaded asynchronous rendering architecture, balances the difference between video acquisition and algorithm processing rates through a circular buffer, and drives the highlighting or downgrading of geographic features based on the confidence level.

[0021] In a second aspect, the present invention provides a passive geographic information overlay method based on the passive geographic information overlay system based on cross-modal features as described in any one of the first aspects above, comprising the following steps:

[0022] Step 1: Construct a retrieval database of UAV video imagery and vector map elements;

[0023] Step 2: During the flight of the UAV, extract the depth visual features of the real-time video frames in the GNSS denied environment and use them as global semantic vectors. Match them with the vector map feature vectors in the map feature library and output a coarse matching candidate set.

[0024] Step 3: Combining inertial measurement data and road topology constraints, perform temporal smoothing and confidence level evaluation on the coarse matching candidate set, and output the geographic feature information that meets the set confidence threshold as the preferred dataset based on the evaluation results;

[0025] Step 4: Map the geographic feature information from the selected dataset back to the video frames, overlay it onto the video screen, and then perform low-latency augmented reality rendering and dynamic coordinate correction to achieve real-time fusion of video and geographic information.

[0026] As a further improvement to the method of the present invention, in step 1, by calibrating the intrinsic and extrinsic parameters of the industrial camera, the geographic elements in the GIS vector map are projected from geographic coordinates to the local map coordinate system, a topological structure map of the road segment and intersection connectivity is established, and a globally unique ID is assigned to each street segment to form a map element library.

[0027] As a further improvement to the method of the present invention, in step 2, during the flight of the UAV, the depth visual features of the real-time video frames are extracted under GNSS denial environment and used as global semantic vectors. These features are then matched with vector map feature vectors in the map feature library to output a coarse matching candidate set. Specifically, this includes the following methods: Step 2.1: Extract the current time step using the pre-trained visual Transformer backbone network. single-frame video global feature vector : ; In the formula, It is a visual encoding network pre-trained with a large amount of data; Step 2.2: Calculate the cosine similarity score between the global feature vector and the map feature representation. : ; In the formula, Indicates the first Map features representing candidate street segments, The norm of a vector; Step 2.3: Sort the cosine similarity scores, select the Top-N results, assign them corresponding initial matching probability scores, and form a coarse matching candidate set. ,in, This represents the identifier of the map feature corresponding to the nth coarse matching candidate set. This represents the matching score corresponding to the nth coarse matching candidate set.

[0028] As a further improvement to the method of the present invention, in step 3, by combining inertial measurement data and road topology constraints, temporal smoothing and confidence level evaluation are performed on the coarse matching candidate set. Based on the evaluation results, geographic feature information that meets the set confidence threshold is output as the preferred dataset. Specifically, this includes the following methods: Step 3.1: Angular velocity based on airborne IMU acquisition With acceleration Calculate the drone from the previous moment up to the current moment Relative rotation matrix and displacement observation; Calculate the relative rotation matrix using angular velocity integral : ; The influence of gravity g is eliminated based on the pre-integral theory, and the current time is calculated. Relative displacement observation : ; In the formula, For antisymmetric matrix operators, The time difference between two frames The velocity at the previous moment, For zero acceleration bias, j represents the time index; Step 3.2: Introduce relative displacement observations at the continuous video frame level to construct short-time stable prior trajectory positions, and use the previous moment... The optimal posterior estimated location and the current moment Relative displacement observation Calculate the prior position at the current moment. : ; Step 3.3: Based on Construct a spatial gating constraint in the form of a Gaussian distribution to eliminate abnormal candidate nodes in the coarse matching candidate set; Step 3.4: Taking into account the difference in cosine similarity scores in the coarse matching candidate set after removing abnormal nodes, the road structure constraint satisfaction, the consistency measure of continuous frames, and the relative motion consistency index, calculate the confidence value of geographic element information and classify it into levels. Step 3.5: Based on the evaluation results, output one or more geographic feature information that meet the set confidence threshold as the preferred dataset, and assign them the corresponding display priority or rendering strategy; Step 3.6: To address the abrupt changes in geographic feature detection results across consecutive video frames and ensure the stability of geographic feature information matching with video frames, a temporal smoothing algorithm is used to optimize the street attribution results.

[0029] As a further improvement to the method of the present invention, in step 3.4, the confidence level value... The calculation is as follows: ; In the formula, and These are the cosine similarity scores of the first and second highest scores in the coarse matching candidate set, respectively. For topological constraints, As a time series continuity indicator, These are weighting coefficients; Among them, the The formula used to evaluate the connectivity rationality of candidate road segments matched between two consecutive frames on the digital map is as follows: ; In the formula, The topological penalty coefficient is... This represents the shortest pathfinding distance between the two candidate road segments on the digital map topology. The The aforementioned consecutive frame consistency metrics and relative motion consistency indices are comprehensively quantified to evaluate the consistency between visually matched displacement and inertial navigation-estimated displacement. The calculation formula is as follows: ; In the formula, This represents the actual geometric displacement of the candidate matching points on the map between the two frames. The set standard deviation for displacement noise; The confidence level classification rule is as follows: Set upper limit threshold With lower threshold ,like If it is judged as high confidence level, If it is judged as low confidence level, Then it is determined to be unusable data.

[0030] As a further improvement to the method of the present invention, in step 4, the geographic element information in the preferred dataset is mapped back to video frames, superimposed on the video image, and then subjected to low-latency augmented reality rendering and dynamic coordinate correction to achieve real-time fusion presentation of video and geographic information. Specifically, this includes the following methods:

[0031] Step 4.1: Construct a parallel processing pipeline that separates perception, computation, and rendering;

[0032] Step 4.2: Set up a circular data buffer between each module to store the IMU high-frequency attitude data sequence, thereby balancing the high frame rate acquisition of video frames with the computation time of the visual positioning algorithm;

[0033] Step 4.3: Introduce temporal smoothing filtering and motion compensation matrices into the rendering pipeline. Utilize the continuous frame information within the buffer to perform reprojection correction on the GIS coordinates of geographic features in the overlaid GIS vector map, and calculate the screen coordinates. It is also assigned a geographic element information label.

[0034] Beneficial effects

[0035] The beneficial effects of this invention are:

[0036] This paper presents a passive geographic information overlay system and method based on cross-modal features, which solves the problems of existing UAVs being unable to obtain geographic location by satellite signals in GNSS-denied environments, poor real-time performance of video stream and geographic information overlay, and label drift caused by visual interference. It enables positioning matching and enhanced display without satellite signals in GNSS-denied environments. By combining airborne edge computing and cross-modal depth feature retrieval technology, and utilizing direct matching of visual features and vector map elements, it achieves accurate geographic information enhancement display under passive conditions, reducing dependence on external communication links. It also overcomes the instability of traditional visual positioning in high-dynamic scenarios, possessing advantages such as strong anti-interference capability, high positioning accuracy, and low overlay latency. It can achieve stable overlay and clear presentation of geographic elements in UAV videos, facilitating users to quickly identify and understand the corresponding geographic location and environmental elements in the image, improving the situational awareness efficiency of UAV reconnaissance and the intuitiveness of command and decision-making. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of a passive geographic information overlay system module structure based on cross-modal features, provided in an embodiment of the present invention.

[0038] Figure 2 A flowchart of a passive geographic information overlay method based on cross-modal features provided in an embodiment of the present invention;

[0039] Figure 3This is a flowchart of the method for outputting a coarse matching candidate set in an embodiment of the present invention;

[0040] Figure 4 This is a flowchart of the method for optimizing the output dataset in an embodiment of the present invention;

[0041] Figure 5 This is a flowchart illustrating the method for real-time fusion and presentation of video and geographic information in an embodiment of the present invention. Detailed Implementation

[0042] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0043] In this embodiment:

[0044] The industrial camera is a panoramic shutter industrial camera with a resolution of 1920×1080 and a frame rate of 30fps.

[0045] The time synchronization unit is a hardware time synchronization board.

[0046] Firstly, such as Figure 1 As shown, this embodiment of the invention provides a passive geographic information overlay system based on cross-modal features, including: an airborne visual acquisition module, an edge computing processing module, and an augmented reality display module.

[0047] The airborne vision acquisition module is installed on the UAV platform and is used to acquire high-definition video frames and flight attitude data of the UAV.

[0048] The airborne vision acquisition module includes an industrial camera, a six-axis inertial measurement unit (IMU), and a time synchronization unit.

[0049] The industrial camera is used to acquire continuous video frames containing street and ground features under GNSS rejection conditions, and is connected to the edge computing processing module via GigE or MIPI interface to transmit video streams;

[0050] The six-axis inertial measurement unit is used to collect the angular velocity and acceleration information of the UAV body;

[0051] The time synchronization unit is used to perform hardware-level alignment of video frames and inertial data.

[0052] The edge computing processing module, as the airborne computing core, is used to receive video frames and perform geographic information processing, calculate confidence level assessment, and output one or more geographic element information that meet the set confidence threshold as the preferred dataset based on the assessment results.

[0053] The edge computing processing module adopts a high-performance embedded AI computing platform (NVIDIA Jetson series), which internally deploys a local map sample construction unit, a cross-modal feature retrieval unit, and a localization and confidence evaluation unit.

[0054] The local map sample construction unit is used to store mapping data from the global vector map to local image features;

[0055] The cross-modal feature retrieval unit, based on a visual Transformer network, performs global semantic encoding on any single frame of the input video frame to obtain visual features, maps the visual features to the vector map feature space, and outputs retrieval results as candidate datasets based on the visual information. The retrieval results include candidate street segment identifiers and key element coordinates.

[0056] The location and confidence assessment unit combines road topology constraints and temporal smoothing strategies to locate the search results, calculate the confidence level of the output results, and filter the search results based on the confidence level. It then selects the geographic feature information that meets the set confidence threshold as the preferred dataset and assigns it the corresponding display priority or rendering strategy.

[0057] The augmented reality display module, which runs on a ground control station or an airborne head-mounted display, includes a video stream rendering engine (developed based on OpenGL / Vulkan) and a visualization interaction interface. It is used to overlay geographic feature information (such as road names and POI icons) from the selected dataset onto the video screen in an augmented reality (AR) manner.

[0058] Specifically, the augmented reality display module adopts a multi-threaded asynchronous architecture, balances the rate difference between video acquisition and algorithm processing through a buffer mechanism, and drives the highlighting or downgrading of geographic features in the video frame according to the confidence level.

[0059] Secondly, such as Figures 2 to 5 As shown, this embodiment of the invention provides a passive geographic information overlay method based on a cross-modal feature-based passive geographic information overlay system, comprising the following steps:

[0060] Step 1: In the mission preparation phase, construct a retrieval database of UAV video images and vector map elements.

[0061] Specifically, by analyzing the intrinsic parameter matrix of industrial cameras And external parameters are calibrated, and geographic features in the GIS vector map (such as road centerline point sets) are calibrated. By projecting geographic coordinates onto a local map coordinate system, a topological structure map of road segments and intersection connections is established, and a globally unique ID is assigned to each street segment, forming a map element library covering street centerlines and prominent landmarks, which serves as a retrieval database.

[0062] Step 2: During the flight of the UAV, extract the depth visual features of the real-time video frames in the GNSS denied environment and use them as global semantic vectors. Match them with the vector map feature vectors in the map feature library to output a coarse matching candidate set containing several candidate street segments and key geographic features.

[0063] Step 2.1: Read the current time through the edge computing module single-frame video The pre-trained visual Transformer backbone network is input, and this network extracts the implicit road direction and building layout semantics through a multi-layer self-attention mechanism to generate a global feature vector containing scene semantics. :

[0064] ;

[0065] In the formula, It is a visual encoding network pre-trained with a large amount of data.

[0066] Step 2.2: Introduce a positioning prediction head to jointly analyze the geometric shape of the road, the topology of the intersection, and the significant features of the ground in the image. Establish the correspondence between visual features and vector map elements, and quantify the matching degree between any single frame of the current video and the i-th street segment to obtain the cosine matching similarity score.

[0067] Specifically, the global feature vector Compared with the pre-calculated vector map feature vectors in the map feature library Mapping to the same feature space, calculate the cosine similarity score between the two. :

[0068] ;

[0069] In the formula, Indicates the first Map features representing candidate street segments, The norm of a vector.

[0070] Step 2.3: Directly infer and output the two-dimensional coordinates of several high-scoring candidate street segment identifiers and key elements in the image from the original video frames. Sort the cosine similarity scores of all street segments in the map element database and select the Top-N (in this embodiment) Each result is assigned an initial matching probability score, forming a coarse matching candidate set for subsequent fine localization. The candidate set contains street segment IDs and their corresponding initial matching probabilities, where, This represents the identifier of the map feature corresponding to the nth coarse matching candidate set. This represents the matching score corresponding to the nth coarse matching candidate set.

[0071] Step 3: Combining inertial measurement data and road topology constraints, perform temporal smoothing and confidence level evaluation on the coarse matching candidate set, and output the geographic feature information that meets the set confidence threshold as the preferred dataset based on the evaluation results.

[0072] Step 3.1: Angular velocity based on airborne IMU acquisition With acceleration Calculate the drone from the previous moment up to the current moment The relative rotation matrix and displacement observation.

[0073] Calculate the relative rotation matrix using angular velocity integral :

[0074] ;

[0075] The influence of gravity g is eliminated based on the pre-integral theory, and the current time is calculated. Relative displacement observation :

[0076] ;

[0077] In the formula, For antisymmetric matrix operators, The time difference between two frames The velocity at the previous moment, For zero acceleration bias, j represents the time subscript.

[0078] Step 3.2: Introduce relative displacement observations at the continuous video frame level to construct short-time stable prior trajectory positions, and use the previous moment... The optimal posterior estimated location and the current moment Relative displacement observation Calculate the prior position at the current moment. :

[0079] ;

[0080] Step 3.3: Based on this prior location Construct spatial gating constraints in the form of Gaussian distributions, and eliminate coarse selection matching sets and Abnormal nodes whose Euclidean distance exceeds a set threshold (e.g., 50 meters) will be identified, and jumps caused by short-term occlusion or local interference will be constrained to small-amplitude disturbances.

[0081] Step 3.4: Taking into account the difference in cosine similarity scores in the coarse matching candidate set after removing abnormal nodes, the road structure constraint satisfaction, the consistency measure of continuous frames, and the relative motion consistency index, calculate the confidence value of geographic element information and classify it into levels.

[0082] The confidence level value The calculation is as follows:

[0083] ;

[0084] In the formula, and These are the cosine similarity scores of the first and second highest scores in the coarse matching candidate set, respectively. For topological constraints, As a time series continuity indicator, These are weighting coefficients;

[0085] Among them, the The formula used to evaluate the connectivity rationality of matching candidate road segments in two consecutive frames on the digital map is as follows:

[0086] ;

[0087] In the formula, The topological penalty coefficient is... This represents the shortest pathfinding distance between the two candidate road segments on the digital map topology.

[0088] The The aforementioned consecutive frame consistency metrics and relative motion consistency indices are comprehensively quantified to evaluate the consistency between visually matched displacement and inertial navigation-estimated displacement. The calculation formula is as follows:

[0089] ;

[0090] In the formula, This represents the actual geometric displacement of the candidate matching points on the map between the two consecutive frames. The standard deviation of displacement noise is set.

[0091] The specific rule for classifying confidence levels is as follows: geographic element information is divided into high confidence levels and low confidence levels based on confidence level values;

[0092] Set upper limit threshold With lower threshold ,like If it is judged as high confidence level, If it is judged as low confidence level, Then it is determined to be unusable data.

[0093] Step 3.5: Based on the evaluation results, output one or more geographic feature information that meet the set confidence threshold as the preferred dataset, and assign them the corresponding display priority or rendering strategy.

[0094] The preferred dataset includes geographic feature information data with both high and low confidence levels. For high-confidence results, the accurate overlay and highlighting of GIS features are directly driven; for low-confidence results, prompts and degradation strategies are activated, presented as candidate bounding boxes or multiple hypotheses to prevent misleading information; unusable data is not displayed.

[0095] Step 3.6: To address the abrupt changes in geographic feature detection results (such as label flickering) in consecutive video frames and ensure the stability of geographic feature information matching with video frames, a temporal smoothing algorithm is used to optimize the street attribution results. That is, the temporal smoothing algorithm is used to optimize the geographic feature information matched by the video frames, connecting the geographic feature information in consecutive video frames to find the path with the lowest cost.

[0096] Specifically, the Viterbi algorithm is used for temporal smoothing, and optimal path estimation constraints are applied to the street affiliation of consecutive video frames, by minimizing the cumulative cost function. Achieving temporal consistency constraints on street affiliation suppresses street skipping and drifting phenomena, improving the reliability of geographic feature matching. Specifically:

[0097] Define the state transition probability based on the Gaussian distribution assumption. :

[0098] ;

[0099] In the formula, The standard deviation of displacement noise, This represents the geometric distance between the centers of two street segments in the map's topology.

[0100] Construct a cumulative cost function that includes observation probability and transition probability. :

[0101] ;

[0102] In the formula, Scoring the cosine similarity. These are the weighting coefficients.

[0103] In this embodiment, the following settings are provided: , ,but The data is determined to be of high confidence level and is accurately overlaid and highlighted. If the data is determined to have low confidence, it will be displayed in a downgraded manner (indicated by a dashed box). The data is unavailable and will not be displayed.

[0104] Step 4: Map the geographic feature information from the selected dataset back to the video frames, overlay it onto the video screen, and then perform low-latency augmented reality rendering and dynamic coordinate correction to achieve real-time fusion of video and geographic information.

[0105] An asynchronous architecture is used to perform geographic information enhancement rendering on video frames, anchoring and displaying high-confidence target data, and providing range prompts for low-confidence areas, thereby achieving real-time fusion and presentation of video and geographic information and solving the problem of image-target separation under high-speed motion.

[0106] Step 4.1: Construct a parallel processing pipeline that separates perception, calculation, and rendering. Separate the video frame input, visual feature retrieval, fine positioning calculation, and AR rendering module into multiple threads. The visual algorithm thread (running frequency of about 15 Hz) is responsible for time-consuming calculation, and the AR rendering thread (running frequency of 60 Hz) is responsible for real-time display.

[0107] Step 4.2: Set up a circular data buffer between each module to store the IMU high-frequency attitude data sequence, balance the high frame rate acquisition of video frames with the computation time of the visual positioning algorithm, and ensure low latency operation of the system in the edge computing environment.

[0108] Step 4.3: Introduce temporal smoothing filtering and motion compensation matrices into the rendering pipeline. Utilize the continuous frame information within the buffer to perform reprojection correction on the GIS coordinates of geographic features in the overlaid GIS vector map, and calculate the screen coordinates. It also assigns geographic feature information tags to eliminate rendering delays.

[0109] Specifically, when the algorithm thread outputs the time... Location results At that time, the rendering thread was already in time. At this point, the IMU data in the buffer is used to calculate from... arrive Motion compensation matrix The superimposed GIS coordinates are reprojected and corrected to calculate the screen coordinates. And assign corresponding geographic element information labels.

[0110] Among them, the latest pose at the rendering time is used With the solution time pose Compensation is provided for differences to eliminate jumps in annotation position caused by sudden changes in UAV attitude, and screen coordinates are calculated. The calculation formula is as follows:

[0111] ;

[0112] In the formula, Here is the motion compensation matrix. For geographic element coordinates, This is the intrinsic parameter matrix of an industrial camera. and These are the rotation matrix and translation vector corresponding to the rendering time, respectively. The coordinates are the projected two-dimensional screen coordinates. This is the depth scaling factor for the projection transformation.

[0113] This step eliminates the lag in geographic feature information labels caused by algorithm delays, ensuring that virtual road name labels remain firmly attached to the real road surface even when the drone is maneuvering rapidly.

[0114] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

[0116] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A passive geographic information overlay system based on cross-modal features, characterized in that, include: Airborne visual acquisition module, edge computing processing module, and augmented reality display module; The airborne vision acquisition module is installed on the UAV platform and is used to acquire video frames and flight attitude data in real time. The edge computing processing module is used to receive video frames and perform geographic information calculation. Based on the visual Transformer network, it performs global semantic encoding on the input video frames to obtain visual features, maps the visual features to the vector map feature space, and outputs the retrieval results as candidate datasets. It combines the Viterbi algorithm to perform temporal smoothing, performs optimal path estimation constraints on the street affiliation of continuous video frames, achieves temporal consistency constraints on street affiliation by minimizing the cumulative cost function, calculates the confidence level, and outputs geographic feature information that meets the set confidence threshold as the preferred dataset. The augmented reality display module is used to overlay geographic feature information from the preferred dataset onto the video screen in an augmented reality manner; The preferred dataset includes geographic feature information that satisfies a set confidence threshold, specifically: Angular velocity based on airborne IMU With acceleration Calculate the drone from the previous moment up to the current moment relative rotation matrix With displacement observation ; Relative displacement observations are introduced at the level of continuous video frames to construct short-time stable prior trajectory positions, and the previous moment is used as the basis for these positions. The optimal posterior estimated location and the current moment Relative displacement observation Calculate the prior position at the current moment. ; based on Construct a spatial gating constraint in the form of a Gaussian distribution to eliminate abnormal candidate nodes in the coarse matching candidate set; Taking into account the difference in cosine similarity score in the coarse matching candidate set after removing abnormal nodes, the road structure constraint satisfaction, the continuous frame consistency measure, and the relative motion consistency index, the confidence value of geographic element information is calculated and classified. Based on the evaluation results, output one or more geographic feature information that meet the set confidence threshold as the preferred dataset, and assign corresponding display priority or rendering strategy to them; To address the abrupt changes in geographic feature detection results across consecutive video frames and ensure the stability of geographic feature information matching with video frames, a temporal smoothing algorithm is used to optimize street attribution results. Wherein, the confidence level value The calculation is as follows: ； In the formula, and These are the cosine similarity scores of the first and second highest scores in the coarse matching candidate set, respectively. For topological constraints, As a time series continuity indicator, These are weighting coefficients; Among them, the The formula used to evaluate the connectivity rationality of matching candidate road segments in two consecutive frames on the digital map is as follows: ； In the formula, The topological penalty coefficient is... This represents the shortest pathfinding distance between the two candidate road segments on the digital map topology. The The aforementioned consecutive frame consistency metrics and relative motion consistency indices are comprehensively quantified to evaluate the consistency between visually matched displacement and inertial navigation-estimated displacement. The calculation formula is as follows: ； In the formula, This represents the actual geometric displacement of the candidate matching points on the map between the two consecutive frames. The set standard deviation for displacement noise; The confidence level classification rule is as follows: Set upper limit threshold With lower threshold ,like If it is judged as high confidence level, If it is judged as low confidence level, Then it is determined to be unusable data.

2. The passive geographic information overlay system based on cross-modal features according to claim 1, characterized in that, The airborne vision acquisition module includes an industrial camera, a six-axis inertial measurement unit (IMU), and a time synchronization unit. The industrial camera is used to acquire continuous video frames containing street and ground features under GNSS rejection conditions; it is connected to the edge computing processing module through a data interface to transmit the video stream. The six-axis inertial measurement unit is used to collect the angular velocity and acceleration information of the UAV body; The time synchronization unit is used to perform hardware-level alignment of video frames and inertial data.

3. The passive geographic information overlay system based on cross-modal features according to claim 2, characterized in that, The edge computing processing module adopts a high-performance embedded AI computing platform, including a local map sample construction unit, a cross-modal feature retrieval unit, and a localization and confidence evaluation unit; The local map sample construction unit is used to store mapping data from the global vector map to local image features; The cross-modal feature retrieval unit, based on a visual Transformer network, performs global semantic encoding on the input video frames to obtain visual features, maps the visual features to the vector map feature space, and outputs retrieval results as a candidate dataset. The retrieval results include candidate street segment identifiers and key element coordinates. The location and confidence assessment unit combines road topology constraints and temporal smoothing strategies to locate the search results, calculate the confidence level of the output results, and filter the search results based on the confidence level. It then selects the geographic feature information that meets the set confidence threshold as the preferred dataset and assigns it the corresponding display priority or rendering strategy.

4. The passive geographic information overlay system based on cross-modal features according to claim 3, characterized in that, The augmented reality display module runs on a ground control station or an airborne head-mounted display. It adopts a multi-threaded asynchronous rendering architecture, balances the difference between video acquisition and algorithm processing rates through a circular buffer, and drives the highlighting or downgrading of geographic features based on the confidence level.

5. The passive geographic information overlay system based on cross-modal features according to claim 4, characterized in that, Calculate the relative rotation matrix using angular velocity integral ,include: ； The influence of gravity g is eliminated based on the pre-integral theory, and the current time is calculated. Relative displacement observation ,include: ； In the formula, For antisymmetric matrix operators, The time difference between two frames The velocity at the previous moment, For zero acceleration bias, j represents the time subscript.

6. A passive geographic information overlay method based on cross-modal features, characterized in that, The passive geographic information overlay system based on cross-modal features as described in claim 1 includes the following steps: Step 1: Construct a retrieval database of UAV video imagery and vector map elements; Step 2: During the flight of the UAV, in a GNSS denied environment, the visual features are obtained by global semantic encoding of the input video frames based on the visual Transformer network, and the visual features are mapped to the vector map feature space. The retrieval results are then output as a coarse matching candidate set. Step 3: Combine the Viterbi algorithm for temporal smoothing, apply optimal path estimation constraints to the street affiliation of consecutive video frames, achieve temporal consistency constraints for street affiliation by minimizing the cumulative cost function, and combine inertial measurement data and road topology constraints to perform temporal smoothing and confidence level evaluation on the coarse matching candidate set. Based on the evaluation results, output the geographic feature information that meets the set confidence threshold as the preferred dataset. Step 4: Map the geographic feature information from the selected dataset back to the video frames, overlay it onto the video screen, and then perform low-latency augmented reality rendering and dynamic coordinate correction to achieve real-time fusion of video and geographic information.

7. The passive geographic information overlay method based on cross-modal features according to claim 6, characterized in that, In step 1, by calibrating the intrinsic and extrinsic parameters of the industrial camera, the geographic elements in the GIS vector map are projected from geographic coordinates to the local map coordinate system, a topological structure map of the road segment and intersection connectivity is established, and a globally unique ID is assigned to each street segment to form a map element library.

8. The passive geographic information overlay method based on cross-modal features according to claim 7, characterized in that, In step 2, during the UAV's flight, depth visual features of real-time video frames are extracted under GNSS-denied conditions and used as global semantic vectors. These vectors are then matched with vector map feature vectors in the map feature library to output a coarse matching candidate set. This process includes the following methods: Step 2.1: Extract the current time step using the pre-trained visual Transformer backbone network. single-frame video global feature vector : ； In the formula, It is a visual encoding network pre-trained with a large amount of data; Step 2.2: Calculate the cosine similarity score between the global feature vector and the map feature representation. : ； In the formula, Indicates the first Map features representing candidate street segments, The norm of a vector; Step 2.3: Sort the cosine similarity scores, select the Top-N results, assign them corresponding initial matching probability scores, and form a coarse matching candidate set. ,in, This represents the identifier of the map feature corresponding to the nth coarse matching candidate set. This represents the matching score corresponding to the nth coarse matching candidate set.

9. The passive geographic information overlay method based on cross-modal features according to claim 8, characterized in that, In step 4, the geographic element information in the selected dataset is mapped back to the video frame, superimposed on the video image, and then subjected to low-latency augmented reality rendering and dynamic coordinate correction to achieve real-time fusion presentation of video and geographic information. Specifically, this includes the following methods: Step 4.1: Construct a parallel processing pipeline that separates perception, computation, and rendering; Step 4.2: Set up a circular data buffer between each module to store the IMU high-frequency attitude data sequence, thereby balancing the high frame rate acquisition of video frames with the computation time of the visual positioning algorithm; Step 4.3: Introduce temporal smoothing filtering and motion compensation matrices into the rendering pipeline. Utilize the continuous frame information within the buffer to perform reprojection correction on the GIS coordinates of geographic features in the overlaid GIS vector map, and calculate the screen coordinates. It is also assigned a geographic element information label.

Images