Unmanned aerial vehicle end-side navigation method and system based on multi-source semantic fusion

Abstract

The application provides a UAV end-side navigation method and system based on multi-source semantic fusion. The method comprises the following steps: constructing a reference unit by collecting aerial photography, satellite map and map data unified coordinates; generating structured semantic text by using a visual large language model and constructing a labeling library. During training, the visual and text encoders are combined, geographic semantics are injected into the cross-view matching process to train the navigation model, and multi-level indexes are constructed for end-side simulation verification. This method significantly improves the robustness, semantic interpretability and end-side deployment reliability of navigation in complex environments such as GPS restrictions.

Description

Technical Field

[0001] This invention belongs to the field of UAV edge-side artificial intelligence deployment, and particularly relates to UAV edge-side navigation methods and systems based on multi-source semantic fusion. Background Technology

[0002] In inspection, surveying, emergency rescue, low-altitude logistics, urban security, and autonomous operations in complex environments, unmanned aerial vehicles (UAVs) typically need to continuously acquire their own position, heading, and local environmental status. Traditional navigation solutions mainly rely on satellite navigation signals such as GPS and BeiDou, as well as inertial measurement units (IMUs). However, in urban canyons, indoor-outdoor transition areas, under bridges, valleys, forests, environments with strong electromagnetic interference, or environments with active denial of navigation, satellite navigation signals are prone to obstruction, drift, interruption, or spoofing, making it difficult for UAVs to stably complete autonomous positioning and navigation.

[0003] To overcome the aforementioned problems, existing research proposes to achieve visual geolocation by utilizing cross-view matching relationships between UAV aerial images and satellite images, orthophotos, or map tiles. These methods typically use oblique UAV views as query images and satellite images or map tiles as a reference library. They extract visual features and calculate similarity using deep networks to obtain candidate geographic locations.

[0004] However, existing purely visual cross-view matching methods still have significant shortcomings. On the one hand, UAV images and satellite images differ significantly in terms of viewpoint, scale, resolution, imaging time, illumination, occlusion, and seasonal variations, making it difficult to achieve stable matching by relying solely on visual texture. On the other hand, in scenarios such as campuses, industrial parks, residential areas, warehouse areas, and gridded road areas, different locations may have similar building roofs, similar road shapes, and similar parking areas, which can easily lead to candidate confusion.

[0005] Map data, road vectors, building outlines, points of interest, terrain, and airborne sensor status can provide additional constraints for UAV localization, but existing methods often only use them as auxiliary layers or post-processing conditions, lacking a unified multi-source semantic fusion modeling mechanism. In particular, high-level semantic information such as road grade, building density, regional function, feature combination, relative spatial relationships, and traffic risks are usually implicit in aeronautical charts or map data and have not been explicitly converted into learnable, screenable, and reusable semantic text.

[0006] In recent years, large language models and visual language models have demonstrated strong capabilities in image and text understanding, scene description, and structured information generation, enabling the conversion of remote sensing images, map tiles, or geographic features into natural language or field-based semantic descriptions. However, directly using large models to generate text can easily lead to problems such as inconsistent description formats, semantic redundancy, unclear confidence levels, inconsistencies with map facts, or illusory information, which in turn affect subsequent model training and navigation inference.

[0007] Furthermore, most existing cross-view geolocation and multimodal navigation models remain at the offline dataset evaluation stage, typically only providing retrieval metrics such as recall and average precision, lacking simulation verification for actual UAV navigation tasks. Specifically, existing models often lack a complete process for verifying navigation output in a unified simulation environment, and also lack systematic testing of the usability of the deployed model on the edge. Due to the lack of such simulation verification mechanism, even if the model achieves high retrieval accuracy on public datasets, it is difficult to prove that it can meet the edge navigation and positioning requirements of UAVs in GPS / GNSS-restricted environments. Therefore, it is necessary to propose a UAV navigation method and simulation verification system based on multi-source semantic information fusion to achieve a complete technical closed loop from semantic sample construction, fusion model training, edge deployment to simulation testing and evaluation. Summary of the Invention

[0008] The purpose of this invention is to propose an edge navigation method and system for unmanned aerial vehicles (UAVs) based on multi-source semantic fusion, so as to solve the above-mentioned problems.

[0009] To achieve the above objectives, a first aspect of the present invention provides an unmanned aerial vehicle (UAV) edge navigation method based on multi-source semantic fusion, the method comprising: S1. Acquire UAV aerial images, satellite remote sensing images and open map data within the target area, and perform coordinate unification, scale unification and spatial alignment processing on the UAV aerial images, satellite remote sensing images and open map data within the same geographical area to construct reference units; S2. Construct a prompt word template based on the reference unit, and use the prompt word template to constrain the visual big language model to generate structured semantic text. The structured semantic text includes fields such as road structure, building distribution, typical features, regional functions, spatial relationships, and traffic risk. The structured semantic text is set to perform field-level confidence verification and map feature consistency verification based on field-level evidence chain to filter and mark the generated structured semantic text, thereby obtaining structured semantic text that meets the preset verification conditions. S3. Perform normalization processing and deduplication of similar texts on the verified structured semantic text to obtain the deduplicated structured semantic text; S4. Bind the deduplicated structured semantic text with the corresponding UAV aerial images, satellite remote sensing images, open map data, and geographic location tags to construct a geographic semantic annotation sample library. S5. Train a multi-source semantic information fusion navigation model based on the geographic semantic annotation sample library, and introduce text semantic information in the visual encoding process to enhance the expression of image features; S6. The performance of the trained multi-source semantic information fusion navigation model is evaluated using a test set. S7. Deploy the trained multi-source semantic information fusion navigation model to the edge device, and perform inference verification and closed-loop re-retrieval in conjunction with the indexing mechanism.

[0010] In a preferred embodiment, this application can be further configured such that the reference unit is a spatial unit obtained by dividing a geographic grid based on a fixed scale, and each reference unit corresponds to a unique geographic identifier and geographic location label.

[0011] In a preferred embodiment, this application can be further configured such that the prompt word template is used to constrain the visual large language model to perform semantic annotation of UAV navigation scene based on satellite remote sensing images and open map tiles, and output structured semantic text according to a preset format.

[0012] In a preferred embodiment, this application can be further configured such that the field-level evidence chain is specifically set as follows: Each semantic field corresponds to an independent field-level evidence chain, which includes at least the field description content, the corresponding open map element, the corresponding image region, the map element existence verification result, the spatial relationship consistency verification result, and the field processing status.

[0013] In a preferred example, this application can be further configured such that the rule for the field processing status is: Fields whose description objects exist and whose spatial relationships are consistent are marked as valid fields; fields whose description objects exist but whose spatial relationships cannot be confirmed are marked as fields to be reviewed; fields whose description objects do not exist or whose spatial relationships are opposite to those of the open map tiles are marked as invalid fields.

[0014] In a preferred embodiment, this application can be further configured such that the field-level evidence chain is bound to the reference unit number in the data structure, and each evidence chain records at least the field type, field description content, candidate open map elements, candidate image areas, map element existence verification results, spatial relationship consistency verification results, field processing status, and subsequent processing permissions. The subsequent processing permissions include whether to allow access to the training sample library, whether to allow the generation of pseudo-visual semantic tags, whether to allow access to the edge semantic index, and whether a second review is required.

[0015] In a preferred embodiment, this application can be further configured such that the normalization process specifically includes: The semantic fields are uniformly formatted, and similar text is deduplicated by combining keyword matching and semantic feature similarity judgment; In the process of deduplicating similar texts, the degree of similarity between structured semantic texts is first determined based on keyword matching and semantic feature similarity. Then, the similar text is judged as redundant text or semantic nearest neighbor reference unit based on whether the geographical range overlaps. The semantic nearest neighbor reference unit is constructed based on field-level evidence chain. If the similarity is greater than the preset semantic nearest neighbor threshold and the geographical ranges of the two reference units do not overlap, then neither text will be deleted. Instead, both will be marked as semantic nearest neighbor reference units and used as hard negative samples. If the two belong to the same geographical grid and their geographical ranges overlap, then the redundant text retention rule will be applied.

[0016] The semantic nearest neighbor reference unit is constructed based on a field-level evidence chain, specifically as follows: For two reference units that are semantically similar but do not overlap geographically, not only is the degree of similarity between the two reference units recorded, but also the effective field types that lead to the similarity and the source of their evidence chains.

[0017] In a preferred embodiment, this application can be further configured as follows:

[0018] The multi-source semantic information fusion navigation model includes a UAV image visual coding branch, a satellite image visual coding branch, a text semantic coding branch, and a cross-modal fusion positioning branch; The UAV image visual coding branch is used to extract features from UAV aerial images, the satellite image visual coding branch is used to extract features from satellite remote sensing images, the text semantic coding branch is used to extract structured semantic text features, and the cross-modal fusion localization branch is used to map UAV image features, satellite image features, and text semantic features to a unified feature space. Before visual encoding, the multi-source semantic information fusion navigation model inputs the structured semantic text obtained after map element existence verification and spatial relationship consistency verification into the text semantic encoding module to obtain field-level text semantic features. The field-level text semantic features are then input into the inverse mapping network to generate pseudo-visual semantic tags. The pseudo-visual semantic tags have the same dimension as the image patch features and are used to combine with the image features in the visual encoder.

[0019] In a preferred example, this application can be further configured such that the combination of indexing mechanism to perform inference verification and closed-loop re-retrieval specifically includes: The system performs joint retrieval based on vector index, semantic index, and position index, and triggers expanded retrieval, secondary sorting, or position correction operations according to visual matching stability, semantic field consistency, and position continuity, respectively, to achieve closed-loop re-retrieval.

[0020] A second aspect of the invention provides an unmanned aerial vehicle (UAV) end-side navigation system based on multi-source semantic fusion, the system comprising: The multi-source data layer is used to acquire UAV aerial images, satellite remote sensing images and open map data within the target area. It performs coordinate unification, scale unification and spatial alignment processing on the UAV aerial images, satellite remote sensing images and open map data within the same geographical area to construct reference units. A geographic reference layer is used to construct prompt word templates based on the reference units, and to use the prompt word templates to constrain the visual big language model to generate structured semantic text. The structured semantic text includes fields such as road structure, building distribution, typical features, regional functions, spatial relationships, and traffic risk. The structured semantic text is set to perform field-level confidence verification and map feature consistency verification based on field-level evidence chains, so as to filter and mark the generated structured semantic text to obtain structured semantic text that meets the preset verification conditions. The semantic annotation module is used to normalize the verified structured semantic text and remove duplicates from similar texts to obtain deduplicated structured semantic text. The text screening module is used to bind the deduplicated structured semantic text with the corresponding UAV aerial images, satellite remote sensing images, open map data, and geographic location tags to construct a geographic semantic annotation sample library. The fusion training module is used to train a multi-source semantic information fusion navigation model based on the geographic semantic annotation sample library, and to introduce text semantic information in the visual encoding process to enhance the expression of image features. The model evaluation module is used to evaluate the performance of the trained multi-source semantic information fusion navigation model using a test set. The edge deployment module is used to deploy the trained multi-source semantic information fusion navigation model to edge devices, and to perform inference verification and closed-loop re-retrieval in conjunction with the indexing mechanism.

[0021] The beneficial technical effects of the present invention are at least as follows: First, this invention transforms road structures, building distributions, typical features, regional functions, and spatial relationships in satellite remote sensing images and open map tiles into structured semantic text, enabling the geographic semantic information originally implicit in aeronautical maps to participate in model training and location matching.

[0022] Second, this invention constrains the output format of the visual large language model and prohibits fiction rules through prompt word templates, and reduces the interference of redundant semantic descriptions on training samples by deduplicating similar texts, thereby improving the quality of the semantic annotation sample library.

[0023] Third, the present invention adopts a text semantic reverse injection method to convert text semantic features into pseudo visual semantic tags and participate in visual encoding, so that text semantics is not just post-processing information, but semantic enhancement of the matching process of UAV aerial images and satellite remote sensing images in the visual feature extraction stage.

[0024] Fourth, this invention not only performs offline model testing, but also deploys the model to the NVIDIA JetsonThor edge computing platform. It combines the TensorRT inference engine, vector index, semantic index, and location index to complete edge simulation verification, which can evaluate the model's positioning accuracy, retrieval performance, and deployment availability in the edge environment.

[0025] Fifth, this invention connects training sample admission, pseudo-visual semantic tag generation, and edge-side closed-loop re-retrieval through a single field-level evidence chain. This ensures that the field screening results no longer remain at the text preprocessing stage but continuously influence the model training input, semantic injection during the visual encoding stage, and the verification of edge-side candidate reference units. This collaborative relationship can simultaneously reduce the risk of large model illusion semantics entering the training sample library, reduce the risk of cross-view mismatches caused by similar ground features, and reduce the risk of the inability to self-recover after spatial jumps in edge-side continuous localization. Attached Figure Description

[0026] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0027] Figure 1 This is a flowchart of the UAV edge navigation method based on multi-source semantic fusion according to an embodiment of the present invention. Detailed Implementation

[0028] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0029] In this embodiment, to ensure that the formula and algorithm can be directly reproduced, the following complete steps are performed: First, read UAV aerial images, satellite remote sensing images, and open map tiles, and unify data within the same geographical area to the same coordinate system and the same reference unit number; second, for each reference unit, call the visual large language model to generate original semantic text, and perform map element existence verification and spatial relationship consistency verification field by field; third, calculate keyword set similarity, text semantic vector cosine similarity, comprehensive text similarity, and field completeness for the structured semantic text formed by effective fields. Highly similar text within the same geographical area is retained as redundancy, and highly similar text within different geographical areas is selected as semantic nearest neighbor reference units. The system first sets up a difficult-to-handle sample set. Then, it inputs the training samples into a multi-source semantic information fusion navigation model to generate pseudo-visual semantic tags for effective field text features. These pseudo-visual semantic tags are then injected into UAV image patch sequences and satellite image patch sequences. After that, the trained model is used to retrieve and rank UAV images on the test set. The model performance is evaluated based on geographical distance error, positioning accuracy, first-response rate, average precision, and average precision of the test set. Finally, inference is performed on the edge platform. Based on visual matching differences, semantic field consistency, and positional continuity, the system triggers expanded retrieval, semantic secondary ranking, or positional continuity correction, respectively. This forms a complete and computable process from field screening, sample construction, model training, model evaluation to edge closed-loop re-retrieval.

[0030] In this embodiment, the three-stage reuse process of field-level evidence chains is as follows: First, after semantic text generation, an evidence chain is established for each semantic field, and the field processing status is determined. Second, during sample library construction and model training, only valid fields and their evidence chains are written into the training samples, and pseudo-visual semantic tags with evidence chain pointers are generated based on the valid fields. Fields to be reviewed and invalid fields do not generate training pseudo-visual semantic tags. Third, during edge-side inference, the valid field evidence chain summary of candidate reference units is checked for consistency with the road structure, building distribution, and typical features identified in the current UAV image. Based on the check results, expanded retrieval, semantic secondary sorting, or positional continuity correction are triggered. The above three stages use the same field evidence chain object, thus forming a reproducible closed loop of credible field screening, field-level pseudo-visual tag injection, and edge-side conditional re-retrieval.

[0031] like Figure 1 As shown in the embodiment of the present invention, the UAV end-side navigation method based on multi-source semantic fusion includes: S1. Acquire UAV aerial images, satellite remote sensing images, and open map data within the target area. Perform coordinate unification, scale unification, and spatial alignment processing on the UAV aerial images, satellite remote sensing images, and open map data within the same geographical area to construct reference units.

[0032] Specifically, the process begins by acquiring drone aerial images, satellite remote sensing images, and open map data for the target area. The drone aerial images are oblique views captured by the drone at different altitudes, perspectives, and flight paths; the satellite remote sensing images are overhead remote sensing images covering the target area; and the open map data includes map tiles, road vectors, building outlines, and points of interest data corresponding to the target area.

[0033] The target area is divided into fixed-scale geographic grids, each grid corresponding to a unique number and geographic location label. Satellite remote sensing images are cropped according to these geographic grids to obtain satellite remote sensing slices; open map data is cropped according to the same geographic grid to obtain open map tiles. Based on the initial positioning information, flight altitude, and attitude angle corresponding to the time of acquisition of UAV aerial images, the UAV aerial images are associated with the corresponding geographic grids. Thus, UAV aerial images, satellite remote sensing images, open map tiles, and geographic location labels within the same geographic area are organized into reference units.

[0034] For the A reference unit can be represented as: ; in, Indicates the first The drone aerial images corresponding to each reference unit Represents satellite remote sensing images, Indicates open map tiles, This indicates the geographic location label of the reference unit.

[0035] S2. Construct a prompt word template based on the reference unit, and use the prompt word template to constrain the visual large language model to generate structured semantic text. The structured semantic text includes fields such as road structure, building distribution, typical features, regional functions, spatial relationships, and traffic risk. The structured semantic text is set to perform field-level confidence verification and map feature consistency verification based on field-level evidence chain to filter and mark the generated structured semantic text, thereby obtaining structured semantic text that meets the preset verification conditions.

[0036] Specifically, for each reference unit, a cue word template is constructed for the visual large language model. This cue word template constrains the visual large language model to perform semantic annotation of UAV navigation scenes based on satellite remote sensing images and open map tiles, and outputs structured semantic text according to a preset format. The cue word template clearly defines the input data source, semantic recognition task, output field format, and rules prohibiting fictitious information generation, ensuring that the text generated by the model includes fields such as road structure, building distribution, typical features, regional functions, and spatial relationships, while prohibiting the generation of geographical information not present in the input images or map tiles.

[0037] For the The process of generating original semantic text using a reference unit and a visual large language model can be represented as follows: ; in, Indicates the first The original semantic text of each reference unit, This represents the generation function of the visual large language model. Represents satellite remote sensing images, Indicates open map tiles, Indicates a prompt word template.

[0038] The prompt word template can require the visual large language model to output the following fields: Road structure field to describe the form of main roads, branch roads, intersections and road directions; Building distribution field to describe building density, building arrangement and building group form; Typical features field to describe identifiable features such as water bodies, playgrounds, squares, and large buildings; Regional function field to describe regional attributes such as residential areas, industrial areas, campus areas, and commercial areas; Spatial relationship field to describe the relative orientation relationship between roads, buildings and typical features.

[0039] Furthermore, in step S2, each semantic field corresponds to an independent field-level evidence chain. This field-level evidence chain includes at least the field description, the corresponding open map element, the corresponding image region, the map element existence verification result, the spatial relationship consistency verification result, and the field processing status. The initial source for determining the field status is the Boolean decision rule in mathematical logic and the rule-based reasoning method in expert systems.

[0040] The modification of this application based on such rules is that instead of treating the entire semantic text output by the visual big language model as either usable or unusable as a whole, it breaks down road structure, building distribution, typical features, regional functions, spatial relationships, and traffic risks into fields, and performs map feature existence verification and spatial relationship consistency verification separately.

[0041] The field processing status is determined according to the following rules: Fields whose descriptive objects exist and have consistent spatial relationships are marked as valid fields; fields whose descriptive objects exist but whose spatial relationships cannot be confirmed are marked as fields awaiting review; fields whose descriptive objects do not exist or whose spatial relationships are opposite to those of the open map tiles are marked as invalid fields. Valid fields enter the text semantic encoding and pseudo-visual semantic tag generation process; fields awaiting review enter the secondary inference or manual verification process; invalid fields are removed and prohibited from participating in model training. The above rules pertain to discrete state determination and do not involve physical quantities such as length, time, or velocity; therefore, there is no issue of inconsistent physical dimensions.

[0042] As a calculation example, if the road structure field describes "an east-west main road and a north-south branch road forming a crossroads", and the open map tiles do indeed contain an east-west main road and a north-south branch road that are connected, then the map element existence verification passes, the spatial relationship consistency verification passes, and the field is determined to be a valid field. If the building distribution field describes the building outline, but the map tiles cannot confirm the orientation relationship of "the building is located on the west side of the road", then the map element existence verification passes, but the spatial relationship cannot be confirmed, and the field is determined to be a field to be reviewed. If the text description contains water features, but the open map tiles do not contain water features, then the map element existence verification fails, and the field is determined to be an invalid field.

[0043] Furthermore, the field-level evidence chain is bound to the reference unit number in the data structure. Each evidence chain records at least the field type, field description content, candidate open map elements, candidate image areas, map element existence verification results, spatial relationship consistency verification results, field processing status, and subsequent processing permissions.

[0044] The subsequent processing permissions include whether access to the training sample library is allowed, whether the generation of pseudo-visual semantic tags is allowed, whether access to the edge semantic index is allowed, and whether a secondary review is required. Through these permission fields, the system can directly pass the screening results of semantic illusions in the large model to the subsequent model training and edge review stages, instead of making a one-time judgment only at the text generation stage.

[0045] S3. Standardize the verified structured semantic text and remove duplicates from similar texts to obtain the deduplicated structured semantic text.

[0046] Specifically, since the original semantic text generated by the visual big language model may have problems such as inconsistent expression, semantic redundancy and repetition of similar texts, this embodiment performs normalization processing on the original semantic text and deduplication of similar texts.

[0047] First, the fields of road structure, building distribution, typical features, regional functions, and spatial relationships in the original semantic text are unified into a preset format.

[0048] Then, the first The original semantic text is represented as , will the The original semantic text is represented as Extract text keyword sets respectively and And obtain the corresponding text semantic vector through the text semantic encoder. and The overall similarity between two semantic texts Calculate using the following formula: ; in, and They represent the first Article and Section The text semantic vector of the original semantic text. and They represent the first Article and Section The set of keywords in the original semantic text. These are the weighting coefficients, and .

[0049] when Greater than or equal to the preset deduplication threshold At that time, the judgment and This is repeated semantic text.

[0050] For text with repetitive semantics, retain the one with higher field completeness. Field completeness Calculate using the following formula: ; in, Indicates the first Field completeness of semantic text. Indicates the total number of preset fields. Indicates the first The first semantic text One field, This is an indicator function. When two semantic text fields have the same completeness, the one with the longer text length is retained. Through the above processing, the deduplicated structured semantic text is obtained.

[0051] Furthermore, in step S3, the initial sources of the original comprehensive similarity formula are the Jaccard keyword set similarity from the information retrieval domain and the cosine similarity from the vector space model. Jaccard similarity measures the overlap ratio of two keyword sets, while cosine similarity measures the similarity of the angle between the semantic vectors of two texts. The modification in this application is that the semantic similarity is first obtained according to the above formula, and then the similarity is determined by whether the geographical scope overlaps to identify redundant text or semantically nearest neighbor reference units. This approach avoids mistakenly deleting semantically similar but geographically different regions.

[0052] As a computational example, let the keyword sets of the two semantic texts be {main road, intersection, rectangular building, playground, parking lot} and {main road, intersection, rectangular building, green space, parking lot}, respectively. The number of keyword intersections is 4, the number of unions is 6, and the keyword set similarity is 4 divided by 6, which is 0.6667. Let the two semantic text vectors be (0.6, 0.3, 0.7, 0.2) and (0.5, 0.4, 0.6, 0.3), respectively. Their dot product is 0.90, and their magnitudes are 0.9899 and 0.9274, respectively. The cosine similarity is 0.90 divided by the product of their magnitudes, which is 0.9803. Taking the keyword similarity weight as 0.4 and the vector similarity weight as 0.6, the overall text similarity is 0.4 multiplied by 0.6667 plus 0.6 multiplied by 0.9803, resulting in 0.8549. If the preset semantic nearest neighbor threshold is 0.85 and the geographical ranges of the two reference units do not overlap, then neither text will be deleted. Instead, both will be marked as semantic nearest neighbor reference units and used as hard-to-bear samples. If the two belong to the same geographical grid and their geographical ranges overlap, then the redundant text retention rule will be applied.

[0053] The initial source of the field completeness formula is the normalized count ratio in statistics. If the preset fields include six items: road structure, building distribution, typical features, regional function, spatial relationship, and traffic risk, and a text has five non-empty items that pass the validation, then the field completeness is 5 divided by 6, which is 0.8333; another text has four non-empty items that pass the validation, then the field completeness is 4 divided by 6, which is 0.6667. If both texts belong to the same geographic grid and their semantic similarity reaches the redundancy threshold, the text with a field completeness of 0.8333 is preferentially retained.

[0054] Furthermore, the construction of semantically nearest neighbor reference units also reuses field-level evidence chains. For two reference units that are semantically similar but whose geographical scopes do not overlap, the system not only records their textual similarity but also records the effective field types that lead to the similarity and the sources of their evidence chains. For example, when two different areas both have intersections, rectangular building clusters, and parking areas, the system uses these common fields as the reasons for the similarity of hard-to-bear samples and, during training, makes the model focus on distinguishing the differences between the two in terms of road connectivity, building arrangement, or the relationship between adjacent features. This processing ensures that hard-to-bear samples are not only triggered by textual similarity thresholds but are supported by traceable and effective field evidence chains.

[0055] S4. Bind the deduplicated structured semantic text with the corresponding UAV aerial images, satellite remote sensing images, open map data, and geographic location tags to construct a geographic semantic annotation sample library.

[0056] Specifically, the deduplicated structured semantic text is bound to corresponding drone aerial images, satellite remote sensing images, open map tiles, and geographic location tags to construct a geographic semantic annotation sample library. Each sample includes at least a drone aerial image, a satellite remote sensing image, an open map tile, structured semantic text, and a geographic location tag.

[0057] To avoid regional leakage between the training and test sets, the sample library is divided according to the principle of non-overlapping geographical regions. Specifically, the target region is divided into several non-overlapping sub-regions according to a geographical grid or administrative / natural regions. A portion of these sub-regions are used as the training set, a portion as the validation set, and the remaining sub-regions as the test set. This division method allows for a more realistic evaluation of the model's generalization ability in unknown regions.

[0058] S5. Train a multi-source semantic information fusion navigation model based on the geographic semantic annotation sample library, and introduce textual semantic information in the visual encoding process to enhance the expression of image features.

[0059] Specifically, the model includes a UAV image visual coding branch, a satellite image visual coding branch, a text semantic coding branch, and a cross-modal fusion localization branch. The UAV image visual coding branch is used to extract features from UAV aerial images, the satellite image visual coding branch is used to extract features from satellite remote sensing images, the text semantic coding branch is used to extract structured semantic text features, and the cross-modal fusion localization branch is used to map UAV image features, satellite image features, and text semantic features to a unified feature space.

[0060] Unlike simply concatenating textual and visual features, this embodiment employs semantic back-injection, allowing textual semantic information to participate in feature extraction during the visual encoding stage. Specifically, for the first... Each reference unit is used to structure its semantic text. Input text semantic encoder This yields a sequence of field-level text semantic features: ; in, Represents a sequence of field-level text semantic features. This indicates the number of pseudo-visual semantic tags, with a value of [value]. Up to 12, Represents the semantic feature dimension of the text. Indicates the first The text semantic features corresponding to each structured semantic field This represents the text semantic features corresponding to the first structured semantic field. This represents the text semantic features corresponding to the second structured semantic field, where the structured semantic fields include at least one of road structure, building distribution, typical features, regional function, spatial relationships, and traffic risk. Optionally, the field-level text semantic feature sequence is averaged and pooled to obtain the global text semantic features. ; in, This indicates the average pooling operation. Indicates the first Global textual semantic features corresponding to each reference unit.

[0061] To enable text semantics to interact with visual image patch features, field-level text semantic features are... Input reverse mapping network Generate pseudo-visual semantic tags: ; ; in, This represents a sequence of pseudo-visual semantic tags obtained by inverse mapping of textual semantic features. Represents the feature dimension of a visual image patch. Indicates the first A pseudo-visual semantic tag Indicates the first A pseudo-visual semantic tag Indicates the first A pseudo-visual semantic label. The inverse mapping network can be composed of a multi-layer perceptron, one implementation of which is: ; in, , , and For learnable parameters, Indicates the hidden layer dimension. This represents a non-linear activation function.

[0062] Through the above reverse mapping process, each field-level text semantic feature is converted into a pseudo-visual semantic tag with the same dimension as the visual image block feature, so that structured semantic information can be injected into the subsequent visual encoding process in the form of visual tags.

[0063] Aerial images taken by drones Image patch embedding is performed to obtain the UAV image patch feature sequence. : ; in, , Indicates the number of image patches from the drone. For satellite remote sensing images... Image patch embedding is performed to obtain satellite image patch feature sequences. : ; in, () indicates the satellite remote sensing image embedding module, which first embeds the satellite remote sensing image Divided according to the preset image block size Each image patch is then flattened and linearly mapped to a dimension C_V, resulting in the final satellite image patch feature sequence. ; Furthermore, pseudo-visual semantic tags These are respectively concatenated to the front end of the UAV image patch feature sequence and the satellite image patch feature sequence to form the semantically enhanced UAV input sequence and the semantically enhanced satellite input sequence: ; ; in, and These represent the category labels in the visual coding branches for UAV images and satellite images, respectively. This indicates a feature splicing operation.

[0064] Furthermore, semantically enhanced drone input sequences Input UAV image visual encoder Obtain semantically enhanced visual features of the drone : ; semantically enhanced satellite input sequence Input satellite image visual encoder Obtain satellite semantically enhanced visual features : ; in, and The output features consist of visual category tags, pseudo-visual semantic tags, and image patch tags, which are then interacted with by the visual encoder. Since the pseudo-visual semantic tags participate in self-attention computation together with the image patch tags in the visual encoder, the textual semantic information can interact with local image details during the visual feature extraction stage, thereby enhancing the guiding role of semantic cues such as road structure, building distribution, typical features, and spatial relationships in cross-view matching.

[0065] Furthermore, from and The class label output and pseudo-visual semantic label output are extracted separately, and then passed through a projection layer to obtain the UAV-side fused features in a unified feature space. Satellite-side fusion features : ; ; in, and These represent the category label outputs from the drone and satellite ends, respectively. and These represent the pseudo-visual semantic tagging outputs from the UAV and satellite ends, respectively. This indicates the average pooling operation. and This represents the feature projection layer.

[0066] In a unified feature space, compute the fused features from the UAV end. Satellite-based fusion features Similarity between them: ; when hour, and Those corresponding to the same geographic reference unit should be considered positive sample pairs; when When the two correspond to different geographic reference units, they should be considered as negative sample pairs.

[0067] Furthermore, during model training, a cross-view contrastive loss constraint is used to ensure that positive sample pairs are close to each other and negative sample pairs are far apart. For a batch of... One sample, the matching loss from the drone to the satellite. It can be represented as: ; Matching loss from satellite to drone direction It can be represented as: ; in, Here, j represents the temperature coefficient, and j represents the index of the UAV image feature sample in the current batch.

[0068] The overall training loss L of the multi-source semantic information fusion navigation model can be expressed as: ; Through the above training process, the UAV aerial images and satellite remote sensing images corresponding to the same reference unit are made close to each other in a unified feature space, while the features corresponding to different reference units are far apart, thus obtaining a fusion positioning and navigation model.

[0069] Furthermore, in step S5, the initial source of the pseudo-visual semantic tag back-injection is the affine mapping in the neural network, the nonlinear transformation of the multilayer perceptron, and the tag sequence input mechanism in the Transformer visual encoder. The traditional Transformer visual encoder receives image patch tag sequences; the modification in this application is that the effective field text features selected through field-level evidence chains are transformed into the visual feature space via an inverse mapping network to form pseudo-visual semantic tags consistent with the feature dimensions of the image patches. These pseudo-visual semantic tags are then used as part of the visual encoder's input sequence for self-attention computation. Pseudo-visual semantic tags are not generated for fields to be reviewed or invalid fields.

[0070] As a computational example, suppose the textual semantic features of a certain effective field are a three-dimensional vector (0.20, 0.50, 0.10), the first-layer mapping matrix is ​​[[1, 0, 0.5], [0, 1, 0.2]], and the bias is (0.01, -0.02). Then the result of the first-layer affine mapping is (0.26, 0.50), which remains (0.26, 0.50) after nonlinear activation. Suppose the second-layer mapping matrix is ​​[[1, 0], [0, 1], [0.5, 0.5], [-0.2, 0.4]], and the bias is (0, 0, 0.01, 0). Then the generated pseudo-visual semantic label is (0.26, 0.50, 0.39, 0.148). If the UAV image patch features and satellite image patch features are both four-dimensional vectors, then the pseudo-visual semantic label has the same dimension as the two types of image patch features and can be inserted into the input sequences of the visual encoders on both sides.

[0071] Furthermore, pseudo-visual semantic tags retain their source field types and field-level evidence chain pointers during generation. For pseudo-visual semantic tags generated from the road structure field, the system records their corresponding road vectors, intersection morphology, and connectivity evidence; for pseudo-visual semantic tags generated from the building distribution field, the system records their corresponding building outlines, building density, and arrangement direction evidence; for pseudo-visual semantic tags generated from the typical feature field, the system records their corresponding water bodies, green spaces, playgrounds, squares, parking areas, or large buildings evidence. Thus, pseudo-visual semantic tags are not simply indiscriminate concatenations of ordinary text vectors, but rather visually encoded inputs authorized by valid fields and carrying map factual sources.

[0072] S6. The performance of the trained multi-source semantic information fusion navigation model is evaluated using a test set.

[0073] Specifically, the trained multi-source semantic information fusion navigation model is evaluated using a test set. Let the test set contain... The first drone query sample, the first The actual geographical location of each query sample is The model outputs the predicted geographical location as The geographical distance error between the two is: ; When geographical distance error Less than or equal to the preset distance threshold At that time, the judgment of the first The query sample was correctly located. Location accuracy. The calculation formula is: ; Furthermore, set Indicates the first The rank of the correct reference unit corresponding to each query sample in the search results is the first-rank recall rate. The formula for calculating @1 is: ; Let the first The length of the search sort list corresponding to each query sample is , Indicates the first in the sorted list If the reference unit is a correct match, then... ,otherwise Before sorting Precision of each result for: ; No. Average precision of each query sample for: ; Overall average accuracy of the test set for: ; Among them, positioning accuracy is used to characterize whether the distance error between the model's output position and the actual position meets the navigation and positioning requirements. @1 is used to indicate whether the correct reference unit is at the beginning of the search results. Used to characterize the average ranking quality of the correct reference unit in the overall retrieval ranking results.

[0074] Furthermore, in step S6, the initial source of geographic distance error is either planar Euclidean distance in geometry or latitude and longitude distance conversion in geodesy. In small-scale UAV navigation scenarios, latitude and longitude coordinates can be projected onto a metric plane coordinate system first, and then the Euclidean distance between the predicted position and the actual position can be calculated; in larger-scale scenarios, a spherical distance approximation can be used. The initial sources of positioning accuracy, first-to-last recall, average precision, and test set average precision are classification evaluation indicators and information retrieval ranking evaluation indicators, respectively. This application uses these to evaluate the retrieval and positioning results of UAV query images in the reference unit library.

[0075] As a computational example, suppose there are 3 query samples in the test set, and the positioning distance threshold is 20 meters. The correct reference unit of the first sample is ranked first, with a first-candidate positioning error of 8 meters; the correct reference unit of the second sample is ranked third, with a first-candidate positioning error of 25 meters; and the correct reference unit of the third sample is ranked first, with a first-candidate positioning error of 14 meters. Since 8 meters is less than 20 meters, 25 meters is greater than 20 meters, and 14 meters is less than 20 meters, the positioning precision is 2 divided by 3, i.e., 0.6667. The first-candidate recall rate is calculated based on whether the correct reference unit is ranked first. The first and third samples hit the target, while the second sample did not. Therefore, the first-candidate recall rate is 2 divided by 3, i.e., 0.6667. If each query sample has only one correct reference unit, the average precision of the first, second, and third samples are 1, 0.3333, and 1, respectively. The average precision of the test set is the average of the three, i.e., 0.7778. The accuracy, recall, and mean precision mentioned above are all dimensionless proportional values, while the distance error and distance threshold are both in meters, ensuring consistent dimensions during comparison.

[0076] S7. Deploy the trained multi-source semantic information fusion navigation model to the edge device, and perform inference verification and closed-loop re-retrieval in conjunction with the indexing mechanism.

[0077] Specifically, the trained multi-source semantic information fusion navigation model is exported as an open neural network exchange format model, and quantized and compressed to generate a TensorRT inference engine adapted to the NVIDIA JetsonThor edge computing platform. The fusion features of each reference unit are extracted offline, and the fusion features, structured semantic text, and geographic location tags are used to establish vector indices, semantic indices, and location indices, respectively.

[0078] The TensorRT inference engine, vector index, semantic index, and location index were deployed to the JetsonThor edge computing platform. During edge loading verification, the consistency of model input / output dimensions, index reading results, and geographic location label mapping was checked. For edge inference performance testing, drone aerial images from the test set were input into the JetsonThor edge model to obtain the candidate reference unit ranking results. Based on the correspondence between the candidate reference unit ranking results and the real geographic location labels, the positioning accuracy, first-to-last recall, and average precision of the edge model were calculated. Simultaneously, single-frame inference latency and model storage usage were measured to verify whether the edge-deployed model met the drone edge navigation and positioning requirements.

[0079] Furthermore, in the end-side navigation closed-loop verification in step S7, the low-confidence re-retrieval is limited to a conditionally triggered end-side closed-loop re-retrieval. The initial sources of this triggering rule include conditional triggering rules in discrete event systems and velocity, time, and displacement constraints in kinematics. The modification in this application is that visual matching stability, semantic field consistency, and positional continuity are respectively assigned to different processing actions, rather than uniformly weighting indicators from different sources and with different meanings into a single comprehensive confidence level. The visual matching stability is determined based on the matching results between image features output by the visual encoder, the semantic consistency is determined based on the matching results between semantic fields in the structured semantic text and semantic fields in the index, and the positional continuity is determined based on the geographic location labels of reference units and their spatial adjacency relationships, and is jointly judged during inference verification and closed-loop re-retrieval.

[0080] The edge actions are determined according to the following rules: If the visual matching difference between the first and second candidate reference units is less than the visual matching stability threshold, the vector index retrieval range is expanded; if the effective semantic fields of the candidate reference units are inconsistent with the semantic fields of the current UAV image recognition, a second sorting of the semantic index is triggered; if the spatial distance between the current positioning result and the previous positioning result is greater than the maximum allowable flight speed of the UAV multiplied by the time interval plus the positioning error margin, position continuity correction is triggered. These three conditions correspond to different processing actions and are not subject to comprehensive scoring and fusion.

[0081] As a calculation example, if the similarity of the first candidate match is 0.82 and the similarity of the second candidate match is 0.79, then the visual matching difference is 0.03. When the visual matching stability threshold is 0.05, 0.03 is less than 0.05, triggering an expansion of the vector index retrieval range. If the effective field of the candidate reference unit is "crossroads, rectangular building complex", while the current UAV image recognition result is "T-junction, rectangular building complex", then the semantic fields are inconsistent, triggering a second sorting of the semantic index. If the distance between the previous positioning result and the current positioning result is 18 meters, the maximum allowed flight speed of the UAV is 8 meters per second, the time interval between two adjacent positioning is 1.5 seconds, and the positioning error margin is 3 meters, then the allowed continuous displacement is 8 times 1.5 plus 3, which is 15 meters. Since 18 meters is greater than 15 meters, position continuity correction is triggered.

[0082] Furthermore, the semantic index on the edge does not store complete natural language text, but rather a summary of the valid field evidence chain of candidate reference units. This evidence chain summary includes the valid field type, field processing status, corresponding map features, and corresponding spatial relationships. When visual matching of candidate reference units on the edge becomes unstable, the system first expands the vector index retrieval range; when the valid field evidence chain summary of a candidate reference unit is inconsistent with the current UAV image recognition result, the system triggers a secondary sorting of the semantic index; when continuous positioning results exceed flight speed and time interval constraints, the system triggers position continuity correction. Therefore, the edge-side closed-loop re-retrieval is not simply a post-processing based on confidence thresholds, but rather reuses the field-level evidence chain formed during the training phase to verify candidate positions.

[0083] In another embodiment of the present invention, an unmanned aerial vehicle (UAV) edge navigation system based on multi-source semantic fusion is provided, the system comprising: The multi-source data layer is used to acquire UAV aerial images, satellite remote sensing images and open map data within the target area. It performs coordinate unification, scale unification and spatial alignment processing on the UAV aerial images, satellite remote sensing images and open map data within the same geographical area to construct reference units. A geographic reference layer is used to construct prompt word templates based on the reference units, and to use the prompt word templates to constrain the visual big language model to generate structured semantic text. The structured semantic text includes fields such as road structure, building distribution, typical features, regional functions, spatial relationships, and traffic risk. The structured semantic text is set to perform field-level confidence verification and map feature consistency verification based on field-level evidence chains, so as to filter and mark the generated structured semantic text to obtain structured semantic text that meets the preset verification conditions. The semantic annotation module is used to normalize the verified structured semantic text and remove duplicates from similar texts to obtain deduplicated structured semantic text. The text screening module is used to bind the deduplicated structured semantic text with the corresponding UAV aerial images, satellite remote sensing images, open map data, and geographic location tags to construct a geographic semantic annotation sample library. The fusion training module is used to train a multi-source semantic information fusion navigation model based on the geographic semantic annotation sample library, and to introduce text semantic information in the visual encoding process to enhance the expression of image features. The model evaluation module is used to evaluate the performance of the trained multi-source semantic information fusion navigation model using a test set. The edge deployment module is used to deploy the trained multi-source semantic information fusion navigation model to edge devices, and to perform inference verification and closed-loop re-retrieval in conjunction with the indexing mechanism.

[0084] Through the above system, the present invention can realize a complete closed loop from geographic semantic sample construction, fusion positioning model training, model edge deployment to simulation testing and evaluation, avoiding the problem that existing methods only stop at offline dataset testing and lack edge usability verification.

[0085] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or units may be electrical, mechanical, or other forms.

[0087] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A UAV edge navigation method based on multi-source semantic fusion, characterized in that, The methods include: S1. Acquire UAV aerial images, satellite remote sensing images and open map data within the target area, and perform coordinate unification, scale unification and spatial alignment processing on the UAV aerial images, satellite remote sensing images and open map data within the same geographical area to construct reference units; S2. Construct a prompt word template based on the reference unit, and use the prompt word template to constrain the visual big language model to generate structured semantic text. The structured semantic text includes fields such as road structure, building distribution, typical features, regional functions, spatial relationships, and traffic risk. The structured semantic text is set to perform field-level confidence verification and map feature consistency verification based on field-level evidence chain to filter and mark the generated structured semantic text, thereby obtaining structured semantic text that meets the preset verification conditions. S3. Perform normalization processing and deduplication of similar texts on the verified structured semantic text to obtain the deduplicated structured semantic text; S4. Bind the deduplicated structured semantic text with the corresponding UAV aerial images, satellite remote sensing images, open map data, and geographic location tags to construct a geographic semantic annotation sample library. S5. Train a multi-source semantic information fusion navigation model based on the geographic semantic annotation sample library, and introduce text semantic information in the visual encoding process to enhance the expression of image features; S6. The performance of the trained multi-source semantic information fusion navigation model is evaluated using a test set. S7. Deploy the trained multi-source semantic information fusion navigation model to the edge device, and perform inference verification and closed-loop re-retrieval in conjunction with the indexing mechanism.

2. The UAV end-side navigation method based on multi-source semantic fusion according to claim 1, characterized in that, The reference unit is a spatial unit obtained by dividing a geographic grid at a fixed scale. Each reference unit corresponds to a unique geographic identifier and geographic location label.

3. The UAV end-side navigation method based on multi-source semantic fusion according to claim 1, characterized in that, The prompt word template is used to constrain the visual large language model to perform semantic annotation of UAV navigation scene based on satellite remote sensing images and open map tiles, and output structured semantic text according to a preset format.

4. The UAV end-side navigation method based on multi-source semantic fusion according to claim 1, characterized in that, The specific settings for the field-level evidence chain are as follows: Each semantic field corresponds to an independent field-level evidence chain, which includes at least the field description content, the corresponding open map element, the corresponding image region, the map element existence verification result, the spatial relationship consistency verification result, and the field processing status.

5. The UAV end-side navigation method based on multi-source semantic fusion according to claim 4, characterized in that, The rules for the field processing status are as follows: Fields whose description objects exist and whose spatial relationships are consistent are marked as valid fields; fields whose description objects exist but whose spatial relationships cannot be confirmed are marked as fields to be reviewed; fields whose description objects do not exist or whose spatial relationships are opposite to those of the open map tiles are marked as invalid fields.

6. The UAV end-side navigation method based on multi-source semantic fusion according to claim 1, characterized in that, The field-level evidence chain is bound to the reference unit number in the data structure. Each evidence chain records at least the field type, field description content, candidate open map elements, candidate image areas, map element existence verification results, spatial relationship consistency verification results, field processing status, and subsequent processing permissions. The subsequent processing permissions include whether to allow access to the training sample library, whether to allow the generation of pseudo-visual semantic tags, whether to allow access to the edge semantic index, and whether a second review is required.

7. The UAV end-side navigation method based on multi-source semantic fusion according to claim 4, characterized in that, The normalization process specifically includes: The semantic fields are uniformly formatted, and similar text is deduplicated by combining keyword matching and semantic feature similarity judgment; In the process of deduplicating similar texts, the degree of similarity between structured semantic texts is first determined based on keyword matching and semantic feature similarity. Then, the similar text is judged as redundant text or semantic nearest neighbor reference unit based on whether the geographical range overlaps. The semantic nearest neighbor reference unit is constructed based on field-level evidence chain. If the similarity is greater than the preset semantic nearest neighbor threshold and the geographical ranges of the two reference units do not overlap, then neither text will be deleted. Instead, both will be marked as semantic nearest neighbor reference units and used as hard-to-bear samples. If the two belong to the same geographical grid and their geographical ranges overlap, then the redundant text retention rule will be applied. The semantic nearest neighbor reference unit is constructed based on a field-level evidence chain, specifically as follows: For two reference units that are semantically similar but do not overlap geographically, not only is the degree of similarity between the two reference units recorded, but also the effective field types that lead to the similarity and the source of their evidence chains.

8. The UAV end-side navigation method based on multi-source semantic fusion according to claim 1, characterized in that, The multi-source semantic information fusion navigation model includes a UAV image visual coding branch, a satellite image visual coding branch, a text semantic coding branch, and a cross-modal fusion positioning branch; The UAV image visual coding branch is used to extract features from UAV aerial images, the satellite image visual coding branch is used to extract features from satellite remote sensing images, the text semantic coding branch is used to extract structured semantic text features, and the cross-modal fusion localization branch is used to map UAV image features, satellite image features, and text semantic features to a unified feature space. Before visual encoding, the multi-source semantic information fusion navigation model inputs the structured semantic text obtained after map element existence verification and spatial relationship consistency verification into the text semantic encoding module to obtain field-level text semantic features. The field-level text semantic features are then input into the inverse mapping network to generate pseudo-visual semantic tags. The pseudo-visual semantic tags have the same dimension as the image patch features and are used to combine with the image features in the visual encoder.

9. The UAV end-side navigation method based on multi-source semantic fusion according to claim 1, characterized in that, The specific steps for performing reasoning verification and closed-loop re-retrieval using the indexing mechanism are as follows: The system performs joint retrieval based on vector index, semantic index, and position index, and triggers expanded retrieval, secondary sorting, or position correction operations according to visual matching stability, semantic field consistency, and position continuity, respectively, to achieve closed-loop re-retrieval.

10. An unmanned aerial vehicle (UAV) edge navigation system based on multi-source semantic fusion, characterized in that, The system includes: The multi-source data layer is used to acquire UAV aerial images, satellite remote sensing images, and open map data within the target area. It performs coordinate unification, scale unification, and spatial alignment processing on the UAV aerial images, satellite remote sensing images, and open map data within the same geographical area to construct reference units. A geographic reference layer is used to construct prompt word templates based on the reference units, and to use the prompt word templates to constrain the visual big language model to generate structured semantic text. The structured semantic text includes fields such as road structure, building distribution, typical features, regional functions, spatial relationships, and traffic risk. The structured semantic text is set to perform field-level confidence verification and map feature consistency verification based on field-level evidence chains to filter and mark the generated structured semantic text, thereby obtaining structured semantic text that meets the preset verification conditions. The semantic annotation module is used to normalize the verified structured semantic text and remove duplicates from similar texts to obtain deduplicated structured semantic text. The text screening module is used to bind the deduplicated structured semantic text with the corresponding UAV aerial images, satellite remote sensing images, open map data, and geographic location tags to construct a geographic semantic annotation sample library. The fusion training module is used to train a multi-source semantic information fusion navigation model based on the geographic semantic annotation sample library, and to introduce text semantic information in the visual encoding process to enhance the expression of image features. The model evaluation module is used to evaluate the performance of the trained multi-source semantic information fusion navigation model using a test set. The edge deployment module is used to deploy the trained multi-source semantic information fusion navigation model to edge devices, and to perform inference verification and closed-loop re-retrieval in conjunction with the indexing mechanism.

Images