Scene matching and localization method for unmanned systems in area denial scenarios

Abstract

This invention proposes a scene matching and localization method for unmanned systems in area-denied scenarios, belonging to the field of image matching and localization technology. First, UAV imagery and satellite base maps are processed, and semantic entities in the images are identified. Relationship strength weights are calculated, and a causal semantic graph is constructed. Then, a keypoint detection mask is generated, obtaining a set of keypoints from the UAV imagery and satellite base maps, and establishing associations between keypoints and semantic entities. The semantic association strength between keypoints is calculated, and enhanced features are obtained. Keypoint matching pairs are filtered to establish an initial matching set. Based on the weighted fusion results, the initial matching set is further filtered to obtain a refined matching set. The node feature matrix is ​​optimized, and the refined matching set is evaluated to obtain the final matching set. The final matching set is used to calculate the deviation vector between the actual and estimated positions of the UAV, obtaining the fused position estimate. This invention can improve the scene matching and localization accuracy in area-denied scenarios.

Description

Technical Field

[0001] This invention relates to the field of image matching and localization technology, specifically to a method for scene matching and localization of unmanned systems in area rejection scenarios. Background Technology

[0002] High-altitude unmanned systems (UAVs), with their wide-area monitoring, remote sensing, and integrated mission capabilities, have established routine patrol mechanisms in multiple coastal areas and are widely used in maritime monitoring, ecological protection, disaster response, and security operations, demonstrating significant technological support value. However, during long-distance, extended-duration coastal patrols, UAVs are susceptible to interference from complex electromagnetic environments. Typical interference methods include satellite navigation signal denial-of-area, navigation spoofing, and communication jamming, affecting areas up to hundreds of kilometers in radius. Furthermore, due to the cumulative error of airborne inertial navigation systems (for example, the pure inertial navigation error of some high-altitude UAVs can reach 1 nautical mile per hour), their positioning accuracy is insufficient for continuous operation in environments where satellite signals are denied or severely interfered with.

[0003] Image matching navigation technology, with its advantages of strong autonomy, good environmental adaptability, high accuracy, and low hardware cost, has become a feasible solution for achieving high-precision autonomous positioning in environments without satellite navigation. This technology acquires ground images in real time using imaging equipment mounted on an unmanned system, performs feature matching with pre-stored satellite base maps, and combines this with pose information to achieve precise positioning. The matching accuracy between real-time imagery and satellite base maps directly determines the accuracy of the positioning results and the stability of the system.

[0004] Currently, commonly used scene matching methods mainly include:

[0005] Feature-point-based matching methods (such as SIFT, SURF, ORB, etc.) perform matching by extracting key points in images and generating feature descriptors; region-based matching methods (such as cross-correlation matching, mutual information matching, etc.) rely on calculating the similarity between regions of two images to achieve matching; deep learning-based methods (such as SuperGlue, LoFTR, etc.) use neural networks to extract high-level features and establish matching relationships.

[0006] However, in complex coastal and near-shore scenarios, existing methods still face several technical challenges:

[0007] First, multi-source interference easily leads to feature space aliasing. Images taken from high altitudes are susceptible to multi-source environmental interference, such as atmospheric turbulence, sea surface ripples, and exposed intertidal reefs, resulting in unstable image features. Existing methods mostly rely on data-driven shallow statistical correlations, failing to fully incorporate semantic and topological constraints, thus having limitations in anti-interference and effective feature extraction. They also lack effective mechanisms for fusing prior information with image features, making it difficult to construct robust feature representations.

[0008] Secondly, the spatiotemporal variations of land features introduce uncertainty into the matching process. Coastal topography is dynamically changing due to tides, seasons, and human activities, while satellite base maps have a limited update frequency and cannot synchronize the actual situation in real time, resulting in significant spatiotemporal heterogeneity in the matching features. Existing algorithms typically have fixed structures, making it difficult to adaptively adjust the network structure or task coordination strategies based on differences in the spatial distribution and temporal dimensions of the matching images.

[0009] Third, the problem of cross-domain feature heterogeneity affects the model's generalization ability. Differences exist between unmanned system-captured images and satellite base maps in terms of shooting perspective, sensor type, and imaging conditions, leading to inconsistent feature distributions. Mainstream supervised learning methods struggle to adequately adapt to the training needs of multi-view, multi-modal data, lacking hierarchical gradient propagation mechanisms and the ability to progressively learn domain-invariant features from unlabeled data.

[0010] Therefore, in order to meet the application needs of coastal and complex geographical environments under navigation signal rejection conditions, it is urgent to develop a robust and highly adaptive scene matching positioning method to improve the positioning accuracy and reliability of unmanned systems in real complex scenarios. Summary of the Invention

[0011] To address the problems existing in the prior art, this invention proposes a scene matching and localization method for unmanned systems in area rejection scenarios, comprising the following steps:

[0012] Step 1: Real-time acquisition of UAV images by the high-altitude unmanned system Perform geometric correction to obtain the corrected UAV image. ; Corrected drone imagery and pre-stored satellite base maps The drone imagery is then processed by equalization. and satellite base map Based on the characteristics of the coastline, the processed drone images are... and satellite base map All are divided into sea areas shoreline area and land areas Three sub-regions, with different weights assigned to each region. ;

[0013] Step 2: Identify drone images and satellite base map The semantic entities in the graph are identified, spatial relationships between them are established, and the relationship strength weights between semantic entities are calculated to construct a causal semantic graph. ;

[0014] Step 3: Based on the causal semantic graph A keypoint detection mask is generated, where the pixel value at a certain position in the mask represents the priority of that position as a keypoint; then, the keypoint detection mask is used to process the UAV imagery. and satellite base map Keypoint detection based on a self-supervised deep learning architecture is performed to obtain UAV imagery. and satellite base map Each corresponding set of key points and And extract deep feature descriptors of key points. and Finally, establish the association between key points and semantic entities. ;

[0015] Step 4: Based on causal semantic graph Calculate the semantic association strength between keypoints; construct a self-attention-based feature enhancement network based on the semantic association strength, and fuse the input features of the feature enhancement network with external environment and time factors to obtain enhanced features. and ; Utilizing enhanced features and Construct an optimal transportation problem with causal semantic constraints, and obtain the optimal matching matrix by solving the optimal transportation matching problem. Using the optimal matching matrix imagery of drones and satellite base map Filter the key point matching pairs between them to establish an initial matching set. ;

[0016] Step 5: For the initial matching set Local neighborhood consistency verification and global structure verification based on causal semantic relationships are performed separately. The local consistency scores obtained from local neighborhood consistency verification, the global consistency scores obtained from global structure verification, and the initial matching scores of the matching pairs are then weighted and fused. Low-confidence matches are filtered out according to the fusion result to obtain a refined matching set. ;

[0017] Step 6: Combine causal semantic graphs For the matching set Construct an enhanced adjacency matrix Then use the adjacency matrix A multi-layer graph convolutional network is applied to optimize the node feature matrix; the optimized node feature matrix is ​​then used to re-evaluate the matching set. Keypoint matching pairs are processed, and matches that do not conform to global semantic consistency are removed to obtain the final matching set. ;

[0018] Step 7: Based on the final matching set Calculate the deviation vector between the actual position and the estimated position of the UAV. ; the deviation vector The position estimate is obtained by fusing the data with inertial navigation data.

[0019] Furthermore, in step 1, the regional weights ,in As the weight of the shoreline area, For land area weights, As the weight of the sea area, and .

[0020] Furthermore, in step 2, the causal semantic graph ,in A collection of nodes with semantic entities as nodes. For a set of nodes There exists a set of edges consisting of the relationship between two semantic entities with a clear spatial connection. For a set of spatial relationships The set of edge weights, which consists of the strength weights of each relation.

[0021] Furthermore, in step 2, according to the formula:

[0022]

[0023] Computational semantic entity inter-entity and The strength weight of the relationship between them; where The function is a stability function, used to evaluate the stability of relationships between semantic entities. The function is a persistence function used to evaluate the degree of change of a relationship over time. The function is a distinctiveness function, used to evaluate the visual distinctiveness and uniqueness of a relationship. The function is the relation weight function, taking:

[0024]

[0025] in As an indicator variable for the shoreline area, For land area indicator variables, This is an indicator variable for the sea area.

[0026] Furthermore, in step 3, according to the formula:

[0027]

[0028] Generate keypoint detection mask ,in Indicates position The guide mask value at a given location indicates the priority of that location as a keypoint. This is an indicator function; it returns 1 if the condition within the parentheses is true, and 0 otherwise. Representing semantic entities In satellite base map The spatial area covered in it semantic entities Importance weights.

[0029] Furthermore, in step 4, according to the formula:

[0030]

[0031] Calculate the key point set based on causal semantic graph. A key point in With key point set A key point in Semantic association strength between ,in It is an indicator function, when Returns 1 if the condition is true, otherwise returns 0; Indicate key points With semantic entities There is a mapping relationship. Indicate key points With semantic entities There is a mapping relationship. semantic entities With semantic entities Relationship strength weight.

[0032] Furthermore, in step 4, enhanced features are utilized. and The optimal transportation problem with causal semantic constraints is constructed as follows:

[0033]

[0034]

[0035]

[0036] in Indicates enhanced features Corresponding key points eigenvectors, Indicates enhanced features Corresponding key points eigenvectors, These are preset coefficients; Key point With key points The cost matrix between It is the optimal matching matrix. Indicate key points in drone imagery Key points in satellite base map Matching probability; constraints Ensure that the sum of the matching probabilities of key points meets the preset distribution. and .

[0037] Furthermore, in step 5, for the initial matching set Key point matching pairs According to the formula:

[0038]

[0039] Computational fusion results ,in , and These are weighting coefficients, satisfying... , Keypoint matching pairs The initial matching score, Keypoint matching pairs Local consistency score, Keypoint matching pairs The global structural consistency score.

[0040] Furthermore, in step 6, the formula for the node feature matrix is:

[0041]

[0042] in and They represent the number respectively. Layer and first The node feature matrix of the layer, It is an enhanced adjacency matrix The degree matrix, It is the first The learnable weight matrix of the layer, It is a non-linear activation function; the enhanced adjacency matrix According to the formula:

[0043]

[0044] Received, among which Based on matching sets The constructed original adjacency matrix, Is with matrix Identity matrices of the same size These are the weighting coefficients for causal relationships. It is a causal adjacency matrix.

[0045] Furthermore, in step 6, according to the formula:

[0046]

[0047] Get the final matching set ,in Key point In the node feature matrix The L2 norm of the corresponding eigenvectors in the equation. Key point In the node feature matrix The L2 norm of the corresponding eigenvectors in the equation. The threshold for graph convolution refinement.

[0048] Beneficial effects:

[0049] This invention proposes a scene matching and localization method for unmanned systems in area-rejection scenarios, which has the following advantages:

[0050] 1. This invention adopts a feature consistency representation method based on causal semantic knowledge embedding, constructs a semantic topology graph of the coastline scene, and performs feature propagation and refinement through a graph convolutional network. This solves the problem of feature space aliasing and distribution drift caused by multi-source interference, achieves stability and consistency of feature expression, and effectively suppresses the influence of interference factors such as atmospheric turbulence and sea surface ripples on matching accuracy.

[0051] 2. This invention achieves accurate identification and matching of coastline features through a matching strategy combining keypoint extraction and causal semantic enhancement. By explicitly modeling the semantic relationships between keypoints, stable feature matching performance can be maintained even under environmental interference or changes in perspective, significantly improving matching accuracy and reliability.

[0052] 3. This invention adopts a multi-source information fusion method for position calculation and accuracy evaluation, optimizes and fuses scene matching results with inertial navigation data, and introduces a reliability evaluation mechanism, enabling the system to evaluate positioning accuracy in real time and make adaptive adjustments, thereby improving the accuracy and reliability of positioning.

[0053] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0054] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0055] Figure 1 Flowchart of scene matching and localization method for unmanned systems in area rejection scenarios. Detailed Implementation

[0056] The embodiments of the present invention are described in detail below. These embodiments are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0057] This embodiment proposes a corresponding positioning method for high-altitude UAV scene matching and positioning in coastal area denial scenarios, including the following steps:

[0058] Step 1: Preprocessing and Adaptive Augmentation of Multi-Source Heterogeneous Data

[0059] High-altitude drones collect real-time images of the coastline area Based on UAV attitude parameters and camera internal reference imagery of drones Perform geometric corrections to eliminate perspective distortion and obtain corrected UAV images. ; Corrected drone imagery and pre-stored satellite base maps Multi-scale adaptive histogram equalization is performed to improve image contrast, resulting in the processed UAV image. and satellite base map Based on the characteristics of the coastline, the processed drone images... and satellite base map All are divided into sea areas shoreline area and land areas Three sub-regions, with different weights assigned to each region. .

[0060] In this embodiment, the input for step 1 is the drone imagery collected in real time by the high-altitude drone. and pre-stored satellite base maps UAV attitude parameters and camera internal reference ;

[0061] The processing procedure is as follows:

[0062] Step 1.1: Based on UAV attitude parameters and camera internal reference Perform geometric corrections to eliminate perspective distortion:

[0063]

[0064] In this step, the geometric correction function By utilizing the UAV's attitude parameters (roll, pitch, and yaw angles) and the camera's intrinsic parameter matrix, a perspective transformation matrix is ​​calculated to correct the tilted images into images with an approximately vertical viewing angle. This makes them more consistent with the observation angle and size of the satellite base map, laying the foundation for subsequent matching. The geometric correction function can be implemented using existing techniques in this field. .

[0065] Step 1.2: Employ multi-scale adaptive histogram equalization to improve image contrast.

[0066] ,

[0067] In this step, This represents a multi-scale adaptive histogram equalization algorithm, which performs histogram equalization at multiple scales separately and then adaptively fuses the results from each scale. Parameters Controlling the degree of local contrast enhancement, appropriately increasing it can improve the visibility of details; Controls global brightness adjustment to compensate for different lighting conditions; This is a detail enhancement factor used to enhance texture features. This processing effectively addresses imaging differences in coastal scenes caused by weather conditions and lighting variations. Multi-scale adaptive histogram equalization algorithm. This is existing technology in this field.

[0068] Step 1.3: Based on the characteristics of the coastline, the images processed in Steps 1.1 and 1.2 are all divided into marine areas. shoreline area and land areas Three sub-regions, each with a different weight:

[0069]

[0070] in, As the weight of the shoreline area, For land area weights, Weighting of sea area This highlights the importance of the shoreline area as the primary matching target; in this embodiment, we take... It is 0.8. It is 0.6. The weight is 0.3. This region division is achieved using image segmentation technology, which identifies the boundaries of different regions by analyzing the color, texture, and spatial continuity features of the image. The weight allocation reflects the importance of each region in the matching process, with particular emphasis on the shoreline region, because shoreline morphology usually has high temporal stability and visual recognizability.

[0071] The final output of this step is the preprocessed drone image. Satellite base map and region weight vector .

[0072] Step 2: Construct a causal semantic graph for the coastline scene:

[0073] Identify the drone images processed in step 1 and satellite base map The semantic entities in the graph are identified, and spatial relationships between them are established. Simultaneously, the relationship strength weights between semantic entities are calculated to obtain a complete causal semantic graph.

[0074]

[0075] in A collection of nodes with semantic entities as nodes. For a set of nodes There exists a set of edges consisting of the relationship between two semantic entities with a clear spatial connection. For a set of spatial relationships The set of edge weights, which consists of the strength weights of each relation.

[0076] In this embodiment, the input for step 2 is the preprocessed UAV image obtained in step 1. Satellite base map and region weight vector ;

[0077] The processing procedure is as follows:

[0078] Step 2.1: Identify the main semantic entities in the coastline scene:

[0079]

[0080] In this step, The function uses a fully convolutional neural network to process the input drone imagery. Satellite base map This function performs pixel-level semantic segmentation to identify various key features. It first extracts multi-scale features, then captures global information through a context aggregation module, and finally generates a high-precision segmentation map using a decoder, resulting in a set of nodes. This includes semantic entities such as shoreline edges, reefs, and buildings. These semantic entities are the basic nodes for constructing a causal semantic graph.

[0081] Step 2.2: Establish spatial relationships between semantic entities:

[0082]

[0083] function Used to analyze the geometric and spatial configuration relationships between semantic entities, when semantic entities and When there is a clear spatial relationship between two points, establish an edge. Spatial relationships include: adjacency (e.g., a dock is adjacent to the shoreline), containment (e.g., a group of reefs is contained within a bay), parallelism (e.g., a road is parallel to the shoreline), and intersection (e.g., a bridge intersects with a river). These relationships reflect the spatial layout of objects in the scene and provide spatial constraints for subsequent matching.

[0084] Step 2.3: Calculate the relationship strength weights based on regional characteristics and prior knowledge:

[0085]

[0086] This formula calculates the strength weight of the relationship between semantic entities, taking into account multiple factors:

[0087] The function is a stability function; in this embodiment, we take... ,in semantic entities stability coefficient, semantic entities The stability coefficient is set as follows. In this embodiment, the stability coefficient of semantic entities such as fixed features, such as buildings, roads, and bridges, is 0.9; the stability coefficient of semantic entities such as semi-fixed features, such as shorelines and large reefs, is 0.7; the stability coefficient of semantic entities such as changing features, such as vegetation and beaches, is 0.5; and the stability coefficient of semantic entities such as dynamic features, such as ships and water ripples, is 0.2. The function is used to evaluate the stability of relationships between semantic entities. For example, the relationship between shoreline and reef fluctuates with tides and is therefore less stable, while the relationship between shoreline and lighthouse is relatively stable.

[0088] The function is a persistent function; in this embodiment, it is taken as... ,in semantic entities The rate of change over time semantic entities The temporal change rate is as follows: In this embodiment, the temporal change rate of semantic entities such as permanent structures, concrete buildings, and rocky coastlines is 0.05, i.e., an annual change rate of 5%; the temporal change rate of semantic entities such as long-term stable structures, asphalt roads, and large wharves is 0.15, i.e., an annual change rate of 15%; the temporal change rate of semantic entities such as medium-term changing features, such as vegetation cover and sandy coastlines, is 0.35, i.e., an annual change rate of 35%; the temporal change rate of semantic entities such as short-term changing features, such as seasonal vegetation and intertidal zones, is 0.60, i.e., an annual change rate of 60%; and the temporal change rate of semantic entities such as high-frequency changing features, such as water surface conditions and temporary facilities, is 0.85, i.e., an annual change rate of 85%. The function is used to evaluate the degree of change of the relationship over time, taking into account changes over time, such as seasonal changes having a greater impact on vegetation but a smaller impact on buildings.

[0089] The function is the discrimination function; in this embodiment, we take... ,in semantic entities The recognizability, semantic entities The recognition score is as follows: In this embodiment, the recognition score of semantic entities such as unique landmarks, lighthouses, and buildings with special shapes is 0.45; the recognition score of semantic entities such as linear features, roads, rivers, and coastlines is 0.35; the recognition score of semantic entities such as areal features, building clusters, and forests is 0.25; the recognition score of semantic entities such as texture features, farmland, and grasslands is 0.15; and the recognition score of semantic entities such as uniform regions, water surfaces, and open spaces is 0.05. The function is used to evaluate the visual recognizability and uniqueness of relationships. For example, relationships between unique landmarks (such as a uniquely shaped bay and a lighthouse) have high recognizability, and the constraints are... That is, the total recognition score does not exceed 0.9.

[0090] The values ​​of the three functions mentioned above are set according to the stability, persistence, and recognizability between specific semantic entities.

[0091] The function is a relation weighting function, which can adjust the relation strength weights according to the region weight vector determined in step 1, thereby giving priority to the semantic entity relations of the shoreline region. In this embodiment, we take... ,in As an indicator variable for the shoreline area, For land area indicator variables, For sea area indicator variables, The value can be 0, 0.5, or 1, when a certain region contains semantic entities. and When the region indicator variable is 1, it indicates that the region contains only semantic entities. and When a region contains no semantic entities, the indicator variable for that region is 0.5. and When that time, the indicator variable for that region is 0.

[0092] The above four factors are multiplied together to obtain the comprehensive relationship strength weight. The higher the value, the more important the relationship is, and the greater the weight in subsequent feature matching.

[0093] Step 2.4: Construct a complete causal semantic graph:

[0094]

[0095] This step will obtain the set of nodes. (Semantic entities), edge sets (Spatial relationships) and edge weight set (Relationship strength) is integrated into a complete causal semantic graph. As the output of this step, the graph is stored in the form of a data structure, which can be an adjacency matrix or an adjacency list, containing node information, edge connections, and edge weights. This semantic graph is the core knowledge representation that guides feature extraction and matching in subsequent steps.

[0096] The final output of this step is a causal semantic graph. .

[0097] Step 3: Keypoint detection and feature extraction guided by causal semantics:

[0098] According to the causal semantic graph Generate a keypoint detection mask of the same size as the image from step 1. The pixel value at a certain position in the mask represents the priority of that position as a keypoint. Then, use the keypoint detection mask to process the UAV image. and satellite base map Keypoint detection based on a self-supervised deep learning architecture is performed to obtain UAV imagery. and satellite base map Each corresponding set of key points and And extract deep feature descriptors of key points. and Finally, establish the association between key points and semantic entities. .

[0099] In this embodiment, the input for step 3 is the preprocessed UAV image. Satellite base map and causal semantic graph ;

[0100] The processing procedure is as follows:

[0101] Step 3.1: Based on causal semantic graph Generate a semantically guided keypoint detection mask:

[0102]

[0103] function The causal semantic graph is converted into a guiding mask for keypoint detection. In this embodiment, the specific function expression is as follows:

[0104]

[0105] in These are the pixel coordinates in the image. Indicates position The guide mask value at a given location indicates the priority of that location as a keypoint. This is an indicator function; it returns 1 if the condition within the parentheses is true, and 0 otherwise. Representing semantic entities In satellite base map The spatial area covered in it semantic entities In this embodiment, the importance weights for semantic entities such as shorelines are set to 0.9, permanent buildings to 0.8, large landmarks to 0.7, road intersections to 0.6, general features to 0.4, and change zones to 0.2. This function analyzes the causal semantic graph. The importance and stability of each semantic entity in the graph are determined, and the causal semantic graph is converted into a mask of the same size as the image in step 1. The pixel value represents the priority of that location as a keypoint, used for keypoint detection. The mask constructed by this function emphasizes the following areas:

[0106] Coastline edge areas are given the highest priority because the coastline is a stable feature that distinguishes land from sea.

[0107] Permanent building areas (such as lighthouses and docks) have the second highest priority; these man-made structures are usually stable and easily identifiable.

[0108] Distinctive landmark areas (such as mountains and unique terrain) have medium priority;

[0109] Waterway boundary areas (such as estuaries and bay entrances) are given lower but meaningful priority;

[0110] This masking guidance mechanism makes the key point detection process focus more on semantically meaningful and stable regions, reducing sensitivity to volatile regions (such as water ripples).

[0111] Step 3.2: Keypoint detection using a self-supervised deep learning architecture:

[0112]

[0113]

[0114] function It is a keypoint detector that uses a deep learning architecture. It takes a preprocessed image and a semantically guided keypoint detection mask as input and outputs a set of keypoints. The working mechanism of the keypoint detector is as follows:

[0115] First, a multi-scale feature pyramid is constructed using a fully convolutional network (FCN) to capture feature information at different scales;

[0116] Then, a key point heatmap of the image is generated through a multi-scale feature pyramid to represent the probability of each pixel location being a key point;

[0117] Next, the semantic guidance keypoint detection mask is applied. Multiplying with the heatmap enhances the response values ​​of semantically important regions;

[0118] Finally, local optima are selected by non-maximum suppression (NMS) and filtered according to a threshold to obtain the final set of key points;

[0119] This deep learning method has a stronger feature representation capability compared to traditional keypoint detectors, and the introduction of semantic guidance makes the detection more focused on valuable regions.

[0120] Step 3.3: Extract the basic feature descriptors of key points:

[0121]

[0122]

[0123] function This function is used to extract feature descriptors for detected keypoints. It is implemented based on a deep neural network, employing a shared encoder architecture to ensure computational efficiency in feature extraction. Through local feature aggregation, it extracts compact 256-dimensional feature descriptors within the receptive field around the keypoints, and then performs L2 normalization on these compact descriptors to facilitate subsequent similarity calculations. Furthermore, it optimizes the discriminative ability of the descriptors through metric learning training, making descriptors of the same object closer together and descriptors of different objects farther apart. This function... The extracted depth feature descriptors exhibit partial invariance and are robust to changes in illumination, viewpoint, and scale. The capture of visual characteristics such as texture, shape, and structure of the area surrounding the keypoint is the basis for subsequent feature matching.

[0124] Step 3.4: Establish the association between key points and semantic entities:

[0125]

[0126] This step establishes a mapping relationship between key points and semantic entities. That is, determining which semantic entity each key point belongs to. (Function) Output semantic entities The covered spatial area, when key points When falling within this area, establish a connection. This association forms the basis for subsequently enhancing feature representations using semantic information, enabling the system to adjust its feature matching strategy based on semantic relationships.

[0127] The final output of this step is a set of key points. , Basic feature descriptor , The relationship between key points and semantic entities .

[0128] Step 4: Feature representation enhanced by causal semantic attention and optimal matching:

[0129] Based on causal semantic graph Calculate the semantic association strength between keypoints; construct a self-attention-based feature enhancement network based on the semantic association strength, and fuse the input features of the feature enhancement network with external environment and time factors to obtain enhanced features. and ; Utilizing enhanced features and Construct an optimal transportation problem with causal semantic constraints, and obtain the optimal matching matrix by solving the optimal transportation matching problem. Using the optimal matching matrix imagery of drones and satellite base map Filter the key point matching pairs between them to establish an initial matching set. .

[0130] In this embodiment, the input for step 4 is a set of key points. , Basic feature descriptor , The relationship between key points and semantic entities and causal semantic graphs ;

[0131] The processing procedure is as follows:

[0132] Step 4.1: Calculate the key point set based on the causal semantic graph A key point and a set of key points The semantic association strength between a certain key point in the text:

[0133]

[0134] in Set of key points A key point in it, Set of key points A key point in the process is calculated using this formula. and Semantic association strength between ,in It is an indicator function, when Returns 1 if the condition is true, otherwise returns 0. Indicate key points With semantic entities There is a mapping relationship. Indicate key points With semantic entities There is a mapping relationship. semantic entities With semantic entities Relationship strength weight.

[0135] The specific logic of this formula is as follows:

[0136] First, identify the key points. Does it belong to a semantic entity? Key points Does it belong to a semantic entity? That is, whether there is a mapping relationship; if both exist, then the semantic entity is calculated. and Relationship strength weight Finally, for all possible semantic entity pairs The above calculations are performed and summed in each case. Thus, when two keypoints belong to semantically closely related entities (e.g., one to a lighthouse, the other to a nearby dock), their semantic association is strong; while keypoint pairs belonging to unrelated semantic entities (e.g., one to a distant building, the other to near-shore waves) have weaker semantic association. This provides semantic-level prior information for subsequent feature matching.

[0137] Step 4.2: Construct a self-attention-based feature enhancement network:

[0138]

[0139]

[0140] in Representing an image The 0th layer feature, , Representing an image Basic feature descriptors, Representing an image The Layer features, Representing an image The Layer features, functions This indicates a multi-head attention mechanism. This represents the semantic association strength; the above formula describes the processing procedure of a multi-layer self-attention network:

[0141] Initialize the features of layer 0 Basic feature descriptor For the first Layers update features through residual connections, meaning the current layer's features equal the previous layer's features plus the output of a multi-head attention mechanism; functions This represents a multi-head attention mechanism that receives features from the previous layer as a query. ,key Sum and semantic association strength As an additional input.

[0142] The specific calculation formula for the multi-head attention mechanism is as follows:

[0143]

[0144]

[0145] Here, each attention head Attention weights are calculated independently and then summed; where , , It is a learnable projection matrix that projects the input features Z onto the query, key, and value space; Calculate query s and keys The similarity matrix between them Normalization is performed to prevent the gradient vanishing problem; Incorporating semantic association strength into attention computation, where These are the weighting coefficients for semantic information; The function converts similarity into a probability distribution; finally, it is combined with the value matrix. Multiplying them together yields the feature representation of the weighted aggregation.

[0146] The above self-attention-based feature enhancement network has The final feature output obtained from the layers is the feature corresponding to the UAV image. Features corresponding to satellite base maps .

[0147] The innovation of this attention mechanism lies in the introduction of semantic association information. This allows the feature enhancement process to consider not only the similarity of the features themselves, but also the semantic-level correlation, thereby generating more robust feature representations.

[0148] Step 4.3: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require and By integrating with external environment and time factors, the feature representation is further enhanced:

[0149]

[0150]

[0151]

[0152]

[0153] in It is a dynamically adjusted coefficient, achieved through a multilayer perceptron. Based on image quality Platform motion parameters and environmental weather conditions Calculations show that the dynamic adjustment coefficient, ranging from 0 to 1, reflects the degree of dependence on prior knowledge under current conditions, and thus image quality. Platform motion parameters and environmental weather conditions Based on the actual scenario; It is a priori mask based on time and geographic information, applied through a convolutional neural network. Based on the time of day ,season Tidal state and geographical location generate; and All of these were obtained through existing technologies in the field; This represents element-wise multiplication, i.e., multiplying element by element; ultimately, the enhanced features corresponding to the UAV image are obtained. Enhanced features corresponding to satellite base maps .

[0154] This step enables flexible adjustment of feature representations based on external conditions. For example, under low light conditions, reliance on prior knowledge can be increased; during periods of significant tidal change, the impact of tides on shoreline morphology can be compensated for through masking.

[0155] Step 4.4: Utilizing Enhanced Features and Construct and solve the optimal transportation problem with causal semantic constraints:

[0156]

[0157]

[0158]

[0159] The above formula models the feature matching problem as a constrained optimal transportation problem; where Indicates enhanced features Corresponding key points eigenvectors, Indicates enhanced features Corresponding key points eigenvectors, These are preset coefficients; Key point With key points The cost matrix between features consists of two parts: the first part is the cosine distance of feature similarity, where a smaller value indicates greater feature similarity; the second part is the complement of semantic association strength, determined by coefficients. Control its influence; It is the optimal matching matrix. Indicate key points in drone imagery Key points in satellite base map Matching probability; constraints Ensure that the sum of the matching probabilities of key points meets the preset distribution. and In this embodiment, it is set and It is uniformly distributed.

[0160] The advantage of this optimal transportation problem model is that it can globally consider all potential matching pairs, rather than simply filtering based on a threshold, thereby improving the overall consistency and accuracy of the matching. Semantic constraints are introduced. This makes it more likely that semantically related point pairs will be matched.

[0161] Since directly solving the optimal transportation problem has high computational complexity, the Sinkhorn algorithm is used here to solve its regularized version, reducing computational complexity. The algorithm's process is as follows:

[0162] Initialize the matching matrix ,in For the cost matrix, It's a temperature parameter that controls the softness of the fit; row and column normalization are alternately performed iteratively.

[0163]

[0164] in and It makes The scaling vectors that satisfy the constraints for rows and columns are used to obtain the optimal matching matrix. The Sinkhorn algorithm typically converges within a few dozen iterations, significantly reducing computational complexity and enabling the system to process large numbers of keypoint matches in real time. Parameters The smaller the value, the closer the match is to a hard assignment (one-to-one matching). The larger the value, the softer the match (one-to-many match).

[0165] Obtain the optimal matching matrix Then, through the threshold Filter high-confidence keypoint matching pairs to construct an initial matching set. :

[0166]

[0167] in For the optimal matching matrix The element value in the threshold. By controlling the sparsity and confidence of the matches, a higher threshold yields more reliable but fewer matches, while a lower threshold yields more results but may include false matches.

[0168] The final output of this step is the enhanced feature. and and the initial matching set .

[0169] Step 5: Mismatch filtering based on causal semantic constraints:

[0170] For the initial matching set Local neighborhood consistency verification and global structure verification based on causal semantic relationships are performed separately. The local consistency scores obtained from local neighborhood consistency verification, the global consistency scores obtained from global structure verification, and the initial matching scores of the matching pairs are then weighted and fused. Low-confidence matches are filtered out according to the fusion result to obtain a refined matching set. .

[0171] In this embodiment, the input for step 5 is the initial matching set. Key point set , and causal semantic graph ;

[0172] The processing procedure is as follows:

[0173] For a certain key point matching pair Local neighborhood consistency verification is performed according to the following formula:

[0174]

[0175]

[0176]

[0177] in Indicate key points of Nearest neighbor set, containing neighbors with Distance less than threshold All the key points, Indicate key points of nearest neighbor set It is a distance function; Keypoint matching pairs The local consistency score is the number of matching pairs in each neighborhood that also exist in the initial matching set; the numerator calculates the number of matches between two neighborhoods, the denominator takes the smaller value of the two neighborhood sizes, and the result is normalized to the [0,1] interval.

[0178] This local consistency test is based on the assumption that correct matches should exhibit a spatial clustering effect, meaning that points within the neighborhood of one point should match points within the neighborhood of another point. A high consistency score indicates that the spatial structure around the matched points is consistent, while a low score may indicate a mismatch.

[0179] Similarly, for a certain key point matching pair Perform global structure verification according to the following formula:

[0180]

[0181] in Indicate key points Semantic entities and key points The relation strength weight of the semantic entity to which it belongs. Indicate key points Semantic entities and key points The relation strength weight of the semantic entity to which it belongs; Indicates keypoint matching pairs The confidence level, i.e., the key point matching pair In the optimal matching matrix The value in [the formula] is used to calculate the global structural consistency score. By weighted average and the current matching pair The reliability of other semantically related matching pairs is weighted by the strength of the semantic relationship. This verification mechanism captures a broader range of structural relationships, not limited to spatial proximity, and can detect mismatches that violate semantic constraints but may satisfy local consistency.

[0182] After obtaining the local consistency score and the global structural consistency score, and combining them with the initial matching score, a weighted keypoint matching pair is obtained. fusion results :

[0183]

[0184] in Keypoint matching pairs The initial matching score is taken as its value in the optimal matching matrix. The values ​​that can be taken in the middle, and , and These are weighting coefficients, satisfying... Then based on the threshold Filtering low-confidence matches yields a refined set of matches. :

[0185]

[0186] This multi-constraint filtering mechanism can effectively remove false matches and improve the accuracy of matching.

[0187] The final output of this step is the refined matching set. .

[0188] Step 6: Optimize the matching results using multi-level causal graph convolution:

[0189] First, combine the causal semantic graph For the matching set Construct an enhanced adjacency matrix Then use the adjacency matrix A multi-layer graph convolutional network is applied to optimize the node feature matrix; the optimized node feature matrix is ​​then used to re-evaluate the matching set. Keypoint matching pairs are processed, and matches that do not conform to global semantic consistency are removed to obtain the final matching set. .

[0190] In this embodiment, the input for step 6 is the refined matching set. and causal semantic graph ;

[0191] The processing procedure is as follows:

[0192] Step 6.1: Construct the enhanced adjacency matrix Used for subsequent graph convolution operations:

[0193]

[0194] in Based on matching sets The constructed original adjacency matrix reflects the direct connections between points. Is with matrix Identity matrices of the same size It is a weighting coefficient for causal relationships, controlling the degree of influence of causal knowledge. It is a causal adjacency matrix that captures indirect causal influences between points. The specific formula is:

[0195]

[0196] in In the adjacency matrix representing causal relationships, semantic entities With semantic entities The adjacency relationship is calculated by taking semantic entities into account. To semantic entities All possible paths The edge weights on each path are multiplied together, and then the sum is applied across all paths. This design captures indirect relationships between distant nodes, allowing information to propagate along the causal chain of the semantic graph. express For a certain path, Representing semantic entities With semantic entities For path The two nodes on, semantic entities With semantic entities Relationship strength weight.

[0197] Step 6.2: Utilize the adjacency matrix A multi-layer graph convolutional network is applied to optimize the node feature matrix, where the formula for the node feature matrix is:

[0198]

[0199] in and They represent the number respectively. Layer and first The node feature matrix of the layer, yes The degree matrix, It is the first The learnable weight matrix of the layer controls feature transformation. It is a non-linear activation function, such as ReLU or LeakyReLU.

[0200] This graph convolution operation allows each node to aggregate information about its neighbors, which is represented by an enhanced adjacency matrix. Definition. By stacking multiple layers, nodes can acquire broader contextual information, thereby generating more global feature representations. Normalization term. Ensure that features do not diverge or disappear as the number of layers increases.

[0201] Step 6.3: Further refine the matching results based on the optimized node feature matrix:

[0202]

[0203] function Utilizing Node feature matrix after optimization of layered graph convolutional network Re-evaluate the matching set The confidence scores of the matching pairs are used to eliminate matches that do not conform to global semantic consistency, resulting in the final matching set. In this embodiment, the specific expression is:

[0204]

[0205] in Key point In the node feature matrix The L2 norm of the corresponding eigenvectors in the equation. Key point In the node feature matrix The L2 norm of the corresponding eigenvectors in the equation. The convolution refinement threshold is used to control the strictness of the matching selection; in this embodiment, the value is set to... The advantage of this step is that graph convolutional networks can capture higher-order relationships and global structures between matching points, going beyond simple pairwise matching constraints, thereby further improving the accuracy and robustness of matching.

[0206] The final output of this step is the final matching set. .

[0207] Step 7: Perform location calculation and accuracy assessment based on multi-source information fusion:

[0208] Based on the final matching set Calculate the deviation vector between the actual position and the estimated position of the UAV. ; the deviation vector By fusing the data with inertial navigation data, a fused position estimate is obtained, and a positioning accuracy evaluation index is calculated to achieve UAV position correction and navigation update.

[0209] In this embodiment, the input for step 7 is the final matching set. Initial position estimation provided by inertial navigation system ;

[0210] The processing procedure is as follows:

[0211] Step 7.1: Based on the final matching set Calculate the deviation vector between the actual position and the estimated position of the UAV. :

[0212]

[0213] in For the final matching set The set of coordinate differences between keypoint matching pairs, i.e., the difference between the geographic coordinates of the keypoints in the satellite base image and the projected coordinates of the matching keypoints in the UAV-captured image. It is a coordinate transformation matrix; based on the principle of common-view geometry, the camera position is deduced by inversely inferring the correspondence between matching key points through coordinate transformation.

[0214] Step 7.2: According to the formula:

[0215]

[0216] Optimal fusion of inertial navigation data and scene matching results was achieved, in which... It is the initial position estimate provided by the inertial navigation system. It is the Kalman gain matrix, which is dynamically adjusted based on the uncertainties of the inertial navigation system and the matching results. This is the final position estimate after fusion. The Kalman gain is included. The calculation takes into account the reliability of two information sources: when the inertial navigation system has just been reset or the matching result is very reliable. A larger value indicates a greater inclination to adopt the matching result; conversely, a smaller value indicates a lower inclination to adopt the matching result when the inertial navigation system is relatively accurate in the short term or the matching result is unreliable. If the value is small, adjust the position more conservatively.

[0217] The final output of this step is the fused final position estimate. The final position estimate after fusion. Feedback is sent to the drone navigation system to adjust navigation parameters, such as heading and speed, to update the navigation status and ensure that the drone can safely cruise along the predetermined mission trajectory.

[0218] Experimental comparison:

[0219] Using a coastline scene matching dataset, the performance of the method of this invention is compared with that of several existing methods, and the results are as follows:

[0220] Table 1: Performance Comparison

[0221]

[0222] The anti-interference capability is comprehensively evaluated through matching stability under interference conditions such as strong light reflection, sea fog, and tidal changes. Experimental results show that the method of this invention outperforms existing methods in key indicators such as matching accuracy, matching time, false alarm rate, and positioning error, especially in terms of anti-interference capability, and can effectively cope with the challenges of complex coastal environments.

[0223] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A method for scene matching and localization of an unmanned system in a region rejection scenario, characterized in that: Includes the following steps: Step 1: Real-time acquisition of UAV images by the high-altitude unmanned system Perform geometric correction to obtain the corrected UAV image. ; Corrected drone imagery and pre-stored satellite base maps The equalization process is performed to obtain the processed UAV imagery. and satellite base map Based on the characteristics of the coastline, the processed drone images are... and satellite base map All are divided into sea areas shoreline area and land areas Three sub-regions, with different weights assigned to each region. ; Step 2: Identify drone images and satellite base map The semantic entities in the graph are identified, spatial relationships between them are established, and the relationship strength weights between semantic entities are calculated to construct a causal semantic graph. ; Step 3: Based on the causal semantic graph Generate a keypoint detection mask, where the pixel value at a certain position in the mask represents the priority of that position as a keypoint; Then, keypoint detection masks were used to process the drone images. and satellite base map Keypoint detection based on a self-supervised deep learning architecture is performed to obtain UAV imagery. and satellite base map Each corresponding set of key points and And extract deep feature descriptors of key points. and Finally, establish the association between key points and semantic entities. ; Step 4: Based on causal semantic graph Calculate the semantic association strength between key points; A self-attention-based feature enhancement network is constructed based on semantic association strength. The input features of the feature enhancement network are then fused with external environmental and temporal factors to obtain enhanced features. and ; Utilizing enhanced features and Construct an optimal transportation problem with causal semantic constraints, and obtain the optimal matching matrix by solving the optimal transportation matching problem. Using the optimal matching matrix imagery of drones and satellite base map Filter the key point matching pairs between them to establish an initial matching set. ; Step 5: For the initial matching set Local neighborhood consistency verification and global structure verification based on causal semantic relationships are performed separately. The local consistency scores obtained from local neighborhood consistency verification, the global consistency scores obtained from global structure verification, and the initial matching scores of the matching pairs are then weighted and fused. Low-confidence matches are filtered out according to the fusion result to obtain a refined matching set. ; Step 6: Combine causal semantic graphs For the matching set Construct an enhanced adjacency matrix Then use the adjacency matrix A multi-layer graph convolutional network is applied to optimize the node feature matrix; the optimized node feature matrix is ​​then used to re-evaluate the matching set. Keypoint matching pairs are processed, and matches that do not conform to global semantic consistency are removed to obtain the final matching set. ; Step 7: Based on the final matching set Calculate the deviation vector between the actual position and the estimated position of the UAV. ; the deviation vector The data is fused with inertial navigation data to obtain a fused position estimate.

2. The scene matching and localization method for an unmanned system in a region rejection scenario according to claim 1, characterized in that: In step 1, the region weights ,in As the weight of the shoreline area, For land area weights, As the weight of the sea area, and .

3. The scene matching and localization method for an unmanned system in a region rejection scenario according to claim 1, characterized in that: In step 2, the causal semantic graph ,in A collection of nodes with semantic entities as nodes. For a set of nodes There exists a set of edges consisting of the relationship between two semantic entities with a clear spatial association. For a set of spatial relationships The set of edge weights, which consists of the strength weights of each relation.

4. The scene matching and localization method for an unmanned system in a region rejection scenario according to claim 2, characterized in that: In step 2, according to the formula Computational semantic entities and The strength weight of the relationship between them; where The function is a stability function, used to evaluate the stability of relationships between semantic entities. The function is a persistence function used to evaluate the degree of change of a relationship over time. The function is a distinctiveness function, used to evaluate the visual distinctiveness and uniqueness of a relationship. The function is a relation weight function, taking... in As an indicator variable for the shoreline area, For land area indicator variables, This is an indicator variable for the sea area.

5. The scene matching and localization method for an unmanned system in a region rejection scenario according to claim 1, characterized in that: In step 3, according to the formula Generate keypoint detection mask ,in Indicates position The guide mask value at a given location indicates the priority of that location as a keypoint. This is an indicator function; it returns 1 if the condition within the parentheses is true, and 0 otherwise. Representing semantic entities In satellite base map The spatial area covered in it semantic entities Importance weights.

6. The scene matching and localization method for an unmanned system in a region rejection scenario according to claim 1, characterized in that: In step 4, according to the formula Calculate the key point set based on causal semantic graph. A key point in With key point set A key point in Semantic association strength between ,in It is an indicator function, when Returns 1 if the condition is true, otherwise returns 0; Indicate key points With semantic entities There is a mapping relationship. Indicate key points With semantic entities There is a mapping relationship. semantic entities With semantic entities Relationship strength weight.

7. The scene matching and localization method for an unmanned system in a region rejection scenario according to claim 6, characterized in that: In step 4, enhanced features are utilized. and The optimal transportation problem with causal semantic constraints is constructed as follows: in Indicates enhanced features Corresponding key points eigenvectors, Indicates enhanced features Corresponding key points eigenvectors, These are preset coefficients; Key point With key points The cost matrix between It is the optimal matching matrix. Indicate key points in drone imagery Key points in satellite base map Matching probability; constraints Ensure that the sum of the matching probabilities of key points meets the preset distribution. and .

8. The scene matching and localization method for an unmanned system in a region rejection scenario according to claim 6, characterized in that: In step 5, for the initial matching set Key point matching pairs According to the formula Computational fusion results ,in , and These are weighting coefficients, satisfying... , Keypoint matching pairs The initial matching score, Keypoint matching pairs Local consistency score, Keypoint matching pairs The global structural consistency score.

9. The scene matching and localization method for an unmanned system in a region rejection scenario according to claim 6, characterized in that: In step 6, the formula for the node feature matrix is: in and They represent the number respectively. Layer and first The node feature matrix of the layer, It is an enhanced adjacency matrix The degree matrix, It is the first The learnable weight matrix of the layer, It is a non-linear activation function; the enhanced adjacency matrix According to the formula Received, among which Based on matching sets The constructed original adjacency matrix, Is with matrix Identity matrices of the same size These are the weighting coefficients for causal relationships. It is a causal adjacency matrix.

10. The scene matching and localization method for an unmanned system in a region rejection scenario according to claim 9, characterized in that: In step 6, according to the formula Get the final matching set ,in Key point In the node feature matrix The L2 norm of the corresponding eigenvectors in the equation. Key point In the node feature matrix The L2 norm of the corresponding eigenvectors in the equation. The threshold for graph convolution refinement.

Images