Cross-view image positioning method and system based on semantic perception graph convolutional network

Abstract

The invention belongs to the technical field of computer vision, and provides a cross-view image positioning method and system based on a semantic perception graph convolutional network, and the method comprises the steps: obtaining the multi-scale global features of each image (including an unmanned plane image and a plurality of satellite images) through a network parameter shared dual-branch feature extraction network; according to the multi-scale global features and the basis vector matrix, a semantic attention graph of each image and an attention sub-graph of the last layer are obtained through a multi-layer cascaded cross attention mechanism; constructing a dynamic adjacency matrix through a semantic attention graph, and obtaining topological structure features of each image by adopting a two-layer graph convolutional neural network; and calculating the cosine similarity between the unmanned aerial vehicle image and each satellite image through the multi-scale global features and the topological structure features, and obtaining the coordinate of the satellite image with the maximum cosine similarity as the coordinate of the man-machine image. According to the invention, high-precision and high-robustness autonomous positioning technical support can be provided for an unmanned aerial vehicle system.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a cross-view image localization method and system based on semantically perceptive graph convolutional networks. Background Technology

[0002] Unmanned aerial vehicle (UAV) systems are widely used in logistics and urban surveying. However, in complex environments such as extreme weather or electromagnetic interference, the Global Navigation Satellite System (GNSS) is prone to signal failure, which can cause UAVs to lose their geospatial positioning capabilities and thus affect equipment operation.

[0003] Currently, autonomous localization methods based on cross-view image matching technology provide a new technical approach to solving this problem. However, due to the differences in viewpoints, geometric deformations, and weakened semantic associations between UAV and satellite images, traditional localization methods suffer from insufficient accuracy. Summary of the Invention

[0004] To address the issue of insufficient localization accuracy in traditional cross-view image matching-based autonomous localization methods, this invention discloses a cross-view image localization method based on semantically aware graph convolutional networks. This method can provide high-precision and robust autonomous localization support for unmanned aerial vehicle (UAV) systems. The method includes the following steps:

[0005] S1. The multi-scale global features of each image are obtained through the SwinV2-T dual-branch feature extraction network with shared network parameters. The images include UAV images and multiple satellite images.

[0006] S2. Based on the multi-scale global features and basis vector matrix, obtain the attention subgraph of each layer through a multi-layer cascaded cross attention mechanism, and obtain the semantic attention graph of each image through all the attention subgraphs;

[0007] S3. Construct a dynamic adjacency matrix using the semantic attention graph, and use a two-layer graph convolutional neural network to obtain the topological features of each image;

[0008] S4. Calculate the cross-view cosine similarity between the UAV image and each of the satellite images using the multi-scale global features and the topological features, and obtain the coordinates of the satellite image with the largest cross-view cosine similarity as the coordinates of the human-machine image.

[0009] Furthermore, in step S1, each branch of the SwinV2-T dual-branch feature extraction network adopts a four-level pyramid Transformer structure, and the two branches share network parameters.

[0010] Further, in step S2, based on the multi-scale global features and basis vector matrix, attention subgraphs for each layer are obtained through a multi-layer cascaded cross-attention mechanism, and semantic attention maps for each image are obtained through all the attention subgraphs, including:

[0011] S21. In the multi-layer cascaded cross-attention mechanism, the input feature of the first layer is the multi-scale global feature, and the input features of the remaining layers are the semantic features output by the previous layer.

[0012] S22. Obtain the semantic category query matrix for each layer by performing residual connections between the basis vector matrix and the input features;

[0013] S23. Based on the semantic category query matrix, the learnable scaling factor, and the temperature coefficient, the attention subgraph of each layer is calculated using the scaling cosine similarity method. The semantic features of each layer are generated using the attention subgraph and the input features.

[0014] S24. A semantic attention graph is obtained by weighted fusion of all the attention subgraphs and the attention weight coefficients of each layer.

[0015] Furthermore, in steps S21-S24, the expression for the input features of each layer in the multi-layered cascaded cross-attention mechanism is as follows:

[0016] in, For the input features of the t-th layer, F global For multi-scale global features, Let N be the semantic features output by the (t-1)th layer, and N be the number of layers.

[0017] The formula for calculating the semantic category query matrix is ​​as follows: Among them, Q (t) Let Q be the semantic category query matrix for layer t. base Let be a basis vector matrix, and Linear(·) denotes a fully connected operation;

[0018] The formula for calculating the attention subgraph is: Among them, A (t) Let be the attention subgraph of the t-th layer, γ be the learnable scaling factor, and τ be the temperature coefficient;

[0019] The expression for semantic features is: in, The semantic features output by the t-th layer;

[0020] The formula for calculating the semantic attention graph is: Where A is the semantic attention graph, λ (t) Let be the learnable attention weight coefficients for the t-th layer.

[0021] Further, in step S3, a dynamic adjacency matrix is ​​constructed using the semantic attention graph, and a two-layer graph convolutional neural network is used to obtain the topological features of each image, including:

[0022] S31. Obtain the adjacency matrix of each semantic instance through the semantic vector of the semantic instance in the semantic attention graph and its transpose, and calculate the sum of all the adjacency matrices to obtain the total adjacency matrix.

[0023] S32. Using the total adjacency matrix, the attention subgraph of the last layer, and the learnable parameters of the first layer, the topological features of the first layer are obtained through the first layer graph convolutional neural network.

[0024] S33. Using the topological features of the first layer, the total adjacency matrix, and the learnable parameters of the second layer, the topological features are obtained through the second layer graph convolutional neural network.

[0025] Furthermore, step S31 above also includes:

[0026] The sum of all the aforementioned adjacency matrices is symmetrically normalized to obtain the final total adjacency matrix.

[0027] Further, in step S4, the cross-view cosine similarity between the UAV image and each of the satellite images is calculated using the multi-scale global features and the topological features, including:

[0028] S41. Perform channel splicing fusion and global average pooling on the multi-scale global features and topological features of each image to obtain pooled composite features.

[0029] S42. Normalize each of the pooled composite features to obtain a normalized feature vector, and then use the formula... Calculate the cross-view cosine similarity between the UAV image and the satellite image, where Similarity is the cross-view cosine similarity. This is the normalized feature vector of a satellite image. This is the normalized feature vector of the UAV image.

[0030] This invention also provides a cross-view image localization system based on a semantically aware graph convolutional network, comprising: a dual-branch feature extraction module, a multi-layer cascaded semantic inference module, a dynamic graph convolutional structure module, and a feature fusion and matching module.

[0031] The dual-branch feature extraction module is used to obtain multi-scale global features of each image through the SwinV2-T dual-branch feature extraction network with shared network parameters. The images include UAV images and multiple satellite images.

[0032] The multi-layer cascaded semantic inference module is used to obtain the attention subgraph of each layer through a multi-layer cascaded cross-attention mechanism based on the multi-scale global features and basis vector matrix, and to obtain the semantic attention map of each image through all the attention subgraphs.

[0033] The dynamic graph convolutional structure module is used to construct a dynamic adjacency matrix through the semantic attention graph and to obtain the topological features of each image using a two-layer graph convolutional neural network.

[0034] The feature fusion and matching module is used to calculate the cross-view cosine similarity between the UAV image and each of the satellite images using the multi-scale global features and the topological features, and to obtain the coordinates of the satellite image with the largest cross-view cosine similarity as the coordinates of the human-machine image.

[0035] Furthermore, each branch in the SwinV2-T dual-branch feature extraction network adopts a four-level pyramid Transformer structure, and the two branches share network parameters.

[0036] Furthermore, the multi-layer cascaded semantic inference module is a four-layer cascaded semantic inference module.

[0037] This invention discloses a cross-view image localization method and system based on a semantically aware graph convolutional network. First, it constructs a shared-weights-based SwinV2-T dual-branch feature extraction backbone network to process images layer by layer, extracting multi-scale global features. Second, it analyzes the contextual relationships of each multi-scale global feature through a multi-layered cascaded Cross-Attention layer of a Transformer, inferring the latent semantic probability distribution to obtain a semantic attention map. Then, it constructs a dynamic adjacency matrix based on the semantic probability distribution and extracts topological features between semantic elements through two-layer graph convolution operations. Finally, it fuses the multi-scale global features and topological features through channel concatenation, forming a composite feature that combines high-level semantic understanding and spatial relationship modeling. The composite feature is then average-pooled to calculate the distance between cross-view image composite features, outputting the scene matching probability (i.e., cross-view cosine similarity). The coordinates of the satellite image with the highest scene matching probability are extracted as the location of the UAV image. This invention provides high-precision and robust autonomous localization technology support for UAV systems. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart of a cross-view image localization method based on semantically aware graph convolutional networks;

[0040] Figure 2 This is a block diagram illustrating the principle of a cross-view image localization method based on semantically aware graph convolutional networks.

[0041] Figure 3 This is a visualization of the principal component analysis (PCA) of cross-view image features using the method of the present invention.

[0042] Figure 4 This is an architecture diagram of a cross-view image localization system based on a semantically aware graph convolutional network.

[0043] Among them, 401 is the dual-branch feature extraction module; 402 is the multi-layer cascaded semantic inference module; 403 is the dynamic graph convolutional structure module; and 404 is the feature fusion and matching module. Detailed Implementation

[0044] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0045] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features of the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] This invention provides a cross-view image localization method based on semantically aware graph convolutional networks. See [link to relevant documentation]. Figure 1 and Figure 2 As shown, the method includes the following steps:

[0047] S1. The multi-scale global features of each image are obtained through the SwinV2-T dual-branch feature extraction network with shared network parameters. The images include UAV images and multiple satellite images.

[0048] S2. Based on the multi-scale global features and basis vector matrix, obtain the attention subgraph of each layer through a multi-layer cascaded cross attention mechanism, and obtain the semantic attention graph of each image through all the attention subgraphs;

[0049] S3. Construct a dynamic adjacency matrix using the semantic attention graph, and use a two-layer graph convolutional neural network to obtain the topological features of each image;

[0050] S4. Calculate the cross-view cosine similarity between the UAV image and each of the satellite images using the multi-scale global features and the topological features, and obtain the coordinates of the satellite image with the largest cross-view cosine similarity as the coordinates of the human-machine image.

[0051] Further, in step S1, the global feature extraction neural network uses a pre-trained SwinV2-T dual-branch feature extraction network as its backbone. SwinV2-T is a lightweight version of the SwinV2 series, with approximately 28 million parameters and a computational cost of approximately 4.5 GFLOPs with a 224×224 input. It is designed for resource-constrained scenarios and is suitable for deployment in embedded devices such as drones. In this invention, the drone and satellite branches use the same backbone network parameters. Through parameter sharing, feature shifts caused by perspective differences can be eliminated, mapping images from different perspectives to the same feature space. The SwinV2-T dual-branch feature extraction network removes the original classification head, retaining only a four-level pyramid Transformer structure. The parameter configurations for each stage are shown in Table 1 below.

[0052] Table 4: Parameter configuration of each network layer in the SwinV2-T dual-branch feature extraction network

[0053] stage number of floors resolution Number of channels Window size Number of attention heads 1 2 64×64 96 8×8 3 2 2 32×32 192 8×8 6 3 6 16×16 384 8×8 12 4 2 8×8 768 8×8 24

[0054] When performing global feature extraction, the corresponding images are input into each branch of the SwinV2-T dual-branch feature extraction network for processing. Taking UAV images as an example, the input UAV image I∈R H×W×3 Divided into Image patches of size H = W = 256 (where H is the vertical resolution and W is the horizontal resolution) are linearly embedded into 96-dimensional feature vectors, then processed through a four-level pyramid hierarchy to output a multi-scale global feature vector F. global ∈R B×C×L (Where B represents batch number 32, and C represents channel number 768) As shown in Formula 4-1:

[0055] F global =SwinV2(I) (4-1).

[0056] Generally, satellite images and drone images taken of the same scene should have consistent semantic distributions. This consistency in semantic distribution is significant for identifying whether a set of satellite and drone images belong to the same scene. This invention analyzes the contextual associations of global feature maps through a cross-attention mechanism to infer potential semantic distributions. By constructing multi-layered cascaded cross-attention layers, each layer contains multiple dynamic semantic category query vectors generated based on the output of preceding layers. In this structure, the lower-level network prioritizes capturing local geometric features (such as building edges and road directions), while the higher-level network gradually aggregates large-scale semantic information (such as regional functional distribution and terrain structure). Finally, by weighted fusion of the multi-layered attention results, a more stable semantic probability distribution is generated, thereby improving the perception capability of multi-granular semantic information. Specifically, in step S2, based on the multi-scale global features and basis vector matrix, attention sub-graphs of each layer are obtained through a multi-layered cascaded cross-attention mechanism. The semantic attention map of each image is obtained through all the attention sub-graphs, including:

[0057] S21. In the multi-layer cascaded cross-attention mechanism, the input feature of the first layer is the multi-scale global feature, and the input features of the remaining layers are the semantic features output by the previous layer.

[0058] For an N-layer cascaded cross-attention layer, the input features of the t-th (t=1,2,…,N) layer... The expression is shown in Formula 4-2:

[0059] For the input features of the t-th layer, F global For multi-scale global features, Let N be the semantic features output by the (t-1)th layer, and N be the layer number.

[0060] S22. Obtain the semantic category query matrix for each layer by performing residual connection between the basis vector matrix and the input features.

[0061] The formula for calculating the semantic category query matrix is ​​shown in equation (4-3) below:

[0062] Q (t) Let Q be the semantic category query matrix for layer t. (t) ∈R B×C×M Q baseThe basis vector matrix contains the basis vectors of each of the M semantic categories. Linear(·) represents the fully connected operation.

[0063] S23. Based on the semantic category query matrix, the learnable scaling factor, and the temperature coefficient, the attention subgraph of each layer is calculated using the scaling cosine similarity method. The semantic features of each layer are generated using the attention subgraph and the input features.

[0064] Attention subgraphs can be calculated using the following equation 4-1:

[0065] A (t) Let A be the attention subgraph of layer t. (t) ∈R B×M×L γ is a learnable scaling factor, and τ is a temperature coefficient.

[0066] The semantic features of each layer are represented by the following equation 4-5:

[0067] Let t be the semantic features output by the t-th layer.

[0068] S24. A semantic attention graph is obtained by weighted fusion of all the attention subgraphs and the attention weight coefficients of each layer.

[0069] The semantic attention map is obtained through Softmax weighted fusion and can be represented by the following equation 4-6: Where A is the semantic attention graph, A∈R B×M×L , λ (t) Let be the learnable attention weight coefficients for the t-th layer.

[0070] Furthermore, during the dynamic graph convolution operation in step S3, the convolution strategy is extended to graph structures by constructing a semantically aware dynamic adjacency matrix that allows propagation between nodes. The adjacency matrix is ​​constructed based on the multi-scale global features obtained in step S1. Its core idea is to dynamically define node connection strength according to the attention distribution, leveraging the topological invariance of the graph structure to improve the robustness of features to viewpoint changes and geometric deformations, thus solving the problem that traditional CNNs struggle to model long-distance dependencies. Simultaneously, compared to the fixed graph structure of traditional KNN, the attention-driven adjacency matrix can adapt to the input content and, through multi-instance fusion, can simultaneously capture the mixed patterns of building outlines and regional functional distributions.

[0071] Since the adjacency matrix is ​​dynamically generated through a cross-attention mechanism, it essentially encodes cross-level semantic association information. The attention weights automatically capture the distribution patterns of different semantic instances in the spatial dimension, making the adjacency matrix itself contain the multi-order association information required for iterative propagation in traditional multi-layer GCNs. This pre-encoding characteristic allows shallow networks to achieve the information fusion effect of deep topologies. Therefore, this invention employs a lightweight structure containing only two graph convolutional layers to perceive semantic structural features, ensuring network computational efficiency and stability. Specifically, this step includes the following steps:

[0072] S31. Obtain the adjacency matrix of each semantic instance by using the semantic vector of the semantic instance in the semantic attention graph and its transpose, and calculate the sum of all the adjacency matrices to obtain the total adjacency matrix.

[0073] For the m-th semantic instance, compute its attention-related adjacency matrix Adj. m ∈R B×L×L As shown in Formula 4-7:

[0074]

[0075] Among them, A :,m,: Let m be the semantic vector of the m-th semantic instance in the semantic attention graph. Let Adj be the transpose of the semantic vector of the m-th semantic instance in the semantic attention graph. m Let be the adjacency matrix of the m-th semantic instance.

[0076] The total adjacency matrix Adj∈R for each image B×L×L The sum of the adjacency matrices of all its semantic instances is shown in Equation 4-8:

[0077]

[0078] Furthermore, to measure node importance, this invention also normalizes the total adjacency matrix. Specifically, a degree matrix D is introduced to symmetrically normalize the total adjacency matrix, where each element D in the degree matrix D... ii This represents the total number of connections between node i and other nodes, as shown in Formula 4-9:

[0079] D ii =∑ j Adj ij (4-9), where Adj ij This represents the connection between node i and node j. It is usually 1 if there is a connection between node i and node j, and zero if there is no connection.

[0080] This invention is achieved through Scaling the adjacency matrix solves the problem of uneven degree distribution and prevents gradient explosion or vanishing, as shown in Equation 4-10:

[0081]

[0082] Among them, Adj norm ∈R B×L×L This represents the normalized adjacency matrix.

[0083] S32. Using the total adjacency matrix, the attention subgraph of the last layer, and the learnable parameters of the first layer, the topological features of the first layer are obtained through the first-layer graph convolutional neural network. The topological features of the first layer can be represented by the following formula 4-11:

[0084] W1∈R C×C Here are the first layer of learnable parameters, and GELU(·) is the activation function. This represents the first layer of topological features. This is the attention subgraph for the last layer.

[0085] S33. Using the first layer topological structure features, the total adjacency matrix, and the second layer learnable parameters, the topological structure features are obtained through the second layer graph convolutional neural network. The second layer topological structure features can be represented by the following formula 4-12:

[0086] W2∈R C×C Here are the learnable parameters for the second layer, and GELU(·) is the activation function. This is the second-layer topological structure feature, which is also the final generated topological structure feature.

[0087] Further, in step S4, the cross-view cosine similarity between the UAV image and each of the satellite images is calculated using the multi-scale global features and the topological features, including:

[0088] S41. Perform channel splicing fusion and global average pooling on the multi-scale global features and topological features of each image to obtain pooled composite features.

[0089] In practical implementation, since the semantic structure features generated by graph convolution and the original global feature tensor generated by SwinV2-T are both 768×64 in size, feature fusion can be performed without interpolation. This invention employs a simple and effective method to directly fuse the global features F obtained in step S1. global Topological features of step S3 The composite features are generated by concatenating along the channel dimension. These composite features are then subjected to global average pooling to compress the spatial dimension, as shown in equations 4-13 and 4-14.

[0090]

[0091] F pooled =AdaptiveAvgPool2d(1)(F fused )#(4-14);

[0092] Where Concat(·) is the channel-level concatenation operation, F fused ∈R B×2C×L For composite features, AdaptiveAvgPool2d(1)(·) represents global average pooling, F pooled ∈R B×2C This represents the composite feature after pooling.

[0093] S42. Normalize each pooled composite feature to obtain a normalized feature vector, and calculate the cross-view cosine similarity between the UAV image and the satellite image.

[0094] In practical implementation, due to significant modal differences between UAV images and satellite images (UAV images typically contain rich local details, while satellite images focus on global layout), traditional single-branch classifiers struggle to simultaneously capture discriminative features of both modalities. Therefore, this invention proposes a dual-branch classifier, employing a divide-and-conquer strategy to improve model performance. Specifically, two independent classifier branches are set up: one for UAVs and one for satellites. The UAV branch optimizes local detail recognition, while the satellite branch focuses on global structure analysis. Both branches share a basic feature extraction layer to learn common information, but each learns its own modality-specific discriminative mode through independent parameters. This parameter isolation mechanism preserves cross-modal general representation capabilities while performing specific optimizations for different imaging characteristics, achieving a balance between feature representation accuracy and cross-modal generalization.

[0095] The pooled composite features are L2 normalized to calculate the cross-view cosine similarity, as shown in formulas 4-15 and 4-16:

[0096]

[0097] Among them, F norm ∈R B×2C The L2-normalized feature vector has a magnitude of 1; and These represent the normalized composite features of a satellite image and a drone image, respectively. Similarity represents the cross-view cosine similarity. This is the normalized feature vector of a satellite image. This is the normalized feature vector of the UAV image.

[0098] This invention uses SUES-200 as the experimental dataset. SUES-200 is an authoritative benchmark in the field of cross-view geolocation and is widely used in experimental verification of cross-view image matching. The statistical information of the dataset is shown in Table 5-1.

[0099] Table 5-1: Comparison of basic attributes between the SUES-200 and University-1652 datasets

[0100]

[0101]

[0102] This invention selects Recall@K and average precision (AP) as performance metrics to evaluate the proposed method. Recall@K represents the proportion of true matching images in the top K ranked matching results; a higher score indicates better method performance. In this invention, K is set to 1 to evaluate the ability to hit true matches in the first recommended result, which meets the requirements of precise UAV positioning. Average precision (AP) is the area under the precision-recall (PR) curve, and its value is also positively correlated with method performance. The combination of average precision (AP) and Recall@K reveals the model's ability to balance "precise matching" and "fault tolerance."

[0103] When performing cross-view geolocation tasks, this invention constructs a phased image preprocessing workflow, and preprocesses the SUES-200 dataset using the following steps:

[0104] (1) Data augmentation during training phase

[0105] Data augmentation during the training phase employs a combination of multimodal geometry and photometric transformation strategies to improve the model's robustness to viewpoint differences. The specific transformations are as follows:

[0106] ① Random scaling and cropping: Randomly crop 0% to 20% of the original image and scale it to a fixed size of 256×256 to simulate the scale changes of the target at different flight altitudes.

[0107] ② Random rotation transformation: Uniform rotation is performed within an angle range of ±30°.

[0108] ③ Random perspective distortion: Set the distortion rate to 0.2 to simulate the change in the perspective of drone aerial photography.

[0109] ④ Fill-Crop Combination: First, perform a 10-pixel random fill, then perform a deterministic crop to preserve the main structure.

[0110] ⑤ Photometric normalization: Perform channel-level normalization based on ImageNet dataset statistics.

[0111] (2) Data standardization during the verification and testing phase

[0112] To ensure consistency in evaluation, deterministic preprocessing is used on both the validation and test sets:

[0113] ① Size normalization: Bilinear interpolation scales the image proportionally to a fixed resolution of 256×256.

[0114] ② Spatial alignment: Center clipping eliminates redundant boundary information, ensuring strict uniformity of input dimensions.

[0115] ③ Normalization: Use the same mean and standard deviation parameters as in the training phase.

[0116] For the preprocessed SUES dataset, this invention selected five methods—SUES-200 baseline model, LCM, LPN, FSRA, and MCCG—for comparative experiments. The results are shown in Table 6-1. Under the same dataset partitioning and experimental parameter settings, the results show that, except for a slightly lower Recall@1 at 300m altitude compared to MCCG, the method of this invention achieved the best results at the other three flight altitudes, with an average improvement of 0.89% across all altitudes (the average Recall@1 of this invention is 91.01%, while the average Recall@1 of existing methods is 90.12%). The AP at 150m, 200m, 250m, and 300m was improved by 4.16%, 2.82%, 1.13%, and 0.25% respectively compared to the state-of-the-art (SOTA) model. Experimental results indicate that this invention has strong semantic structural information modeling capabilities, can better adapt to drastic changes in viewpoint, and possesses unique advantages for low-altitude UAV image matching.

[0117] Table 6-1: Performance Comparison of SUES-200 Dataset with Different Cross-View Geolocation Methods

[0118]

[0119] Furthermore, this invention also utilizes the feature extraction capabilities of the PCA plot analysis model. Principal Component Analysis (PCA) plots, by projecting high-dimensional features onto a two- or three-dimensional space, can visually display the distribution patterns of different samples in the feature space. Each point in the plot represents a sample, its length direction represents the maximum variance distribution of the data, and color or shape distinguishes different categories. If similar sample points are closely clustered and dissimilar samples have significant spacing, it indicates that the features extracted by the model have strong discriminative power; conversely, a mixed distribution suggests insufficient feature discrimination.

[0120] like Figure 3As shown, the features extracted by the SwinV2-T dual-branch architecture through network parameter sharing in this invention exhibit significant intra-class aggregation and inter-class separation characteristics in the low-dimensional space. Drone-satellite feature points (of the same color) in similar scenes form compact clusters in the two-dimensional space, while clusters in dissimilar scenes (of different colors) show significant spacing. This geometric consistency across the cross-view feature space verifies the effectiveness of semantically aware graph convolution in eliminating modal differences.

[0121] Based on the same inventive concept, this invention also provides a cross-view image localization system based on semantically aware graph convolutional networks, as described in the following embodiments. Since the principle of the cross-view image localization system based on semantically aware graph convolutional networks is similar to the cross-view image localization method based on semantically aware graph convolutional networks disclosed in the above embodiments, the implementation of the cross-view image localization system based on semantically aware graph convolutional networks can refer to the implementation of the cross-view image localization method based on semantically aware graph convolutional networks disclosed in the above embodiments, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0122] Figure 4 This is a structural block diagram of a cross-view image localization system based on a semantically aware graph convolutional network disclosed in an embodiment of the present invention, such as... Figure 4 As shown, the structure includes a dual-branch feature extraction module 401, a multi-layer cascaded semantic inference module 402, a dynamic graph convolutional structure module 403, and a feature fusion and matching module 404. The structure is described below.

[0123] The dual-branch feature extraction module 401 is used to obtain multi-scale global features of each image through the SwinV2-T dual-branch feature extraction network with shared network parameters. The images include UAV images and multiple satellite images.

[0124] The multi-layer cascaded semantic inference module 402 is used to obtain the attention subgraph of each layer through a multi-layer cascaded cross-attention mechanism based on the multi-scale global features and basis vector matrix, and to obtain the semantic attention map of each image through all the attention subgraphs.

[0125] The dynamic graph convolutional structure module 403 is used to construct a dynamic adjacency matrix through the semantic attention graph and to obtain the topological features of each image using a two-layer graph convolutional neural network.

[0126] The feature fusion and matching module 404 is used to calculate the cross-view cosine similarity between the UAV image and each of the satellite images through the multi-scale global features and the topological features, and to obtain the coordinates of the satellite image with the largest cross-view cosine similarity as the coordinates of the human-machine image.

[0127] Furthermore, each branch in the SwinV2-T dual-branch feature extraction network adopts a four-level pyramid Transformer structure, and the two branches share network parameters.

[0128] Furthermore, the multi-layer cascaded semantic inference module 402 is a four-layer cascaded semantic inference module.

[0129] This invention discloses a cross-view image localization method and system based on a semantically aware graph convolutional network. First, it constructs a shared-weights-based SwinV2-T dual-branch feature extraction backbone network to process images layer by layer, extracting multi-scale global features. Second, it analyzes the contextual relationships of each multi-scale global feature through a multi-layered cascaded Cross-Attention layer of a Transformer, inferring the latent semantic probability distribution to obtain a semantic attention map. Then, it constructs a dynamic adjacency matrix based on the semantic probability distribution and extracts topological features between semantic elements through two-layer graph convolution operations. Finally, it fuses the multi-scale global features and topological features through channel concatenation, forming a composite feature that combines high-level semantic understanding and spatial relationship modeling. The composite feature is then average-pooled to calculate the distance between cross-view image composite features, outputting the scene matching probability (i.e., cross-view cosine similarity). The coordinates of the satellite image with the highest scene matching probability are extracted as the location of the UAV image. This invention provides high-precision and robust autonomous localization technology support for UAV systems.

[0130] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

[0131] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cross-view image localization method based on semantically aware graph convolutional networks, characterized in that, include: Multi-scale global features of each image are obtained through a SwinV2-T dual-branch feature extraction network with shared network parameters. The images include UAV images and multiple satellite images. Based on the multi-scale global features and basis vector matrix, attention subgraphs of each layer are obtained through a multi-layer cascaded cross-attention mechanism, and semantic attention maps of each image are obtained through all the attention subgraphs. A dynamic adjacency matrix is ​​constructed using the semantic attention graph, and a two-layer graph convolutional neural network is used to obtain the topological features of each image. Using the multi-scale global features and the topological features, the cross-view cosine similarity between the UAV image and each of the satellite images is calculated, and the coordinates of the satellite image with the largest cross-view cosine similarity are obtained as the coordinates of the human-machine image.

2. The cross-view image localization method based on semantically aware graph convolutional networks according to claim 1, characterized in that, Each branch in the SwinV2-T dual-branch feature extraction network adopts a four-level pyramid Transformer structure, and the two branches share network parameters.

3. The cross-view image localization method based on semantically aware graph convolutional networks according to claim 1, characterized in that, Based on the multi-scale global features and basis vector matrix, an attention subgraph for each layer is obtained through a multi-layered cascaded cross-attention mechanism. A semantic attention map for each image is then obtained from all the attention subgraphs, including: In the multi-layer cascaded cross-attention mechanism, the input feature of the first layer is defined as the multi-scale global feature, and the input features of the remaining layers are the semantic features output by the previous layer. By performing a residual connection between the basis vector matrix and the input features, the semantic category query matrix of each layer is obtained; Based on the semantic category query matrix, the learnable scaling factor, and the temperature coefficient, the scaling cosine similarity method is used to calculate the attention subgraph of each layer. The semantic features of each layer are generated through the attention subgraph and the input features. A semantic attention graph is obtained by weighted fusion of all the attention subgraphs and the attention weight coefficients of each layer.

4. The cross-view image localization method based on semantically aware graph convolutional networks according to claim 3, characterized in that, The expression for the input features of each layer in a multi-layered cascaded cross-attention mechanism is as follows: in, For the input features of the t-th layer, F global For multi-scale global features, Let N be the semantic features output by the (t-1)th layer, and N be the number of layers. The formula for calculating the semantic category query matrix is ​​as follows: Among them, Q (t) Let Q be the semantic category query matrix for layer t. base Let be a basis vector matrix, and Linear(·) denotes a fully connected operation; The formula for calculating the attention subgraph is: Among them, A (t) Let be the attention subgraph of the t-th layer, γ be the learnable scaling factor, and τ be the temperature coefficient; The expression for semantic features is in, The semantic features output by the t-th layer; The formula for calculating the semantic attention graph is: Where A is the semantic attention graph, λ (t) Let be the learnable attention weight coefficients for the t-th layer.

5. The cross-view image localization method based on semantically aware graph convolutional networks according to claim 1, characterized in that, A dynamic adjacency matrix is ​​constructed using the semantic attention map, and a two-layer graph convolutional neural network is used to obtain the topological features of each image, including: By using the semantic vectors of semantic instances in the semantic attention graph and their transposes, the adjacency matrix of each semantic instance is obtained, and the sum of all the adjacency matrices is calculated to obtain the total adjacency matrix. The first layer topological features are obtained through the total adjacency matrix, the attention subgraph of the last layer, and the learnable parameters of the first layer, using the first layer graph convolutional neural network. The topological features are obtained through the first layer topological features, the total adjacency matrix, and the second layer learnable parameters, using a second-layer graph convolutional neural network.

6. The cross-view image localization method based on semantically aware graph convolutional networks according to claim 5, characterized in that, Also includes: The sum of all the aforementioned adjacency matrices is symmetrically normalized to obtain the final total adjacency matrix.

7. The cross-view image localization method based on semantically aware graph convolutional networks according to claim 1, characterized in that, Calculating the cross-view cosine similarity between the UAV image and each of the satellite images using the multi-scale global features and the topological features, including: Channel splicing and global average pooling are performed on the multi-scale global features and topological features of each image to obtain pooled composite features. Each pooled composite feature is normalized to obtain a normalized feature vector, which is then processed using the formula... Calculate the cross-view cosine similarity between the UAV image and the satellite image, where Similarity is the cross-view cosine similarity. This is the normalized feature vector of a satellite image. This is the normalized feature vector of the UAV image.

8. A cross-view image localization system based on a semantically aware graph convolutional network, characterized in that, include: A dual-branch feature extraction module is used to obtain multi-scale global features of each image through the SwinV2-T dual-branch feature extraction network with shared network parameters. The images include UAV images and multiple satellite images. A multi-layer cascaded semantic inference module is used to obtain the attention subgraph of each layer through a multi-layer cascaded cross-attention mechanism based on the multi-scale global features and basis vector matrix, and to obtain the semantic attention map of each image through all the attention subgraphs. A dynamic graph convolutional structure module is used to construct a dynamic adjacency matrix through the semantic attention graph and to obtain the topological features of each image using a two-layer graph convolutional neural network. The feature fusion and matching module is used to calculate the cross-view cosine similarity between the UAV image and each of the satellite images using the multi-scale global features and the topological features, and to obtain the coordinates of the satellite image with the largest cross-view cosine similarity as the coordinates of the human-machine image.

9. The cross-view image localization system based on semantically aware graph convolutional networks according to claim 8, characterized in that, Each branch in the SwinV2-T dual-branch feature extraction network adopts a four-level pyramid Transformer structure, and the two branches share network parameters.

10. The cross-view image localization system based on semantically aware graph convolutional networks according to claim 8, characterized in that, The multi-layer cascaded semantic inference module is a four-layer cascaded semantic inference module.

Images