A cross-view image matching method integrating semantics and spatial priors

Abstract

This invention relates to a cross-view image matching method that integrates semantic and spatial priors, comprising the following steps: obtaining semantic prior features through a cross-view image semantic information mining method based on knowledge distillation; obtaining spatial prior features through a cross-view image spatial alignment method guided by satellite imagery; and obtaining global image features by fusing semantic and spatial prior features through an attention mechanism. This invention is the first to introduce image semantics and its spatial distribution information into the cross-view image matching task, designing a Siamese network combining a large semantic segmentation model and a knowledge distillation learning paradigm to achieve efficient extraction of consistent semantic features between images from different view domains. Furthermore, this invention introduces a semantic spatial alignment mechanism to effectively enhance the consistency and robustness of common features across cross-view images. This invention has advantages such as high matching accuracy, strong generalization ability, high inference efficiency, and lightweight model, making it suitable for applications such as cross-view image geolocation in complex environments.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, specifically a cross-view image matching method that integrates semantics and spatial priors. Background Technology

[0002] With the continuous advancement of artificial intelligence technology, geolocation has become a key supporting technology in fields such as intelligent transportation, autonomous driving, and robotic navigation. However, in urban areas with dense high-rise buildings and complex electromagnetic environments, traditional satellite navigation systems (such as GPS or BeiDou) often suffer from reduced positioning accuracy due to signal blockage and multipath interference, severely affecting the system's practicality and stability. To cope with complex environments such as signal limitations or rejection, geolocation methods based on cross-view image matching have gradually emerged. This method effectively supplements traditional navigation systems by performing feature matching between the query image and a geotagged reference image (such as satellite or map images). Especially in urban environments, utilizing stable visual features such as building outlines and road structures, this method still possesses strong positioning capabilities even without satellite signals. However, there are significant differences in visual appearance between ground images and satellite images, posing a severe challenge to traditional image feature extraction methods. Due to these differences, traditional feature extraction methods often struggle to extract discriminative cross-view common features, thus affecting the accuracy and robustness of positioning. Therefore, overcoming this problem is key to improving the performance of cross-view image matching. Summary of the Invention

[0003] The technical problem this invention aims to solve is that, given the significant visual differences between terrestrial and satellite images, traditional image feature extraction methods struggle to extract discriminative cross-view domain common features. This invention proposes a cross-view domain image matching method that integrates semantic and spatial priors. This method enables semantic consistency modeling and spatial structure alignment of cross-view domain images, thereby constructing more robust and discriminative common feature descriptors.

[0004] The technical solution adopted by this invention to solve the above-mentioned technical problems is as follows: a cross-view image matching method that integrates semantics and spatial priors, comprising the following steps:

[0005] S1, semantic prior features are obtained by a cross-view image semantic information mining method based on knowledge distillation;

[0006] S1-1, using paired distillation loss to obtain locally consistent semantic prior features;

[0007] S1-2, using global distillation loss to obtain global structural semantic prior features;

[0008] S2, spatial prior features are obtained based on a cross-view image spatial alignment method guided by satellite imagery;

[0009] S2-1, Offset estimation based on building outline information in satellite imagery;

[0010] S2-2, perform region reprojection adjustment based on offset and offset direction vector to obtain feature points after image reprojection;

[0011] S2-3, use a convolutional network to extract features from the reprojected feature points to obtain the spatial prior features of the image;

[0012] S3 uses an attention mechanism to fuse semantic prior features and spatial prior features to obtain global image features.

[0013] Preferably, the specific steps of step S1-1 are as follows:

[0014] S1-1-1 uses a network framework of distillation learning to perform average pooling on the features extracted by the teacher network and the student network respectively, and compresses the features at each spatial location into a fixed-dimensional feature vector to form the processed feature map.

[0015] S1-1-2, Build nodes in the processed feature map, and use the feature similarity between nodes to convert the processed feature map into a static affinity map;

[0016] S1-1-3, Calculate the first [missing information] using a static affinity diagram. The node and the first Feature similarity of nodes The formula is as follows:

[0017] (1.1)

[0018] S1-1-4, using the above formula (1.1), calculate the first step of the teacher network and the student network respectively. The node and the first The characteristic phase of each node and And calculate the paired distillation loss according to formula (1.2).

[0019] (1.2)

[0020] in, Represents the set of all spatial location nodes;

[0021] S1-1-5, Train the network based on the pairwise distillation loss obtained in the previous step to obtain locally consistent semantic prior features.

[0022] Preferably, the specific steps of step S1-2 are as follows:

[0023] S1-2-1 uses an RGB image as input, uses the segmentation results generated by the student network as "pseudo-samples", and uses the segmentation results generated by the teacher network as "real samples".

[0024] S1-2-2 introduces Wasserstein distance as the loss function to measure the difference in distribution between "pseudo-samples" and "real samples" in the embedding space, judge the consistency of their semantic structure, and guide the generation of the optimal segmentation map. The specific calculation method is shown in Equation (1.3).

[0025] (1.3)

[0026] in, For the input image, and The segmentation graphs generated for the teacher network and the student network are respectively. This is the default scoring function for the embedded spatial discriminator;

[0027] S1-2-3, the optimal segmentation map and the original RGB image are concatenated in the channel dimension and used as the input of the embedding network. The network consists of a five-layer convolutional structure and two self-attention modules, which are used to capture global dependencies and structural features.

[0028] In S1-2-4, the network maps high-dimensional embedded features to a single score value through a global pooling layer, guiding the student network to gradually approximate the global semantic distribution of the teacher model, while obtaining global structural semantic prior features.

[0029] Preferably, the specific steps of step S2-1 are as follows:

[0030] S2-1-1 extracts spatial structure features from ground images using a neural network and captures building height information. This is used to distinguish between the low-rise and high-rise areas of a building;

[0031] S2-1-2, the reference satellite image is divided into a regular grid, the proportion of building pixels within each grid is calculated, and the coefficients are adjusted accordingly. Quantize the structural density of an image;

[0032] S2-1-3, to adapt to the characteristic distribution of different altitude regions, introduces a nonlinear migration strategy combining a gating mechanism and a logarithmic function. Less than the preset threshold At this point, the area is considered the ground level or the lower level of a building, with an offset set to zero to maintain the original geometry. When the offset exceeds a threshold, a logarithmic function is used to calibrate the offset growth. The offset calculation formula is as follows.

[0033] (2.1)

[0034] in, For the set depth threshold, is the scaling factor for the logarithmic function.

[0035] Preferably, the specific steps of step S2-2 are as follows:

[0036] S2-2-1, Let the coordinates of the image center be... The original position of the pixel is Then its offset direction vector Defined as,

[0037] (2.2)

[0038] S2-2-2, based on the calculated offset With offset direction vector The reprojection coordinates of the feature points are,

[0039] (2.3)

[0040] This allows us to obtain the feature points after image reprojection.

[0041] Preferably, step S3 specifically includes the following steps:

[0042] S3-1, the self-attention layer generates attention weights between positions by calculating the self-similarity between different positions of similar input features;

[0043] S3-2, the cross-attention layer is used to calculate the correlation between different types of features, explicitly model the interaction between spatial prior features and semantic prior features, so that the two types of features can achieve complementarity and alignment;

[0044] S3-3, Network structure for obtaining global image features based on attention mechanism fusion, global image features The specific calculation formula is as follows:

[0045] (3.1)

[0046] in, and These correspond to spatial prior features and semantic prior features, respectively; It is responsible for mapping spatial prior features and semantic prior features to the same subspace; and These represent self-attention and cross-attention operations, respectively. Indicates a splicing operation; This represents a linear projection operation used to convert a feature matrix into a global descriptor vector.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. Inspired by human map perception and localization mechanisms, this invention introduces semantic category information and its spatial distribution into cross-view image matching tasks for the first time, enhancing the model's ability to extract discriminative common features from cross-view images;

[0049] 2. This invention adopts a teacher-student architecture, using a large semantic segmentation model to guide a lightweight network to learn high-order semantic features, maintaining high matching accuracy while compressing model parameters, and possessing good deployment efficiency and inference ability;

[0050] 3. This invention introduces paired structural constraints and a global adversarial mechanism during the distillation process, enabling the student model to balance local spatial consistency and global semantic distribution, thereby improving the integrity and discriminability of semantic information;

[0051] 4. This invention utilizes structural density and building outline information in satellite images to construct a region-adaptive nonlinear offset mechanism, achieving accurate feature projection from the ground view to the overhead view, reducing outline distortion and structural misalignment.

[0052] 5. This invention combines regional offset and spatial direction information to perform feature coordinate adjustment with consistent direction, thereby improving the structural continuity and semantic clarity of projected features in the top view and enhancing the geometric stability and matching robustness of high-rise building areas.

[0053] 6. This invention is applicable to various application scenarios such as image geolocation, intelligent navigation, and scene recognition. Attached Figure Description

[0054] Figure 1 This is a diagram of the semantic information mining network framework based on knowledge distillation of the present invention;

[0055] Figure 2 This is a flowchart of the cross-view image spatial alignment method guided by satellite imagery of the present invention;

[0056] Figure 3 This is a diagram illustrating the offset estimation method for building outlines based on satellite images, as described in this invention.

[0057] Figure 4 This invention is based on the attention mechanism to fuse and obtain global features of an image. Detailed Implementation

[0058] The following will combine Figure 1-4 The present invention will be described in detail below. The illustrative embodiments and descriptions herein are used to explain the invention, but are not intended to limit the invention.

[0059] A cross-view image matching method that integrates semantic and spatial priors includes the following steps:

[0060] S1, semantic prior features are obtained by a cross-view image semantic information mining method based on knowledge distillation;

[0061] S1-1, using paired distillation loss to obtain locally consistent semantic prior features;

[0062] S1-2, using global distillation loss to obtain global structural semantic prior features;

[0063] S2, spatial prior features are obtained based on a cross-view image spatial alignment method guided by satellite imagery;

[0064] S2-1, Offset estimation based on building outline information in satellite imagery;

[0065] S2-2, perform region reprojection adjustment based on offset and offset direction vector to obtain feature points after image reprojection;

[0066] S2-3, use a convolutional network to extract features from the reprojected feature points to obtain the spatial prior features of the image;

[0067] S3 uses an attention mechanism to fuse semantic prior features and spatial prior features to obtain global image features.

[0068] In this embodiment, the specific steps of step S1-1 are as follows:

[0069] S1-1-1, as shown Figure 1 As shown, a network framework for learning by distillation is used to perform average pooling on the features extracted by the teacher network and the student network respectively, and compress the features at each spatial location into a feature vector of fixed dimension to form the processed feature map.

[0070] S1-1-2, Construct nodes in the processed feature map, such as in a map of size... In the feature map, a total of Each node is used to convert the processed feature map into a static affinity map by utilizing the feature similarity between nodes.

[0071] S1-1-3, Calculate the first [missing information] using a static affinity diagram. The node and the first Feature similarity of nodes The formula is as follows:

[0072] (1.1)

[0073] S1-1-4, using the above formula (1.1), calculate the first step of the teacher network and the student network respectively. The node and the first The characteristic phase of each node and And calculate the paired distillation loss according to formula (1.2).

[0074] (1.2)

[0075] in, Represents the set of all spatial location nodes;

[0076] This loss function not only supervises the consistency of single-point features, but also guides the student network to reconstruct higher-order similarity relationships in the space through pairwise structural constraints, thereby significantly improving its ability to locally perceive semantic structures in cross-view images.

[0077] S1-1-5, Train the network based on the pairwise distillation loss obtained in the previous step to obtain locally consistent semantic prior features.

[0078] The specific steps of step S1-2 are as follows:

[0079] S1-2-1 uses an RGB image as input, uses the segmentation results generated by the student network as "pseudo-samples", and uses the segmentation results generated by the teacher network as "real samples".

[0080] S1-2-2 introduces Wasserstein distance as the loss function to measure the difference in distribution between "pseudo-samples" and "real samples" in the embedding space, judge the consistency of their semantic structure, and guide the generation of the optimal segmentation map. The specific calculation method is shown in Equation (1.1).

[0081] (1.3)

[0082] in, For the input image, and The segmentation graphs generated for the teacher network and the student network are respectively. This is the default scoring function for the embedded spatial discriminator;

[0083] S1-2-3, the optimal segmentation map and the original RGB image are concatenated in the channel dimension and used as the input of the embedding network. The network consists of a five-layer convolutional structure and two self-attention modules, which are used to capture global dependencies and structural features.

[0084] In S1-2-4, the network maps high-dimensional embedded features to a single score value through a global pooling layer, guiding the student network to gradually approximate the global semantic distribution of the teacher model, while obtaining global structural semantic prior features.

[0085] The specific steps of step S2-1 are as follows:

[0086] First, the offset of each region of interest is modeled;

[0087] S2-1-1 extracts spatial structure features from ground images using a neural network and captures building height information. This is used to distinguish between the low-rise and high-rise areas of a building;

[0088] S2-1-2, the reference satellite image is divided into a regular grid, the proportion of building pixels within each grid is calculated, and the coefficients are adjusted accordingly. Quantize the structural density of an image;

[0089] S2-1-3, to adapt to the characteristic distribution of different altitude regions, introduces a nonlinear migration strategy combining a gating mechanism and a logarithmic function. Less than the preset threshold At this time, the area is considered as the ground or lower level of a building, and the offset is set to zero to maintain the original geometry. When it exceeds a threshold, a logarithmic function is used to calibrate the offset growth to prevent abrupt changes in upper-level features. The overall process is as follows: Figure 2 As shown, the formula for calculating the offset is as follows:

[0090] (2.1)

[0091] in, A set depth threshold is used to remove invalid offsets from the ground floor and the lower levels of buildings. This is a scaling factor for the logarithmic function, used to adjust the rate of offset growth.

[0092] After calculating the offset, we further determine the offset direction of the features based on the spatial relationship between the image center and each pixel, and adjust the pixel coordinates of the region accordingly to achieve spatial structure alignment under cross-viewpoint mapping, such as... Figure 3 As shown.

[0093] The specific steps of step S2-2 are as follows:

[0094] S2-2-1, Let the coordinates of the image center be... The original position of the pixel is Then its offset direction vector Defined as,

[0095] (2.2)

[0096] S2-2-2, based on the calculated offset With offset direction vector The reprojection coordinates of the feature points are,

[0097] (2.3)

[0098] This allows us to obtain the feature points after image reprojection.

[0099] This reprojection mechanism ensures that feature points in different height areas are projected and unfolded along a reasonable spatial direction, so that the building facade information presents a clear and continuous spatial layout in the top view, and finally obtains more consistent spatial prior features through a convolutional neural network.

[0100] Finally, the specific implementation process of step S3 is as follows:

[0101] S3-1, the self-attention layer generates attention weights between positions by calculating the self-similarity between different positions of similar input features, so as to strengthen or suppress the importance of similar features in different regions, thereby optimizing the expressive power of global features;

[0102] S3-2, the cross-attention layer is used to calculate the correlation between different types of features, explicitly model the interaction between spatial prior features and semantic prior features, so that the two types of features can achieve complementarity and alignment;

[0103] S3-3, the network structure for obtaining global image features based on attention mechanism fusion is as follows: Figure 4 As shown, global features of the image The specific calculation formula is as follows:

[0104] (3.1)

[0105] in, and These correspond to spatial prior features and semantic prior features, respectively; It is responsible for mapping spatial prior features and semantic prior features to the same subspace so that they can be fused. and These represent self-attention and cross-attention operations, respectively. Indicates a splicing operation; This represents a linear projection operation used to convert a feature matrix into a global descriptor vector.

[0106] This invention has the advantages of high matching accuracy, strong generalization ability, high inference efficiency, and lightweight model, and is suitable for application scenarios such as cross-view image geolocation in complex environments.

[0107] The technical solutions provided by the embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the embodiments of the present invention. The descriptions of the embodiments above are only for helping to understand the principles of the embodiments of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the embodiments of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for cross-view image matching by fusing semantic and spatial priors, characterized in that, The method comprises the following steps: S1, obtaining semantic prior features by a cross-view image semantic information mining method based on knowledge distillation; S1-1, obtaining local consistent semantic prior features by using a pair-wise distillation loss; S1-2, obtaining global structure semantic prior features by using an overall distillation loss; S2, obtaining spatial prior features by a cross-view image spatial alignment method based on satellite image guidance; S2-1, estimating an offset based on building contour information in a satellite image; S2-1-1, extract the spatial structure features of the ground image through the neural network, capture the building height information to distinguish the low-rise and high-rise areas of the building; S2-1-2, the reference satellite image is divided into regular grids, the proportion of building pixels in each grid is counted, and the proportion is adjusted by a coefficient quantifying the structural density of the image; S2-1-3, to adapt to the characteristics of different height areas, a gating mechanism is introduced and combined with a nonlinear offset strategy of logarithmic function, when less than the preset threshold , the area is considered as ground or low-rise building, the offset is set to zero, and the original geometry is kept unchanged; when greater than the threshold, the offset growth is calibrated by the logarithmic function, and the offset calculation formula is as follows, （2.1） wherein is a set depth threshold, is a scaling factor for the logarithmic function; S2-2, performing regional re-projection adjustment based on the offset and an offset direction vector to obtain feature points after image re-projection; S2-2-1, assuming the image center coordinate is , the pixel point original position is , then its offset direction vector is defined as, （2.2） S2-2-2, according to the calculated offset with the offset direction vector , the re-projection coordinates of the feature points are （2.3） Thus, the feature points after image re-projection are obtained; S2-3, performing feature extraction on the feature points after re-projection by using a convolutional network to obtain spatial prior features of the image; S3, obtaining image global features by fusing the semantic prior features and the spatial prior features through an attention mechanism.

2. The method of claim 1, wherein the method further comprises: The specific steps of the step S1-1 are as follows: S1-1-1, adopting a distillation learning network framework, performing average pooling on the features extracted by a teacher network and a student network respectively, and compressing the features at each spatial position into a fixed-dimension feature vector to form a processed feature map; S1-1-2, constructing nodes in the processed feature map, and converting the processed feature map into a static affinity graph by using the feature similarity between the nodes; S1-1-3, computing the feature similarity of the th node to the th node by static affinity map , as follows, （1.1） S1-1-4, respectively calculate the features of the first node and the first node under the teacher network and the student network by using the above formula (1.1) , and calculate the pair-wise distillation loss according to formula (1.2), （1.2） wherein denotes the set of all spatial position nodes; S1-1-5, training the network according to the pair-wise distillation loss obtained in the previous step to obtain local consistent semantic prior features.

3. The method of claim 1, wherein the method further comprises: determining a semantic label for each of the plurality of image regions; and determining a semantic label for each of the plurality of image regions based on the semantic label of the image region and the semantic label of the image region. The specific steps of the step S1-2 are as follows: S1-2-1, using an RGB image as input, using a predicted segmentation result generated by a student network as a "pseudo sample", and using a segmentation result generated by a teacher network as a "real sample"; S1-2-2, introducing a Wasserstein distance as a loss function to measure the distribution difference between the "pseudo sample" and the "real sample" in an embedding space, judge the consistency of the semantic structure, and guide the generation of an optimal segmentation map, and the specific calculation method is shown in formula (1.3), （1.3） wherein, is an input image, and are segmentation maps generated by the teacher network and the student network, respectively, is a default scoring function for the embedding space discriminator; S1-2-3, splicing the optimal segmentation map and the original RGB image in the channel dimension to serve as an embedding network input, the network is composed of five layers of convolutional structure and two self-attention modules, and is used for capturing global dependence and structure features; S1-2-4, the network maps high-dimensional embedding features into a single score value through a global pooling layer, guides the student network to gradually approximate the global semantic distribution of the teacher model, and simultaneously obtains global structure semantic prior features.

4. The method of claim 1, wherein the fusion of semantic and spatial prior cross-view image matching method is characterized by, The step S3 specifically comprises the following steps: S3-1, a self-attention layer generates attention weights between positions by calculating the self-similarity between different positions of the same type of input features; S3-2, a cross-attention layer is used to calculate the correlation between different types of features, and the interaction relationship between the spatial prior features and the semantic prior features is explicitly modeled, so that the two types of features are complementary and aligned; S3-3, based on attention mechanism fusion to obtain image global feature network structure, image global feature The specific calculation formula is as follows: （3.1） wherein, and correspond to spatial prior features and semantic prior features, respectively; is responsible for mapping spatial prior features and semantic prior features to the same subspace; and denote self-attention and cross-attention operations, respectively; denotes concatenation operation; denotes linear projection operation for converting feature matrix to global descriptor vector.

Images