Cross-vision-field image matching method fusing semantics and spatial prior

Abstract

The invention relates to a cross-vision-field image matching method fusing semantics and spatial prior. The method comprises the following steps: obtaining semantic prior features by a cross-vision-field image semantic information mining method based on knowledge distillation; spatial prior features are obtained based on a satellite image guided cross-vision-field image spatial alignment method; and fusing the semantic priori features and the spatial priori features through an attention mechanism to obtain global features of the image. Image semantics and spatial distribution information of the image semantics are introduced into a cross-vision-field image matching task for the first time, a twin network combining a semantic segmentation large model and a knowledge distillation learning normal form is designed, and efficient extraction of consistent semantic features among images of different vision fields is achieved. And secondly, a semantic space alignment mechanism is further introduced, so that the consistency and robustness of common characteristics of the cross-vision-field image are effectively enhanced. The method has the advantages of high matching precision, strong generalization ability, high reasoning efficiency, light model weight and the like, and is suitable for application scenes such as cross-vision field image geographic positioning in a complex environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and image processing, and particularly relates to a cross-view image matching method fusing semantic and spatial priori. BACKGROUND

[0002] With the continuous progress of artificial intelligence technology, geographic positioning has become a key supporting technology in the fields of intelligent transportation, unmanned driving and robot autonomous navigation. However, in urban areas with high-rise buildings and complex electromagnetic environments, the positioning accuracy of traditional satellite navigation systems (such as GPS or Beidou) often decreases due to signal blockage and multipath interference, seriously affecting the practicability and stability of the system. To cope with complex environments such as signal limitation or denial, geographic positioning methods based on cross-view image matching have gradually emerged. This method matches the query image with the reference image (such as satellite images or map images) with geographic markers as an effective supplement to traditional navigation systems. Especially in urban environments, using stable visual features such as building outlines and road structures, this method still has strong positioning ability under the condition of no satellite signal. However, there are significant differences in visual appearance between ground images and satellite images, which poses a serious challenge to traditional image feature extraction methods. Due to these differences, traditional feature extraction methods often fail to extract discriminative cross-view common features, which in turn affects the accuracy and robustness of positioning. Therefore, how to overcome this problem is the key to improving the performance of cross-view image matching. SUMMARY

[0003] The technical problem to be solved by the present application is that the significant differences in visual appearance between ground images and satellite images make it difficult for traditional image feature extraction methods to extract discriminative cross-view common features. To solve this problem, the present application proposes a cross-view image matching method fusing semantic and spatial priori. This method can realize semantic consistency modeling and spatial structure alignment of cross-view images, thereby constructing common feature descriptors with higher robustness and discriminability.

[0004] The technical solution adopted by the present application to solve the above technical problem is a cross-view image matching method fusing semantic and spatial priori, comprising the following steps:

[0005] S1, obtaining semantic priori features by a cross-view image semantic information mining method based on knowledge distillation;

[0006] S1-1, obtaining local consistent semantic priori features by using pair-wise distillation loss;

[0007] S1-2, obtaining global structure semantic priori features by using overall distillation loss;

[0008] S2, obtaining spatial priori features by a cross-view image spatial alignment method guided by satellite images;

[0009] S2-1, estimating the offset based on the building contour information in the satellite image;

[0010] S2-2, performing regional re-projection adjustment based on the offset and the offset direction vector to obtain the feature points after re-projection of the image;

[0011] S2-3, performing feature extraction on the feature points after re-projection using a convolutional network to obtain the spatial prior features of the image;

[0012] S3, obtaining the global features of the image by fusing the semantic prior features and the spatial prior features through an attention mechanism.

[0013] As a preferred, the specific steps of the step S1-1 are:

[0014] S1-1-1, using a network framework of distillation learning, performing average pooling on the features extracted by the teacher network and the student network respectively, and compressing the features of each spatial position into a fixed-dimension feature vector to form a processed feature map;

[0015] S1-1-2, constructing nodes in the processed feature map, and converting the processed feature map into a static affinity graph by using the feature similarity between the nodes;

[0016] S1-1-3, calculating the feature similarity between the i-th node and the j-th node in the static affinity graph, and the formula is as follows,

[0017] (1.1)

[0018] S1-1-4, calculating the feature similarity between the i-th node and the j-th node under the teacher network and the student network respectively by using the above formula (1.1), and calculating the pairwise distillation loss according to formula (1.2),

[0019] (1.2)

[0020] wherein, represents the set of all spatial position nodes;

[0021] S1-1-5, training the network according to the pairwise distillation loss obtained in the above step to obtain the local consistent semantic prior features.

[0022] As a preferred, the specific steps of the step S1-2 are:

[0023] ​​​​​​​S1-2-1, using the RGB image as input, the student network generates a predicted segmentation result as a "pseudo sample", and the segmentation result generated by the teacher network is used as a "real sample";

[0024] S1-2-2, introduce the Wasserstein distance as the loss function, measure the distribution difference of "pseudo sample" and "real sample" in the embedding space, judge the consistency of its semantic structure, guide to generate the optimal segmentation map, the specific calculation method is shown in formula (1.3),

[0025] (1.3)

[0026] Wherein, is the input image, and are the segmentation maps generated by the teacher network and the student network respectively, is the default scoring function of the embedding space discriminator;

[0027] S1-2-3, the optimal segmentation map and the original RGB image are spliced in the channel dimension and used as the input of the embedding network, which is composed of five layers of convolution structure and two self-attention modules, which is used to capture global dependence and structural features;

[0028] S1-2-4, the network maps high-dimensional embedding features to a single score value through a global pooling layer, guiding the student network to gradually approach the global semantic distribution of the teacher model, while obtaining global structural semantic prior features.

[0029] As preferred, the specific steps of step S2-1 are:

[0030] S2-1-1, extract the spatial structure features of the ground image through the neural network, and capture the building height information , to distinguish the low and high areas of the building;

[0031] S2-1-2, divide the reference satellite image into regular grids, count the proportion of building pixels in each grid, and quantize the structural density of the image by adjusting the coefficient ;

[0032] S2-1-3, in order to adapt to the feature distribution of different height areas, a nonlinear offset strategy combined with a gating mechanism and a logarithmic function is introduced, when is less than a preset threshold , the area is regarded as the ground or the low layer of the building, and the offset amount is set to zero, keeping the original geometry unchanged; when it is greater than the threshold, the logarithmic function is used to calibrate the offset growth, and the offset amount is calculated as follows,

[0033] (2.1)

[0034] wherein, is a set depth threshold, is a scaling factor of the logarithmic function.

[0035] As preferred, the specific steps of the step S2-2 are:

[0036] S2-2-1, assuming the image center coordinates as , the pixel point original position as , then its offset direction vector is defined as,

[0037] (2.2)

[0038] S2-2-2, according to the calculated offset and the offset direction vector , the re-projection coordinates of the feature points are,

[0039] (2.3)

[0040] Thus, the image re-projection feature points are obtained.

[0041] As preferred, the step S3 specifically comprises the following steps:

[0042] S3-1, the self-attention layer generates the attention weight between positions by calculating the self-similarity between different positions of the same input feature;

[0043] S3-2, the cross-attention layer is used to calculate the correlation between different types of features, and explicitly models the interaction between the spatial prior feature and the semantic prior feature, so that the two types of features are complementary and aligned;

[0044] S3-3, based on the attention mechanism, the image global feature network structure is obtained, and the image global feature The specific calculation formula is as follows:

[0045] (3.1)

[0046] wherein, and correspond to the spatial prior feature and the semantic prior feature, respectively; is responsible for mapping the spatial prior feature and the semantic prior feature to the same subspace; and represent the self-attention and cross-attention operations, respectively; represents the splicing operation; represents the linear projection operation, which is used to convert the feature matrix into a global descriptor vector.

[0047] Compared with the prior art, the present application has the following beneficial effects:

[0048] 1、 The present application is inspired by human map perception and positioning mechanism, for the first time introduces semantic category information and its spatial distribution into cross-view image matching task, enhances the extraction ability of the model to the common features of cross-view images with discriminative characteristics;

[0049] 2、 The present application adopts teacher-student architecture, uses semantic segmentation large model to guide lightweight network to learn high-order semantic features, while maintaining high matching accuracy, has good deployment efficiency and reasoning ability;

[0050] 3、 The present application introduces pair structure constraint and overall adversarial mechanism in the distillation process, so that the student model considers local spatial consistency and global semantic distribution, improves the integrity and discriminability of semantic information;

[0051] 4、 The present application uses the structure density and building contour information in satellite images to construct a region-adaptive nonlinear offset mechanism, realizes accurate feature projection from ground view to overhead view, reduces contour distortion and structure misplacement;

[0052] 5、 The present application combines regional offset and spatial direction information to perform direction-consistent feature coordinate adjustment, improves the structure continuity and semantic clarity of projected features in overhead view, enhances the geometric stability and matching robustness of high-rise building area;

[0053] 6、 The present application is suitable for image geographic positioning, intelligent navigation, scene recognition and other application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is the semantic information mining network framework of the present application based on knowledge distillation;

[0055] Figure 2 is the satellite image guided cross-view image spatial alignment method flow chart of the present application;

[0056] Figure 3 is the offset amount estimation method based on satellite image building contour of the present application;

[0057] Figure 4 is the image global feature fusion and acquisition based on attention mechanism of the present application. DETAILED DESCRIPTION

[0058] The specific embodiments of the present application will be described below. Figures 1-4 The present application will be described in detail below, the schematic embodiments of the present application and the description are used to explain the present application, but not as a limitation of the present application.

[0059] A cross-view image matching method fusing semantic and spatial priori, comprising the following steps:

[0060] S1, obtaining semantic priori features by a cross-view image semantic information mining method based on knowledge distillation;

[0061] S1-1, obtaining local consistent semantic priori features by using a pair-wise distillation loss;

[0062] S1-2, obtaining global structure semantic priori features by using an overall distillation loss;

[0063] S2, obtaining spatial priori features by a cross-view image spatial alignment method based on satellite image guidance;

[0064] S2-1, estimating offset based on building contour information in a satellite image;

[0065] S2-2, performing regional re-projection adjustment based on the offset and offset direction vector to obtain feature points after image re-projection;

[0066] S2-3, extracting features on the feature points after re-projection by using a convolutional network to obtain spatial priori features of the image;

[0067] S3, obtaining image global features by fusing the semantic priori features and the spatial priori features through an attention mechanism.

[0068] In the embodiment, the specific steps of step S1-1 are as follows:

[0069] S1-1-1, as shown in the figure, a network framework of distillation learning is adopted, the features extracted by the teacher network and the student network are respectively subjected to average pooling, and the features of each spatial position are compressed into fixed-dimension feature vectors to form a processed feature map; Figure 1 S1-1-2, nodes are constructed in the processed feature map, for example, in a feature map with a size of

[0070] nodes are constructed, and the processed feature map is converted into a static affinity graph by using the feature similarity between the nodes; S1-1-3, the feature similarity between the i th node and the j th node is calculated by the static affinity graph

[0071] , the formula is as follows,

[0072] (1.1)

[0073] S1-1-4, the above formula (1.1) is used to respectively calculate the i th node under the teacher network and the student network ​​​​a node and a feature of a first a node and a feature of a first and and the pairwise distillation loss is calculated according to formula (1.2),

[0074] (1.2)

[0075] wherein, denotes a set of all spatial position nodes;

[0076] The loss function not only supervises the consistency of single-point features, but also guides the student network to reconstruct high-order similarity relationships in the space through a pairwise structure constraint, thereby significantly improving the local perception ability of the student network to semantic structures in cross-view images.

[0077] S1-1-5, training the network according to the pairwise distillation loss obtained in the previous step to obtain local consistent semantic prior features.

[0078] The specific steps of step S1-2 are as follows:

[0079] S1-2-1, using an RGB image as input, using the predicted segmentation result generated by the student network as a "pseudo sample", and using the segmentation result generated by the teacher network as a "real sample";

[0080] S1-2-2, introducing the Wasserstein distance as a loss function to measure the distribution difference between the "pseudo sample" and the "real sample" in the embedding space, judge the consistency of the semantic structure, and guide the generation of the optimal segmentation map, the specific calculation method is shown in formula (1.1),

[0081] (1.3)

[0082] wherein, is an input image, and are the segmentation maps generated by the teacher network and the student network respectively, is a default scoring function of the embedding space discriminator;

[0083] S1-2-3, the optimal segmentation map is spliced with the original RGB image in the channel dimension and used as the input of the embedding network, which is composed of five layers of convolution structure and two self-attention modules, for capturing global dependencies and structural features;

[0084] S1-2-4, the network maps the high-dimensional embedding features to a single score value through a global pooling layer, guiding the student network to gradually approximate the global semantic distribution of the teacher model, while obtaining global structural semantic prior features.

[0085] The specific steps of step S2-1 are as follows:

[0086] First, the offset of each region of interest is modeled;

[0087] S2-1-1, the spatial structure features of the ground image are extracted by the neural network, and the building height information is captured , to distinguish the low and high areas of the building;

[0088] S2-1-2, the reference satellite image is divided into a regular grid, the proportion of building pixels in each grid is counted, and the coefficient is adjusted to quantify the structural density of the image;

[0089] S2-1-3, to adapt to the feature distribution of different height areas, a nonlinear offset strategy combined with a gating mechanism and a logarithmic function is introduced, when is less than the preset threshold , the area is considered as ground or low building, and the offset is set to zero, keeping the original geometry unchanged; when it is greater than the threshold, the logarithmic function is used to calibrate the offset growth, preventing the high layer feature from being mutated, and the overall process is as shown in Figure 2 , the offset calculation formula is as follows,

[0090] (2.1)

[0091] wherein, is the set depth threshold, used to eliminate invalid offset of ground and building bottom layer area, is the scaling factor of the logarithmic function, used to adjust the offset growth rate.

[0092] After the offset calculation is completed, we further determine the offset direction of the feature according to the spatial relationship between the image center and each pixel point, and adjust the region pixel coordinate position accordingly, to realize the spatial structure alignment under the cross-view mapping, as shown in Figure 3 .

[0093] The specific steps of step S2-2 are as follows:

[0094] S2-2-1, let the image center coordinates be , the original position of the pixel point be , and the offset direction vector be defined as,

[0095] (2.2)

[0096] S2-2-2, according to the calculated offset and offset direction vector , the re-projection coordinates of the feature point are,

[0097] (2.3)

[0098] Thus, the feature points after image re-projection are obtained.

[0099] The re-projection mechanism ensures that the feature points of different height regions are projected and expanded along reasonable spatial directions, so that the building facade information presents a clear structure and continuous contour in the spatial layout in the top view, and finally more consistent spatial prior features are obtained through the convolutional neural network.

[0100] Finally, the specific implementation process of step S3 is as follows:

[0101] S3-1, the self-attention layer generates attention weights between positions by calculating the self-similarity between different positions of the same input feature, so as to strengthen or suppress the importance of the same feature in different regions, thereby optimizing the expression ability of the global feature;

[0102] S3-2, the cross-attention layer is used to calculate the correlation between different types of features, and the interaction between the spatial prior feature and the semantic prior feature is explicitly modeled, so that the two types of features are complementary and aligned;

[0103] S3-3, based on the attention mechanism fusion, the image global feature network structure is as shown in Figure 4 The image global feature The specific calculation formula is as follows:

[0104] (3.1)

[0105] Wherein, and correspond to the spatial prior feature and the semantic prior feature respectively; is responsible for mapping the spatial prior feature and the semantic prior feature to the same subspace for fusion; and represent the self-attention and cross-attention operations respectively; represents the splicing operation; represents the linear projection operation, which is used to convert the feature matrix into a global descriptor vector.

[0106] The application has the advantages of high matching precision, strong generalization ability, high reasoning efficiency, light model and the like, and is suitable for application scenes such as cross-view image geographic positioning in complex environments.

[0107] The technical solutions provided by the embodiments of the present application are described in detail above, and specific examples are applied in this paper to describe the principles and implementation manners of the embodiments of the present application. The above description of the embodiments is only used to help understand the principles of the embodiments of the present application; meanwhile, for the general skilled in the art, the embodiments of the present application will have changes in the specific implementation manners and application ranges, and the above description should not be understood as a limitation on the present application.

Claims

1. A method for cross-view image matching by fusing semantic and spatial priors, characterized in that, The method comprises the following steps: S1, obtaining semantic prior features by a cross-view image semantic information mining method based on knowledge distillation; S1-1, obtaining local consistent semantic prior features by using a pair-wise distillation loss; S1-2, obtaining global structure semantic prior features by using an overall distillation loss; S2, obtaining spatial prior features by a cross-view image spatial alignment method based on satellite image guidance; S2-1, estimating an offset based on building contour information in a satellite image; S2-2, performing regional re-projection adjustment based on the offset and an offset direction vector to obtain feature points after image re-projection; S2-3, performing feature extraction on the feature points after re-projection by using a convolutional network to obtain spatial prior features of the image; S3, obtaining image global features by fusing the semantic prior features and the spatial prior features through an attention mechanism.

2. The method of claim 1, wherein the method further comprises: The specific steps of the step S1-1 are as follows: S1-1-1, adopting a distillation learning network framework, performing average pooling on features extracted by a teacher network and a student network respectively, and compressing features of each spatial position into a fixed-dimension feature vector to form a processed feature map; S1-1-2, constructing nodes in the processed feature map, and converting the processed feature map into a static affinity graph by using feature similarity between the nodes; S1-1-3, computing the feature similarity of the th node to the th node by static affinity map , as follows, （1.1） S1-1-4, respectively calculate the features of the first node and the first node under the teacher network and the student network by using the above formula (1.1) , and calculate the pair-wise distillation loss according to formula (1.2), （1.2） wherein denotes the set of all spatial position nodes; S1-1-5, training the network according to the pair-wise distillation loss obtained in the above step to obtain local consistent semantic prior features.

3. The method of claim 1, wherein the method further comprises: The specific steps of the step S1-2 are as follows: S1-2-1, using an RGB image as input, using a predicted segmentation result generated by a student network as a "pseudo sample", and using a segmentation result generated by a teacher network as a "real sample"; S1-2-2, introducing a Wasserstein distance as a loss function to measure distribution difference of the "pseudo sample" and the "real sample" in an embedding space, judge consistency of semantic structure thereof, and guide generation of an optimal segmentation map, and a specific calculation method is shown in formula (1.3), （1.3） wherein, is an input image, and are segmentation maps generated by the teacher network and the student network, respectively, is a default scoring function for the embedding space discriminator; S1-2-3, splicing the optimal segmentation map and an original RGB image in a channel dimension to serve as input of an embedding network, the network comprising five layers of convolutional structures and two self-attention modules, and being used for capturing global dependence and structure features; S1-2-4, mapping high-dimensional embedding features into a single score value by a global pooling layer, guiding a student network to gradually approximate a global semantic distribution of a teacher model, and simultaneously obtaining global structure semantic prior features.

4. The method of claim 1, wherein the method further comprises: The specific steps of the step S2-1 are as follows: S2-1-1, extract the spatial structure features of the ground image through the neural network, capture the building height information to distinguish the low-rise and high-rise areas of the building; S2-1-2, the reference satellite image is divided into regular grids, the proportion of building pixels in each grid is counted, and the proportion is adjusted by a coefficient quantifying the structural density of the image; S2-1-3, to adapt to the characteristics of different height areas, a gating mechanism is introduced and combined with a nonlinear offset strategy of logarithmic function, when less than the preset threshold , the area is considered as ground or low-rise building, the offset is set to zero, and the original geometry is kept unchanged; when greater than the threshold, the offset growth is calibrated by the logarithmic function, and the offset calculation formula is as follows, （2.1） wherein is a set depth threshold, is a scaling factor for the logarithmic function.

5. The method of claim 4, wherein the fusion of semantic and spatial prior in cross-view image matching is characterized by: The specific steps of the step S2-2 are as follows: S2-2-1, assuming that the image center coordinate is , the pixel point original position is , and the offset direction vector is defined as, （2.2） S2-2-2, according to the calculated offset with the offset direction vector , the re-projection coordinates of the feature points are （2.3） Thus, the feature points after image re-projection are obtained.

6. The method of claim 1, wherein the fusing semantic and spatial prior cross-view image matching method is characterized by, The step S3 specifically comprises the following steps: S3-1, a self-attention layer generates attention weights between positions by calculating self-similarity between different positions of the same kind of input features; S3-2, a cross-attention layer is used for calculating correlation between different kinds of features, explicitly modeling an interaction relationship between the spatial prior features and the semantic prior features, and enabling the two kinds of features to be complementary and aligned; S3-3, based on attention mechanism fusion to obtain image global feature network structure, image global feature The specific calculation formula is as follows: （3.1） wherein, and correspond to spatial prior features and semantic prior features, respectively; is responsible for mapping spatial prior features and semantic prior features to the same subspace; and denote self-attention and cross-attention operations, respectively; denotes concatenation operation; denotes linear projection operation for converting feature matrix to global descriptor vector.

Images