Panoramic image feature matching method based on position-aware thin plate spline transform

Abstract

The present invention discloses a panoramic image feature matching method based on position-aware thin plate spline transformation, comprising the following steps: constructing a feature matching model; the feature matching model comprises a key point detection module, a position-aware thin plate spline transformation module and a key point description module; inputting a panoramic image to be matched into the key point detection module to obtain key points; inputting the obtained key points and an intermediate feature map into the position-aware thin plate spline transformation module to generate a deformed polar coordinate patch; inputting the deformed polar coordinate patch into the key point description module to generate a feature descriptor of the panoramic image to be matched; and performing brute force matching on the feature descriptor to complete feature matching of the panoramic image to be matched; the present invention parametrizes spatial dependencies between pixels through a learnable position map, so that the network can adaptively adjust convolution operations for different image regions, generate high-dimensional compact feature descriptors, and effectively alleviate the feature blurring problem caused by reduced resolution in traditional methods.

Description

Technical Field

[0001] The present invention relates to the technical field of panoramic image feature matching, and in particular to a panoramic image feature matching method based on position-aware thin plate spline transformation. Background Art

[0002] With the continuous development of panorama technology, its resolution is getting higher and higher, capable of presenting wider perspectives and richer scene information than traditional images. The true image of a panorama is a sphere, and its relationship with a two-dimensional panorama is similar to the mapping of the earth to a world map. When a three-dimensional spherical image is projected onto a two-dimensional plane, the two-dimensional panorama cannot fully preserve the geometric characteristics of the original spherical image, resulting in inconsistent distortion in different areas.

[0003] The characteristics of panoramas place higher demands on feature matching algorithms. Traditional planar panorama feature point matching methods do not perform well in today's high-resolution scenes. This is mainly due to insufficient matching accuracy, fixed input image resolution requirements, and poor performance in cases of severe geometric distortion.

[0004] While deep learning has achieved impressive results in image feature matching, its application to panoramas remains a challenge. Geometric distortion in panoramas severely impacts feature detection, matching, and geometric alignment. Conventional convolutional neural networks cannot adapt to these complex geometric variations during feature extraction, making them difficult to process panoramas.

[0005] In recent years, some researchers have proposed methods based on spherical convolution, which project a flat panorama onto a virtual sphere to better address geometric distortion. However, this method has drawbacks. First, the calculation process requires frequent conversion between flat panorama pixel coordinates and spherical coordinates. This conversion process is not only complex but also consumes a lot of computing resources, making it difficult to deploy on resource-constrained devices. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the present invention provides a panoramic image feature matching method based on position-aware thin plate spline transform, which aims to solve the problems in the background technology.

[0007] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a panoramic image feature matching method based on position-aware thin plate spline transform, comprising the following steps: reading a panoramic image to be matched; constructing a feature matching model; the feature matching model comprises a key point detection module, a position-aware thin plate spline transform module and a key point description module; inputting the panoramic image to be matched into the key point detection module, performing feature extraction on the panoramic image to be matched to obtain a key point map, a feature heat map and an intermediate feature map, performing feature point screening on the key point map and the feature heat map to obtain key points; inputting the obtained key points and the intermediate feature map into the position-aware thin plate spline transform module, and performing feature extraction on the key point map and the feature heat map to obtain key points; In the module, a thin plate spline transformation tensor is first generated based on the position perception of the key points and the intermediate feature map, and then a polar coordinate grid of the key points is generated. The polar coordinate grid of the key points is thin-plate spline transformed based on the thin plate spline transformation tensor to obtain a deformed polar coordinate grid of the key points. The panorama to be matched is sampled according to the deformed polar coordinate grid of the key points to generate a deformed polar coordinate patch. The deformed polar coordinate patch is input into the key point description module to generate a feature descriptor of the panorama to be matched. Finally, a brute force matching is performed on the feature descriptors of the panorama to be matched. Two panoramas to be matched are set to be read, and finally a brute force matching is performed on the feature descriptors of the two obtained panoramas to be matched.

[0008] Among them, the key point detection module consists of a feature extraction module and a key point screening module. The feature extraction module extracts features from the panorama to be matched to obtain a key point map, a feature heat map and an intermediate feature map; the key point screening module screens feature points from the key point map and the feature heat map to obtain key points with high confidence.

[0009] Furthermore, the feature extraction module is composed of a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, a fifth convolution block, a sixth convolution block and a seventh convolution block; the processing flow of the feature extraction module is as follows: the panoramic image to be matched passes through the first convolution block and the second convolution block respectively, the output of the first convolution block and the output of the second convolution block are added, and then sequentially pass through the fourth convolution block, the fifth convolution block and the sixth convolution block to obtain an intermediate feature map F, and the intermediate feature map F passes through the seventh convolution block and the Sigmoid function in sequence to obtain a feature heat map H; the panoramic image to be matched is divided into multiple grids, and each grid is reshaped into a feature vector , then the eigenvector After passing through the third convolution block, the key point map K is obtained;

[0010] The first convolution block is composed of a first convolution layer, a first convolution downsampling layer, a first MambaBlock module, a second MambaBlock module, and a second convolution downsampling layer connected in sequence; the panorama to be matched is sequentially passed through the first convolution layer, the first convolution downsampling layer, the first MambaBlock module, the second MambaBlock module, and the second convolution downsampling layer to obtain the output of the first convolution block;

[0011] The second convolution block is composed of a first average pooling layer and a second convolution layer connected in sequence; the panorama to be matched is sequentially passed through the first average pooling layer and the second convolution layer to obtain the output of the second convolution block;

[0012] The third convolution block is composed of the third convolution layer, the fourth convolution layer, the fifth convolution layer and the sixth convolution layer connected in sequence; the feature vector After passing through the third convolutional layer, the fourth convolutional layer, the fifth convolutional layer and the sixth convolutional layer in sequence, the key point map K is obtained;

[0013] The fourth convolution block is composed of a third MambaBlock module and a fourth MambaBlock module connected in sequence; the output of the first convolution block and the output of the second convolution block are added, and the output is passed through the third MambaBlock module and the fourth MambaBlock module in sequence to obtain the output of the fourth convolution block;

[0014] The fifth convolution block is composed of a third convolution downsampling layer, a fifth MambaBlock module, and a sixth MambaBlock module connected in sequence; the output of the fourth convolution block is sequentially passed through the third convolution downsampling layer, the fifth MambaBlock module, and the sixth MambaBlock module to obtain the output of the fifth convolution block;

[0015] The sixth convolutional block is composed of a seventh convolutional layer, an eighth convolutional layer, and a ninth convolutional layer connected in sequence; the output of the fifth convolutional block is sequentially passed through the seventh convolutional layer, the eighth convolutional layer, and the ninth convolutional layer to obtain the output of the sixth convolutional block;

[0016] The seventh convolution block is composed of a tenth convolution layer, an eleventh convolution layer, and a twelfth convolution layer connected in sequence; the output of the sixth convolution block is sequentially passed through the tenth convolution layer, the eleventh convolution layer, and the twelfth convolution layer to obtain the output of the seventh convolution block.

[0017] Furthermore, the first MambaBlock module, the second MambaBlock module, the third MambaBlock module, the fourth MambaBlock module, the fifth MambaBlock module and the sixth MambaBlock module have the same structure, and are all composed of a first MambaLayer module, a second MambaLayer module, a third MambaLayer module, a fourth MambaLayer module and a thirteenth convolutional layer; the processing flow of the first MambaBlock module is: the output of the first convolutional downsampling layer is subjected to group normalization and ReLU activation function in sequence to obtain a first enhanced feature, the first enhanced feature is input into the first MambaLayer module for processing to obtain a second enhanced feature, and the second enhanced feature is processed according to The third enhanced feature is obtained by transposing, group normalizing and ReLU activation function, and the third enhanced feature is sequentially passed through the second MambaLayer module and transposed to obtain the fourth enhanced feature; the output of the first convolutional downsampling layer is transposed first, and then sequentially passed through group normalization and ReLU activation function to obtain the fifth enhanced feature, the fifth enhanced feature is passed through the third MambaLayer module and transposed to obtain the sixth enhanced feature, the sixth enhanced feature is passed through group normalization and ReLU activation function to obtain the seventh enhanced feature, the seventh enhanced feature is passed through the fourth MambaLayer module to obtain the eighth enhanced feature, the fourth enhanced feature and the eighth enhanced feature are spliced and sequentially passed through the thirteenth convolutional layer, batch normalization and ReLU activation function to obtain the final output of the first MambaBlock module.

[0018] Furthermore, the first MambaLayer module, the second MambaLayer module, the third MambaLayer module and the fourth MambaLayer module have the same structure; the processing flow of the first MambaLayer module is: first, transpose the first enhancement feature;

[0019] Perform layer normalization on the transposed first enhanced features, and perform feature extraction on the layer normalized features through the Mamba model;

[0020] A linear layer is used to map the feature dimensions output by the Mamba model, and then the mapped feature dimensions are restored to obtain the output of the first MambaLayer module.

[0021] Furthermore, the specific process of filtering the key points of the key point map K and the feature heat map H by the key point screening module to obtain the key points is as follows:

[0022] Input the key point map K and feature heat map H into the key point screening module. First, the key point map Convert to keypoint heatmap ; Use nearest neighbor interpolation from keypoint heatmap Extract the characteristic response value of the candidate key point;

[0023] Use bilinear interpolation to extract the confidence scores of candidate key points from the feature heat map H; multiply the feature response value of the candidate key point and the confidence score of the candidate key point to obtain the comprehensive score of each candidate key point;

[0024] Combined with the geometric distribution information of the panorama to be matched, the comprehensive scores of the candidate key points are weighted and adjusted;

[0025] Multiply the adjusted weight value by the comprehensive score of the key point to adjust the final score of the candidate key point;

[0026] According to the final score of the adjusted candidate key points, all candidate key points are sorted in descending order and the top ones with the highest scores are selected. candidate key points as the final set of key points.

[0027] Furthermore, the obtained key points and intermediate feature maps F are input into the position-aware thin plate spline transform module. In the position-aware thin plate spline transform module, a thin plate spline transform tensor is first generated based on the position perception of the key points and the intermediate feature maps. Then, a polar coordinate grid of the key points is generated. The polar coordinate grid of the key points is thin plate spline transformed based on the thin plate spline transform tensor to obtain a deformed polar coordinate grid of the key points. The panorama to be matched is sampled based on the deformed polar coordinate grid of the key points to generate a deformed polar coordinate patch. The specific process is as follows:

[0028] The intermediate feature map F is first convolved twice and then nonlinearly transformed to generate the global thin plate spline transform coefficients ;

[0029] Using interpolation Projected to the position of each key point, the interpolator is based on the position of the key point from Generate the thin plate spline parameters corresponding to each key point ;

[0030] Will Input fully connected network to generate variation coefficient ;

[0031] Combining key points Axis coordinate pair Perform weighting and perform different thin plate spline transformations on key points at different positions; Weighted weight Expressed as:

[0032] ;

[0033] Where, Represents the learnable dimension weight parameters; is the height of the panorama to be processed;

[0034] Will and Multiply to get the thin plate spline transformation coefficient of each key point related to its own position ;

[0035] Generate an undeformed polar grid for each key point; interpolate the base grid of the generated undeformed polar grid into a control point grid , using the thin plate spline transformation function, Applied to the control point grid and polar coordinate grid, the deformed polar coordinate grid of the key points is obtained :

[0036] ;

[0037] Where, represents the thin plate spline transformation function; represents the resulting undeformed polar grid;

[0038] The captured panorama is sampled according to a deformed polar grid of key points to generate deformed polar patches.

[0039] Furthermore, using the thin plate spline transformation function, The process applied to the control point grid and the polar grid to obtain the deformed polar grid of the key points is:

[0040] First, according to the control point grid Compute the radial basis function:

[0041] ;

[0042] Where, represents the radial basis function, yes midpoint To control point grid Middle control point The Euclidean distance of

[0043] from Extract the affine parameter a and the radial basis function weight w from , multiply the affine parameter a and the radial basis function weight w; aggregate by matrix multiplication The radial basis function contribution of each control point in Perform matrix multiplication of the points in and the affine parameters to obtain The affine transformation contribution of each point in is added to the radial basis function contribution to obtain the deformed polar coordinate grid of the key points .

[0044] Furthermore, the key point description module is composed of a thirteenth convolutional layer, a fourteenth convolutional layer, a first depth-space conversion convolutional layer, a first spatial adaptive convolutional layer, a second depth-space conversion convolutional layer, a second spatial adaptive convolutional layer, a second average pooling layer, a fifteenth convolutional layer, a sixteenth convolutional layer, a seventeenth convolutional layer and an eighteenth convolutional layer connected in sequence; the processing flow of the key point description module is: the deformed polar coordinate patch passes through the thirteenth convolutional layer, the fourteenth convolutional layer, the first depth-space conversion convolutional layer, the first spatial adaptive convolutional layer, the second depth-space conversion convolutional layer, the second spatial adaptive convolutional layer, the second average pooling layer, the fifteenth convolutional layer, the sixteenth convolutional layer, the seventeenth convolutional layer and the eighteenth convolutional layer in sequence to obtain a feature descriptor.

[0045] Furthermore, the first spatial adaptive convolution layer and the second spatial adaptive convolution layer have the same structure, both including the 19th convolution layer, the 20th convolution layer and the 21st convolution layer; the processing flow of the first spatial adaptive convolution layer is: first, a learnable position map with the same output size as the first depth spatial conversion convolution layer is generated, and the learnable position map is passed through the 19th convolution layer, the 20th convolution layer and the 21st convolution layer in sequence to generate weights, and the weights and the output of the first depth spatial conversion convolution layer are reshaped and then multiplied element by element, and the sum is taken in the third dimension after reshaping to obtain the output of the first spatial adaptive convolution layer.

[0046] Furthermore, the Euclidean distance is used to complete the brute force matching of the feature descriptors of the two panoramas to be matched. For the feature descriptors of the given two panoramas to be matched, the Euclidean distance between the two is calculated to measure the similarity and perform matching.

[0047] Compared with the existing technology, the present invention has the following beneficial effects:

[0048] (1) The present invention constructs a new panoramic image feature matching model based on deep learning. First, through a feature extraction module composed of a MamaBlock module and a convolutional layer, the Mamba model is used to optimize the feature extraction efficiency and generate high-quality intermediate features and key points. Then, a position-aware thin plate spline transformation module is designed to generate accurate thin plate spline transformation coefficients and polar coordinate grids based on the position information of key points. The polar coordinate grid is then corrected by thin plate spline transformation, which effectively alleviates the geometric distortion of the panoramic image caused by spherical projection. Finally, a lightweight descriptor generation network is designed, which introduces deep spatial conversion convolution and spatial adaptive convolution to process low-resolution polar coordinate grids, and uses angular average pooling to improve the rotation invariance of the descriptor, thereby enhancing the quality and expressiveness of the descriptor.

[0049] (2) The present invention uses the MambaBlock module to scan and model the input features using two independent paths, row-first and column-first. The first path captures local horizontal continuity features along the horizontal direction (row-first), while the second path strengthens the vertical continuity features along the vertical direction (column-first). The two paths complement each other in the scanning direction through the transposition operation, perform selective state space modeling, and eliminate the influence of redundant information.

[0050] (3) The present invention designs a thin plate spline transformation module based on position perception. This module constructs a polar coordinate grid with the key point as the center, uses the radial-angular coordinate system instead of the traditional Cartesian coordinate system to eliminate the tangential stretching distortion caused by spherical projection, and calculates the latitude weight and combines the thin plate spline function to perform elastic grid deformation to alleviate the geometric distortion of the polar coordinate grid.

[0051] (4) The present invention designs spatially adaptive convolution, which dynamically generates convolution weights that are highly adapted to the current position context information by learning the spatial relationship of each pixel in the image. The spatial dependency between pixels is modeled through learnable position graph parameters, enabling the network to adaptively adjust the convolution operation for different image regions and generate high-dimensional compact feature descriptors, effectively alleviating the feature blurring problem caused by the reduction of resolution in traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Schematic diagram of the feature matching model structure of the present invention.

[0053] Figure 2 Schematic diagram of the feature extraction module structure of the present invention.

[0054] Figure 3 This is a schematic diagram of the MambaBlock module structure of the present invention.

[0055] Figure 4This is a schematic diagram of the module structure describing the key points of the present invention. DETAILED DESCRIPTION

[0056] like Figure 1 As shown, the present invention provides a technical solution: reading the panorama to be matched (set to two); building a feature matching model; the feature matching model includes a key point detection module, a position-aware thin plate spline transformation module and a key point description module; inputting the panorama to be matched into the key point detection module, performing feature extraction on the panorama to be matched to obtain a key point map K, a feature heat map H and an intermediate feature map F; performing feature point screening on the key point map K and the feature heat map H to obtain key points with high confidence; inputting the obtained key points and the intermediate feature map F into the position-aware thin plate spline transformation module (Thin Plate Spline Transformation Module). In the position-aware thin plate spline transform module, a thin plate spline transform tensor is first generated based on the position perception of the key points and the intermediate feature map, and then a polar coordinate grid of the key points is generated. The polar coordinate grid of the key points is thin plate spline transformed based on the thin plate spline transform tensor (to correct the polar coordinate grid of the key points and alleviate the geometric distortion and nonlinear deformation problems caused by the panorama), and a deformed polar coordinate grid of the key points is obtained. The panorama to be matched is sampled according to the deformed polar coordinate grid of the key points to generate a deformed polar coordinate patch, which is input into the key point description module to generate a feature descriptor of the panorama to be matched. Finally, a brute force match is performed on the feature descriptor of the panorama to be matched.

[0057] Among them, the Euclidean distance is used to complete the brute force matching of the feature descriptors of the two panoramas to be matched; for the feature descriptors of the given two panoramas to be matched, the matching is performed by calculating the Euclidean distance between the two to measure the similarity.

[0058] Among them, the key point detection module consists of a feature extraction module and a key point screening module. The feature extraction module extracts features from the panorama to be matched to obtain the key point map K, feature heat map H and intermediate feature map F; the key point screening module screens feature points from the key point map K and feature heat map H to obtain key points with high confidence.

[0059] like Figure 2As shown, the feature extraction module is designed based on the MambaBlock module, and the feature extraction module consists of a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, a fifth convolution block, a sixth convolution block, and a seventh convolution block; the processing flow of the feature extraction module is as follows: the input (panoramic image to be matched) passes through the first convolution block and the second convolution block respectively, and the output of the first convolution block and the output of the second convolution block are added and then passed through the fourth convolution block, the fifth convolution block, and the sixth convolution block in sequence to obtain an intermediate feature map F, and the intermediate feature map F passes through the seventh convolution block and the Sigmoid function in sequence to obtain a feature heat map H; the input (panoramic image to be matched) is divided into multiple 8×8 grids, and each grid is reshaped into a 64-dimensional feature vector , used to model the key point distribution within each grid, and then the feature vector After passing through the third convolution block, the key point map K is obtained.

[0060] The first convolutional block consists of the first convolutional layer (3×3), the first convolutional downsampling layer (3×3), the first MambaBlock module, the second MambaBlock module, and the second convolutional downsampling layer (3×3) connected in sequence. The input (the panorama to be matched) passes through the first convolutional layer, the first convolutional downsampling layer, the first MambaBlock module, the second MambaBlock module, and the second convolutional downsampling layer in sequence to obtain the output of the first convolutional block.

[0061] Among them, the second convolution block consists of the first average pooling layer (4×4) and the second convolution layer (1×1) connected in sequence; the input (panoramic image to be matched) passes through the first average pooling layer and the second convolution layer in sequence to obtain the output of the second convolution block.

[0062] Among them, the third convolution block is composed of the third convolution layer (1×1), the fourth convolution layer (1×1), the fifth convolution layer (1×1) and the sixth convolution layer (1×1) connected in sequence; the input (feature vector ) After passing through the third convolutional layer, the fourth convolutional layer, the fifth convolutional layer and the sixth convolutional layer, the key point map K is obtained.

[0063] Among them, the fourth convolution block is composed of the third MambaBlock module and the fourth MambaBlock module connected in sequence; the input (the output of the first convolution block and the output of the second convolution block are added) passes through the third MambaBlock module and the fourth MambaBlock module in sequence to obtain the output of the fourth convolution block.

[0064] Among them, the fifth convolution block consists of the third convolution downsampling layer (3×3), the fifth MambaBlock module and the sixth MambaBlock module connected in sequence; the input (the output of the fourth convolution block) passes through the third convolution downsampling layer, the fifth MambaBlock module and the sixth MambaBlock module in sequence to obtain the output of the fifth convolution block.

[0065] Among them, the sixth convolutional block is composed of the seventh convolutional layer (1×1), the eighth convolutional layer (1×1) and the ninth convolutional layer (1×1) connected in sequence; the input (the output of the fifth convolutional block) passes through the seventh convolutional layer, the eighth convolutional layer and the ninth convolutional layer in sequence to obtain the output of the sixth convolutional block.

[0066] Among them, the seventh convolution block is composed of the tenth convolution layer (1×1), the eleventh convolution layer (1×1) and the twelfth convolution layer (1×1) connected in sequence; the input (the output of the sixth convolution block) passes through the tenth convolution layer, the eleventh convolution layer and the twelfth convolution layer in sequence to obtain the output of the seventh convolution block.

[0067] Among them, the first convolution downsampling layer, the second convolution downsampling layer and the third convolution downsampling layer have the same structure, which are all convolutions with a convolution kernel of 3×3 and a stride of 2.

[0068] like Figure 3As shown in the figure, the first MambaBlock module, the second MambaBlock module, the third MambaBlock module, the fourth MambaBlock module, the fifth MambaBlock module and the sixth MambaBlock module have the same structure, and are composed of the first MambaLayer module, the second MambaLayer module, the third MambaLayer module, the fourth MambaLayer module and the thirteenth convolution layer; the processing flow of the first MambaBlock module is: the input (the output of the first convolution downsampling layer) is sequentially subjected to group normalization (Group Normalization) and ReLU activation function, the first enhanced feature is obtained, and the first enhanced feature is input into the first MambaLayer module for global context information modeling, while capturing long-distance dependencies and refining local features to obtain the second enhanced feature. The second enhanced feature is sequentially subjected to transposition, group normalization and ReLU activation function to obtain the third enhanced feature. The third enhanced feature is sequentially subjected to the second MambaLayer module and transposition to obtain the fourth enhanced feature; the input (the output of the first convolutional downsampling layer) is first transposed, and then sequentially subjected to group normalization and ReLU activation function to obtain the fifth enhanced feature. The fifth enhanced feature is subjected to the third MambaLayer module and transposition to obtain the sixth enhanced feature. The sixth enhanced feature is subjected to group normalization and ReLU activation function to obtain the seventh enhanced feature. The seventh enhanced feature is subjected to the fourth MambaLayer module to obtain the eighth enhanced feature. The fourth enhanced feature and the eighth enhanced feature are concatenated and sequentially subjected to the thirteenth convolution layer, batch normalization (Batch Normalization) and ReLU activation function are performed to obtain the final output of the first MambaBlock module. Since the second MambaBlock module, the third MambaBlock module, the fourth MambaBlock module, the fifth MambaBlock module and the sixth MambaBlock module have the same structure as the first MambaBlock module, their processing flow will not be described in detail here.

[0069] Among them, the first MambaLayer module, the second MambaLayer module, the third MambaLayer module and the fourth MambaLayer module have the same structure; the processing flow of the first MambaLayer module is as follows: first, the input (first enhanced feature) is transposed: the spatial dimension of the input (first enhanced feature) is flattened into a single dimension, and the channel dimension and the sequence dimension are interchanged. This transposition operation uses the sequence dimension as the main modeling dimension, which helps to capture the contextual relationship of the feature points.

[0070] In order to maintain the consistency of feature distribution and avoid gradient problems, layer normalization is performed on the transposed first enhanced features, and feature extraction is performed on the layer normalized features through the Mamba model.

[0071] A linear layer is used to map the feature dimensions output by the Mamba model. Subsequently, the mapped feature dimensions are restored: First, the sequence dimension of the features output by the Mamba model is restored ( ) and feature dimension ( ) and then transpose it. Finally, the reshape operation is used to restore the transposed sequence dimension to its original spatial shape, and the shape of the output feature is restored to be consistent with the input shape, that is, the output of the first MambaLayer module is obtained.

[0072] Among them, the specific process of filtering the key points of the key point map K and the feature heat map H through the key point screening module to obtain the key points with high confidence is as follows:

[0073] Input the key point map K and feature heat map H into the key point screening module. First, the key point map Convert to a higher resolution keypoint heatmap , locate key points more accurately; use nearest neighbor interpolation method to get the key point heat map The characteristic response values of the candidate key points are extracted from the CNN. This method can quickly obtain the characteristic response strength of the candidate key points.

[0074] The confidence scores of candidate key points are extracted from the feature heat map H using bilinear interpolation to evaluate the reliability and stability of the candidate key points. The feature response value of the candidate key point and the confidence score of the candidate key point are multiplied together to obtain the comprehensive score of each candidate key point. The comprehensive score is a key indicator for measuring the quality of the key point and can generate a set of high-quality feature points and their corresponding confidence scores.

[0075] Combined with the geometric distribution information of the panorama to be matched, the weight of the comprehensive score of the candidate key points is adjusted; in the panorama, due to geometric projection, the geometric deformation of the image near the equator is small, while the geometric deformation of the image near the North and South Poles is large. Therefore, the candidate key points near the equator are given a higher weight, while the candidate key points near the North and South Poles are given a lower weight. This weight adjustment method can effectively compensate for the influence of geometric distortion in the panorama; the weight of the candidate key points after the comprehensive score adjustment Expressed as:

[0076] ;

[0077] Where, For key points axis coordinates; The height of the panorama to be processed.

[0078] The final score of the candidate key points is adjusted by multiplying the adjusted weight value with the comprehensive score of the key point to ensure that the candidate key points with higher scores not only have high response values and high confidence scores, but also have smaller geometric deformation in the panorama. To further optimize the results, for invalid key points (points with all zero coordinates), their scores are set to −1 to ensure that these invalid key points are eliminated in the subsequent sorting and screening process.

[0079] According to the final score of the adjusted candidate key points, all candidate key points are sorted in descending order and the top ones with the highest scores are selected. candidate key points as the final set of key points. The value of is generally set to 2048, that is, the best 2048 key points are selected from all possible candidate key points.

[0080] The key points and intermediate feature maps F are input into the position-aware thin plate spline transform module. In the position-aware thin plate spline transform module, a thin plate spline transform tensor is first generated based on the position perception of the key points and the intermediate feature maps. Then, a polar coordinate grid of the key points is generated. The polar coordinate grid of the key points is thin plate spline transformed based on the thin plate spline transform tensor (to correct the polar coordinate grid of the key points and alleviate the geometric distortion and nonlinear deformation problems caused by the panorama), and a deformed polar coordinate grid of the key points is obtained. The panorama to be matched is sampled based on the deformed polar coordinate grid of the key points to generate a deformed polar coordinate patch. The specific process is as follows:

[0081] The intermediate feature map F is first subjected to two 3×3 convolutions and then nonlinear transformations to generate global thin plate spline transform coefficients. .

[0082] In order to generate specific thin plate spline parameters for each key point, an interpolator is used to Projected to the position of each key point, the interpolator is based on the position of the key point from Generate the thin plate spline parameters corresponding to each key point :

[0083] ;

[0084] Where, represents an interpolator; Indicates the location of key points; Indicates the height of the panorama to be processed; Indicates the width of the panorama to be processed.

[0085] Will Input the fully connected network to generate the required variation coefficients of the thin plate spline (TPS) .

[0086] Combining key points Axis coordinate pair Perform weighting and perform different thin plate spline transformations on key points at different positions; Weighted weight Expressed as:

[0087] ;

[0088] Where, Represents the learnable dimension weight parameters.

[0089] Will and Multiply to get the thin plate spline transformation coefficient of each key point related to its own position .

[0090] Generate an undeformed polar grid for each key point; interpolate the base grid of the generated undeformed polar grid into a control point grid , using the thin plate spline transformation function, Applied to the control point grid and polar coordinate grid, the deformed polar coordinate grid of the key points is obtained :

[0091] ;

[0092] Where, represents the thin plate spline transformation function; represents the resulting undeformed polar grid.

[0093] Use the grid_sample function to sample the panorama to be matched according to the deformed polar coordinate grid of key points to generate deformed polar coordinate patches.

[0094] Among them, using the thin plate spline transformation function, The process applied to the control point grid and the polar grid to obtain the deformed polar grid of the key points is:

[0095] First, according to the control point grid Compute the radial basis function:

[0096] ;

[0097] Where, represents the radial basis function, yes midpoint To control point grid Middle control point The Euclidean distance of .

[0098] from Extract the affine parameter a and the radial basis function weight w from , multiply the affine parameter a and the radial basis function weight w; aggregate by matrix multiplication The radial basis function contribution of each control point in Perform matrix multiplication of the points in and the affine parameters to obtain The affine transformation contribution of each point in is added to the radial basis function contribution to obtain the deformed polar coordinate grid of the key points .

[0099] like Figure 4 As shown in the figure, the key point description module consists of the 13th convolutional layer (3×3), the 14th convolutional layer (3×3), the first depth-space conversion convolutional layer, the first spatial adaptive convolutional layer, the second depth-space conversion convolutional layer, the second spatial adaptive convolutional layer, the second average pooling layer (8×1), the 15th convolutional layer (1×3), the 16th convolutional layer (1×3), the 17th convolutional layer (1×3) and the 18th convolutional layer (1×2), which are connected in sequence; the processing flow of the key point description module is as follows: the input (deformed polar coordinate patch) passes through the 13th convolutional layer, the 14th convolutional layer, the first depth-space conversion convolutional layer, the first spatial adaptive convolutional layer, the second depth-space conversion convolutional layer, the second spatial adaptive convolutional layer, the second average pooling layer, the 15th convolutional layer, the 16th convolutional layer, the 17th convolutional layer and the 18th convolutional layer in sequence to obtain the output (feature descriptor).

[0100] Among them, the first deep spatial conversion convolution layer (SPD-Conv) and the second deep spatial conversion convolution layer have the same structure.

[0101] Among them, the first spatial adaptive convolution layer and the second spatial adaptive convolution layer have the same structure, both including the 19th convolution layer (3×3), the 20th convolution layer (3×3) and the 21st convolution layer (3×3); the processing flow of the first spatial adaptive convolution layer is: first, a learnable position map of the same size as the input (the output of the first depth-space conversion convolution layer) is generated, and the learnable position map is passed through the 19th convolution layer, the 20th convolution layer and the 21st convolution layer in sequence to generate weights, and the weights and input (the output of the first depth-space conversion convolution layer) are reshaped and then multiplied element by element, and the sum is taken in the third dimension after reshaping to obtain the output of the first spatial adaptive convolution layer.

[0102] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A panoramic image feature matching method based on position-aware thin plate spline transform, characterized in that: The method includes the following steps: reading the panorama to be matched; building a feature matching model; The feature matching model includes a key point detection module, a position-aware thin plate spline transformation module and a key point description module; the panorama to be matched is input into the key point detection module, and features of the panorama to be matched are extracted to obtain a key point map, a feature heat map and an intermediate feature map, and feature points are screened on the key point map and the feature heat map to obtain key points; the obtained key points and intermediate feature maps are input into the position-aware thin plate spline transformation module, in which a thin plate spline transformation tensor is first generated based on the position perception and intermediate feature map of the key points, and then a polar coordinate grid of the key points is generated, and a thin plate spline transformation is performed on the polar coordinate grid of the key points based on the thin plate spline transformation tensor to obtain a deformed polar coordinate grid of the key points; the panorama to be matched is sampled according to the deformed polar coordinate grid of the key points to generate a deformed polar coordinate patch, and the deformed polar coordinate patch is input into the key point description module to generate a feature descriptor of the panorama to be matched, and finally a brute force matching is performed on the feature descriptors of the panorama to be matched; two panoramas to be matched are read, and finally a brute force matching is performed on the feature descriptors of the two obtained panoramas to be matched; Among them, the key point detection module consists of a feature extraction module and a key point screening module. The feature extraction module extracts features from the panorama to be matched to obtain a key point map, a feature heat map and an intermediate feature map; the key point screening module screens feature points from the key point map and the feature heat map to obtain key points.

2. The panoramic image feature matching method based on position-aware thin plate spline transform according to claim 1, characterized in that: The feature extraction module consists of a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, a fifth convolution block, a sixth convolution block and a seventh convolution block. The processing flow of the feature extraction module is as follows: the panoramic image to be matched passes through the first convolution block and the second convolution block respectively, the output of the first convolution block and the output of the second convolution block are added, and then the intermediate feature map F is obtained after the intermediate feature map F passes through the seventh convolution block and the Sigmoid function in sequence to obtain the feature heat map H; the panoramic image to be matched is divided into multiple grids, and each grid is reshaped into a feature vector , then the eigenvector After passing through the third convolution block, the key point map K is obtained; The first convolution block is composed of a first convolution layer, a first convolution downsampling layer, a first MambaBlock module, a second MambaBlock module, and a second convolution downsampling layer connected in sequence; the panorama to be matched is sequentially passed through the first convolution layer, the first convolution downsampling layer, the first MambaBlock module, the second MambaBlock module, and the second convolution downsampling layer to obtain the output of the first convolution block; The second convolution block is composed of a first average pooling layer and a second convolution layer connected in sequence; the panorama to be matched is sequentially passed through the first average pooling layer and the second convolution layer to obtain the output of the second convolution block; The third convolutional block consists of a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer and a sixth convolutional layer connected in sequence; The feature vector After passing through the third convolutional layer, the fourth convolutional layer, the fifth convolutional layer and the sixth convolutional layer in sequence, the key point map K is obtained; The fourth convolution block is composed of a third MambaBlock module and a fourth MambaBlock module connected in sequence; the output of the first convolution block and the output of the second convolution block are added, and the output is passed through the third MambaBlock module and the fourth MambaBlock module in sequence to obtain the output of the fourth convolution block; The fifth convolution block is composed of a third convolution downsampling layer, a fifth MambaBlock module, and a sixth MambaBlock module connected in sequence; the output of the fourth convolution block is sequentially passed through the third convolution downsampling layer, the fifth MambaBlock module, and the sixth MambaBlock module to obtain the output of the fifth convolution block; The sixth convolutional block is composed of a seventh convolutional layer, an eighth convolutional layer, and a ninth convolutional layer connected in sequence; the output of the fifth convolutional block is sequentially passed through the seventh convolutional layer, the eighth convolutional layer, and the ninth convolutional layer to obtain the output of the sixth convolutional block; The seventh convolution block is composed of a tenth convolution layer, an eleventh convolution layer, and a twelfth convolution layer connected in sequence; the output of the sixth convolution block is sequentially passed through the tenth convolution layer, the eleventh convolution layer, and the twelfth convolution layer to obtain the output of the seventh convolution block.

3. The panoramic image feature matching method based on position-aware thin plate spline transform according to claim 2, characterized in that: The first MambaBlock module, the second MambaBlock module, the third MambaBlock module, the fourth MambaBlock module, the fifth MambaBlock module and the sixth MambaBlock module have the same structure, and are all composed of a first MambaLayer module, a second MambaLayer module, a third MambaLayer module, a fourth MambaLayer module and a thirteenth convolutional layer; the processing flow of the first MambaBlock module is: the output of the first convolutional downsampling layer is sequentially subjected to group normalization and ReLU activation function to obtain a first enhanced feature, the first enhanced feature is input into the first MambaLayer module for processing to obtain a second enhanced feature, the second enhanced feature is sequentially subjected to transposition, group normalization and ReLU activation function to obtain a third enhanced feature, and the third enhanced feature is sequentially subjected to the second MambaLayer module and transposition to obtain a fourth enhanced feature; The output of the first convolutional downsampling layer is first transposed, and then sequentially subjected to group normalization and ReLU activation function to obtain the fifth enhanced feature. The fifth enhanced feature is passed through the third MambaLayer module and transposed to obtain the sixth enhanced feature. The sixth enhanced feature is passed through group normalization and ReLU activation function to obtain the seventh enhanced feature. The seventh enhanced feature is passed through the fourth MambaLayer module to obtain the eighth enhanced feature. The fourth and eighth enhanced features are spliced and sequentially subjected to the thirteenth convolutional layer, batch normalization and ReLU activation function to obtain the final output of the first MambaBlock module.

4. The panoramic image feature matching method based on position-aware thin plate spline transform according to claim 3, characterized in that: The first MambaLayer module, the second MambaLayer module, the third MambaLayer module and the fourth MambaLayer module have the same structure; the processing flow of the first MambaLayer module is: first, the first enhancement feature is transposed; Perform layer normalization on the transposed first enhanced features, and perform feature extraction on the layer normalized features through the Mamba model; A linear layer is used to map the feature dimensions output by the Mamba model, and then the mapped feature dimensions are restored to obtain the output of the first MambaLayer module.

5. The panoramic image feature matching method based on position-aware thin plate spline transform according to claim 4, characterized in that: The specific process of filtering the key points of the key point map K and the feature heat map H by the key point screening module to obtain the key points is as follows: Input the key point map K and feature heat map H into the key point screening module. First, the key point map Convert to keypoint heatmap ; Use nearest neighbor interpolation from keypoint heatmap Extract the characteristic response value of the candidate key point; Use bilinear interpolation to extract the confidence scores of candidate key points from the feature heat map H; multiply the feature response value of the candidate key point and the confidence score of the candidate key point to obtain the comprehensive score of each candidate key point; Combined with the geometric distribution information of the panorama to be matched, the comprehensive scores of the candidate key points are weighted and adjusted; Multiply the adjusted weight value by the comprehensive score of the key point to adjust the final score of the candidate key point; According to the final score of the adjusted candidate key points, all candidate key points are sorted in descending order and the top ones with the highest scores are selected. candidate key points as the final set of key points.

6. The panoramic image feature matching method based on position-aware thin plate spline transform according to claim 5, characterized in that: The obtained key points and intermediate feature maps F are input to the position-aware thin plate spline transform module. In the position-aware thin plate spline transform module, a thin plate spline transform tensor is first generated based on the position perception of the key points and the intermediate feature maps. Then, a polar coordinate grid of the key points is generated. The polar coordinate grid of the key points is transformed by thin plate spline transform based on the thin plate spline transform tensor to obtain a deformed polar coordinate grid of the key points. The panorama to be matched is sampled based on the deformed polar coordinate grid of the key points to generate a deformed polar coordinate patch. The specific process is as follows: The intermediate feature map F is first convolved twice and then nonlinearly transformed to generate the global thin plate spline transform coefficients ; Using interpolation Projected to the position of each key point, the interpolator is based on the position of the key point from Generate the thin plate spline parameters corresponding to each key point ; Will Input fully connected network to generate variation coefficient ; Combining key points Axis coordinate pair Perform weighting and perform different thin plate spline transformations on key points at different positions; Weighted weight Expressed as: ； Where, Represents the learnable dimension weight parameters; is the height of the panorama to be processed; Will and Multiply to get the thin plate spline transformation coefficient of each key point related to its own position ; Generate an undeformed polar grid for each key point; interpolate the base grid of the generated undeformed polar grid into a control point grid , using the thin plate spline transformation function, Applied to the control point grid and polar coordinate grid, the deformed polar coordinate grid of the key points is obtained : ； Where, represents the thin plate spline transformation function; represents the resulting undeformed polar grid; The captured panorama is sampled according to a deformed polar grid of key points to generate deformed polar patches.

7. The panoramic image feature matching method based on position-aware thin plate spline transform according to claim 6, characterized in that: Using the thin plate spline transformation function, The process applied to the control point grid and the polar grid to obtain the deformed polar grid of the key points is: First, according to the control point grid Compute the radial basis function: ； Where, represents the radial basis function, yes midpoint To control point grid Middle control point The Euclidean distance of from Extract the affine parameter a and the radial basis function weight w from , multiply the affine parameter a and the radial basis function weight w; aggregate by matrix multiplication The radial basis function contribution of each control point in Perform matrix multiplication of the points in and the affine parameters to obtain The affine transformation contribution of each point in is added to the radial basis function contribution to obtain the deformed polar coordinate grid of the key points .

8. The panoramic image feature matching method based on position-aware thin plate spline transform according to claim 7, characterized in that: The key point description module is composed of a thirteenth convolutional layer, a fourteenth convolutional layer, a first depth-space conversion convolutional layer, a first spatial adaptive convolutional layer, a second depth-space conversion convolutional layer, a second spatial adaptive convolutional layer, a second average pooling layer, a fifteenth convolutional layer, a sixteenth convolutional layer, a seventeenth convolutional layer and an eighteenth convolutional layer connected in sequence; the processing flow of the key point description module is: the deformed polar coordinate patch passes through the thirteenth convolutional layer, the fourteenth convolutional layer, the first depth-space conversion convolutional layer, the first spatial adaptive convolutional layer, the second depth-space conversion convolutional layer, the second spatial adaptive convolutional layer, the second average pooling layer, the fifteenth convolutional layer, the sixteenth convolutional layer, the seventeenth convolutional layer and the eighteenth convolutional layer in sequence to obtain a feature descriptor.

9. The panoramic image feature matching method based on position-aware thin plate spline transform according to claim 8, characterized in that: The first spatial adaptive convolutional layer and the second spatial adaptive convolutional layer have the same structure, both including the 19th convolutional layer, the 20th convolutional layer and the 21st convolutional layer; the processing flow of the first spatial adaptive convolutional layer is as follows: first, a learnable position map with the same output size as the first depth-space conversion convolutional layer is generated, and the learnable position map is passed through the 19th convolutional layer, the 20th convolutional layer and the 21st convolutional layer in sequence to generate weights, and the weights and the output of the first depth-space conversion convolutional layer are reshaped and then multiplied element by element, and the sum is taken in the third dimension after reshaping to obtain the output of the first spatial adaptive convolutional layer.

10. The panoramic image feature matching method based on position-aware thin plate spline transform according to claim 1, characterized in that: The Euclidean distance is used to complete the brute force matching of the feature descriptors of the two panoramas to be matched. For the feature descriptors of the given two panoramas to be matched, the Euclidean distance between the two is calculated to measure the similarity and perform matching.

Images