Real-time feature point extraction and matching method based on star operation and gateMLP

Abstract

The application provides a real-time feature point extraction and matching method based on star operation and GateMLP, which comprises the following steps: S1, preprocessing the input image and inputting the image into a lightweight convolutional network to extract initial features; S2, based on the output feature map, deep feature modeling is performed through the structure of the fusion of a convolution branch and a star operation module, and a dense descriptor and a feature point heat map are output; S3, non-maximum suppression is performed on the heat map, and key point coordinates are screened in combination with a confidence score; S4, cosine similarity and a nearest neighbor strategy are adopted to coarsely match the key points of left and right views, and an initial matching pair is obtained; and S5, the matching descriptor obtained through coarse matching is input into a gated multilayer perceptron for fine-grained matching, and accurate matching point pairs are output. The method has good real-time performance and robustness while ensuring high extraction and matching accuracy, and is suitable for practical scenes such as unmanned aerial vehicle navigation and SLAM that require fast response.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a real-time feature point extraction and matching method based on star operation and GateMLP. Background Technology

[0002] Feature point extraction and matching are core technologies in computer vision. By identifying stable local features in images and establishing cross-viewpoint correspondences, they provide fundamental support for tasks such as autonomous driving, drone navigation, and SLAM. In these real-time dynamic scenarios, the accuracy and speed of feature point extraction and matching directly determine the system's response performance. Therefore, methods that balance high accuracy and high real-time performance are key requirements for practical applications.

[0003] In existing technologies, feature point extraction and matching methods face the challenge of balancing real-time performance and accuracy. Traditional manual feature algorithms, such as SIFT and ORB, are relatively stable in specific scenarios, but their matching accuracy drops significantly when faced with complex environments. On the other hand, deep learning-based methods, such as SuperPoint and LF-Net, while possessing certain accuracy and robustness, suffer from large model parameters and high computational complexity, making them difficult to meet the needs of resource-constrained or real-time-critical scenarios such as mobile devices and autonomous driving. Summary of the Invention

[0004] To address these issues, this invention proposes a real-time feature point extraction and matching method based on star computation and GateMLP. Through lightweight network design and refined matching strategies, it achieves efficient inference while ensuring accuracy, solving the problem of difficulty in balancing accuracy and real-time performance in existing technologies and meeting the needs of practical applications.

[0005] This invention relates to feature point extraction and matching technology in computer vision, and is particularly suitable for feature point extraction and matching tasks in dynamic visual scenes with high real-time requirements. It can be applied to practical application scenarios such as autonomous driving, drone navigation, SLAM, and robot vision that require fast and accurate feature point extraction and matching.

[0006] The main technical solution adopted in this invention is as follows:

[0007] This invention provides a real-time feature point extraction and matching method based on star operation and GateMLP, comprising the following steps:

[0008] Step S1: Preprocess the input image, i.e., normalize it. The input image includes the left and right views and is fed into a lightweight convolutional network to extract initial features and output feature maps at multiple scales.

[0009] Step S1.1: Convert the input image pairs to be processed into tensors in (B,C,H,W) format (B is the batch size, C is the number of channels, H is the height of the image, and W is the width of the image). Then, adjust the height and width of the image to be the closest integer multiple of 32 that is not greater than the original value through bilinear interpolation. Normalize the pixel values ​​to the [0,1] range and record the scaling ratio for subsequent key point coordinate restoration.

[0010] Step S1.2: Input the preprocessed image into a lightweight convolutional network consisting of blocks 1 to 5. This network contains 16 ordinary convolutional layers with a kernel size of 3×3 as the main type and 1×1 convolution in some layers. Multi-scale downsampling from the input image size H×W to H / 32×W / 32 is achieved through convolutional layers with a stride of 2. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. Skip connections are introduced in block 2 to fuse initial features to enhance feature propagation. Finally, feature maps of multiple scales are output. Step S2: Based on the output feature map, deep feature modeling is performed by fusing the structure of convolutional branches and star operation modules, and dense descriptors and feature point heatmaps are output.

[0011] Step S2.1: Construct a parallel structure that integrates the convolutional branch and the star operation module, containing multiple layers of 3×3 ordinary convolutions and 1×1 convolutions, used for fusing and adjusting channels of multi-scale features; the star operation module processes multi-scale features (H / 8×W / 8, H / 16×W / 16, H / 32×W / 32) separately. First, local features are extracted by 3×3 depthwise separable convolutions, and then feature maps x1 and x2 are generated by two parallel 1×1 convolutions. x1 is activated by ReLU6 and then multiplied element-wise with x2. The resulting feature map is adjusted in dimension by 1×1 convolutions and further processed by 3×3 depthwise separable convolutions. Finally, it is added to the input features to form a residual connection, realizing the aggregation of global context features.

[0012] Step S2.2: After adding and fusing the output features of the convolution branch and the star operation module, a dense descriptor with a dimension of 64 is generated by 1×1 convolution compression to represent local image features. At the same time, a nonlinear transformation is performed on the fused features and a sigmoid activation is applied to generate a heatmap, where each pixel value represents the confidence that a feature point exists at that location. In addition, by performing 8×8 neighborhood window feature extraction and regression processing on the input image, a tensor for predicting the offset of key point positions is generated. The three together constitute the output of step S2.2, providing basic data for subsequent key point detection and feature matching.

[0013] Step S3: Perform non-maximum suppression (NMS) on the heatmaps corresponding to the left and right views, and filter the key point coordinates by combining the confidence score to obtain the set of key points and their descriptors for the left and right views;

[0014] Step S3.1: Perform non-maximum suppression on the heatmaps corresponding to the left and right views. Use a 5×5 neighborhood window to perform NMS. Find local maximum points through maximum pooling. Only retain points in the heatmap that are equal to the local maximum and whose values ​​are higher than the detection threshold as candidate key points.

[0015] Step S3.2: Combine the descriptors corresponding to the candidate key points with the heatmap scores to calculate the confidence level, filter out the candidate points with confidence levels higher than the set threshold, map their coordinates back to the original image size according to the scaling ratio recorded in the preprocessing stage, record the mapped coordinates as the final key points, and retain their corresponding descriptor vectors for subsequent matching.

[0016] Step S4: Use cosine similarity and nearest neighbor strategy (MNN) to perform coarse matching on the key points of the left and right views to obtain initial matching pairs;

[0017] Step S4.1: Calculate the cosine similarity between the key point descriptor of the left view and all key point descriptors of the right view, and select the right view key point with the highest similarity for each left view key point as a candidate match;

[0018] Step S4.2: Perform nearest neighbor verification: If the best match of key point A in the left view is key point B in the right view, and the best match of key point B in the right view is also key point A in the left view, then retain the matching pair as the coarse matching result, and remove matching pairs with similarity below the set threshold.

[0019] Step S5: Input the pairing descriptors obtained from coarse matching into a gated multilayer perceptron for fine-grained matching, and output accurate matching point pairs.

[0020] Step S5.1: Concatenate the descriptors of the coarse matching pairs along the channel dimension into a 128-dimensional feature vector, and input it into the GateMLP network consisting of 3 layers of gated linear units. Each layer of gated linear units includes linear transformation, batch normalization and gate mechanism.

[0021] Step S5.2: The GateMLP network outputs the sub-pixel level offset probability distribution within an 8×8 window. After normalization by the Softmax function, the offset with the highest probability is taken as the correction value.

[0022] Step S5.3: Overlay the correction value onto the coarse matching key point coordinates to obtain accurate sub-pixel level coordinates, remove matching pairs whose distance exceeds the preset threshold after correction, and output the final matching point pairs.

[0023] The specific steps for feature point extraction in steps S1, S2, and S3 are as follows:

[0024] The beneficial effects of this invention are that it provides a real-time feature point extraction and matching method based on star operation and GateMLP, which can well balance the accuracy and real-time requirements in the feature point extraction and matching process, and improve the stability of feature point extraction and the accuracy of matching in complex scenarios.

[0025] The method proposed in this invention differs from previous feature point extraction and matching algorithms. Firstly, in the feature extraction stage, a parallel structure integrating convolutional branches and a star operation module is designed. The star operation module aggregates local texture and global context features through depthwise separable convolutions and residual connections, reducing computation while maintaining the discriminative power of feature descriptors. This lightweight network structure meets real-time requirements. Secondly, in the matching stage, GateMLP is introduced for fine-grained optimization. Coordinate correction is achieved by predicting sub-pixel-level offset distribution within an 8×8 window, combined with a nearest neighbor strategy to select reliable matching pairs, improving matching accuracy while avoiding complex calculations. In summary, the method of this invention offers superior performance compared to traditional algorithms for feature point extraction and matching in dynamic scenes with high real-time requirements, achieving a balance between accuracy and efficiency. Attached Figure Description

[0026] Figure 1 This is a flowchart of a real-time feature point extraction and matching method based on star operation and GateMLP;

[0027] Figure 2 Overall model structure diagram;

[0028] Figure 3 Descriptor extraction model structure diagram;

[0029] Figure 4 Structure diagram of coarse-to-fine matching model

[0030] Figure 5 Output of the disparity map after multi-disparity optimization. Detailed Implementation

[0031] The following figures illustrate in detail the various aspects involved in the technical solution of this invention.

[0032] A real-time feature point extraction and matching method based on star operation and GateMLP is presented. The overall structure of the feature point extraction model is shown in Figure 2. Figure 1 As shown, the specific steps are as follows:

[0033] Step S1: Preprocess the input image, i.e. normalize it. The input image includes the left view and the right view. It is fed into a lightweight convolutional network to extract initial features and output feature maps at multiple scales.

[0034] Step S2: Based on the output feature map, deep feature modeling is performed by fusing the structure of convolutional branches and star operation modules, and the dense descriptors and feature point heatmaps of the left and right views are output.

[0035] Step S3: Perform non-maximum suppression on the heatmaps corresponding to the left and right views, and filter the key point coordinates by combining the confidence score to obtain the set of key points and their descriptors for the left and right views;

[0036] Step S4: Use cosine similarity and nearest neighbor strategy (MNN) to perform coarse matching on the key points of the left and right views to obtain initial matching pairs;

[0037] Step S5: Input the pairing descriptors obtained from coarse matching into a gated multilayer perceptron for fine-grained matching, and output accurate matching point pairs.

[0038] The specific steps involved in step S1, which involve preprocessing the input image pairs and feeding them into a lightweight convolutional network to extract initial features, are as follows:

[0039] Step S1.1: Convert the input image pairs to be processed into tensors in (B,C,H,W) format (B is the batch size, C is the number of channels, H is the height of the image, and W is the width of the image). Then, adjust the height and width of the image to be the closest integer multiple of 32 that is not greater than the original value through bilinear interpolation. Normalize the pixel values ​​to the [0,1] range and record the scaling ratio for subsequent key point coordinate restoration.

[0040] Step S1.2: Input the preprocessed image into a lightweight convolutional network consisting of blocks 1 to 5. This network contains 16 ordinary convolutional layers with a kernel size of 3×3 as the main type and 1×1 convolution in some layers. Multi-scale downsampling from the input image size H×W to H / 32×W / 32 is achieved through convolutional layers with a stride of 2. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. Skip connections are introduced in block 2 to fuse initial features to enhance feature propagation. Finally, feature maps of multiple scales are output.

[0041] Step S2 involves deep feature modeling by fusing convolutional branches and star operation modules, outputting dense descriptors and feature point heatmaps, as shown in Figure 3. The specific steps are as follows:

[0042] Step S2.1: Construct a parallel structure that integrates the convolutional branch and the star operation module, containing multiple layers of 3×3 ordinary convolutions and 1×1 convolutions, used for fusing and adjusting channels of multi-scale features; the star operation module processes multi-scale features (H / 8×W / 8, H / 16×W / 16, H / 32×W / 32) separately. First, local features are extracted by 3×3 depthwise separable convolutions, and then feature maps x1 and x2 are generated by two parallel 1×1 convolutions. x1 is activated by ReLU6 and then multiplied element-wise with x2. The resulting feature map is adjusted in dimension by 1×1 convolutions and further processed by 3×3 depthwise separable convolutions. Finally, it is added to the input features to form a residual connection, realizing the aggregation of global context features.

[0043] Step S2.2: After adding and fusing the output features of the convolution branch and the star operation module, a dense descriptor with a dimension of 64 is generated by 1×1 convolution compression to represent local image features. At the same time, a nonlinear transformation is performed on the fused features and a sigmoid activation is applied to generate a heatmap, where each pixel value represents the confidence that a feature point exists at that location. In addition, by performing 8×8 neighborhood window feature extraction and regression processing on the input image, a tensor for predicting the offset of key point positions is generated. The three together constitute the output of step S2.2, providing basic data for subsequent key point detection and feature matching.

[0044] The specific steps involved in step S3, which involve performing non-maximum suppression on the heatmaps corresponding to the left and right views and filtering the key point coordinates based on the confidence score, are as follows:

[0045] Step S3.1: Perform non-maximum suppression on the heatmaps corresponding to the left and right views. Use a 5×5 neighborhood window to perform NMS. Find local maximum points through maximum pooling. Only retain points in the heatmap that are equal to the local maximum and whose values ​​are higher than the detection threshold as candidate key points.

[0046] Step S3.2: Combine the descriptors corresponding to the candidate key points with the heatmap scores to calculate the confidence level, filter out the candidate points with confidence levels higher than the set threshold, map their coordinates back to the original image size according to the scaling ratio recorded in the preprocessing stage, record the mapped coordinates as the final key points, and retain their corresponding descriptor vectors for subsequent matching.

[0047] The feature point matching methods designed in steps S4 and S5 are shown in Figure 4.

[0048] The specific steps involved in step S4, which use cosine similarity and nearest neighbor (MNN) strategies to perform coarse matching of key points in the left and right views, are as follows:

[0049] Step S4.1: Calculate the cosine similarity between the key point descriptor of the left view and all key point descriptors of the right view, and select the right view key point with the highest similarity for each left view key point as a candidate match;

[0050] Step S4.2: Perform nearest neighbor verification: If the best match of key point A in the left view is key point B in the right view, and the best match of key point B in the right view is also key point A in the left view, then retain the matching pair as the coarse matching result, and remove matching pairs with similarity below the set threshold.

[0051] The specific steps involved in step S5, which involve inputting the pairing descriptors obtained from coarse matching into a gated multilayer perceptron for fine-grained matching and outputting accurate matching point pairs, are as follows:

[0052] Step S5.1: Concatenate the descriptors of the coarse matching pairs along the channel dimension into a 128-dimensional feature vector, and input it into the GateMLP network consisting of 3 layers of gated linear units. Each layer of gated linear units includes linear transformation, batch normalization and gate mechanism.

[0053] Step S5.2: The GateMLP network outputs the sub-pixel level offset probability distribution within an 8×8 window. After normalization by the Softmax function, the offset with the highest probability is taken as the correction value.

[0054] Step S5.3: Overlay the correction value onto the coarse matching key point coordinates to obtain accurate sub-pixel level coordinates. Remove matching pairs whose distance exceeds the preset threshold after correction, and output the final matching point pairs, as shown in Figure 5.

Claims

1. A real-time feature point extraction and matching method based on star operation and GateMLP, characterized in that, Includes the following steps: Step S1: Preprocess the input image, i.e., normalize it. The input image includes the left view and the right view. It is fed into a lightweight convolutional network to extract initial features and output feature maps at multiple scales. Step S2: Based on the output feature map, deep feature modeling is performed by fusing the structure of convolutional branches and star operation modules, and the dense descriptors and feature point heatmaps of the left and right views are output. Step S3: Perform non-maximum suppression on the heatmaps corresponding to the left and right views, and filter the key point coordinates by combining the confidence score to obtain the set of key points and their descriptors for the left and right views; Step S4: Use cosine similarity and nearest neighbor strategies to perform coarse matching on the key points of the left and right views to obtain initial matching pairs; Step S5: Input the pairing descriptors obtained from coarse matching into a gated multilayer perceptron for fine-grained matching, and output accurate matching point pairs; The specific steps of step S1 are as follows: Step S1.1: Convert the input image pairs to be processed into tensors in (B,C,H,W) format, where B is the batch size, C is the number of channels, H is the height of the image, and W is the width of the image. Then, adjust the height and width of the image to be integer multiples of 32 closest to and no greater than the original values ​​through bilinear interpolation. Normalize the pixel values ​​to the [0,1] interval and record the scaling ratio for subsequent key point coordinate restoration. Step S1.2: Input the preprocessed image into a lightweight convolutional network consisting of blocks 1 to 5. The lightweight convolutional network contains 16 ordinary convolutional layers with a kernel size of 3×3 as the main type and 1×1 convolution in some layers. Multi-scale downsampling from the input image size H×W to H / 32×W / 32 is achieved through convolutional layers with a stride of 2. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. Skip connections are introduced in block 2 to fuse initial features to enhance feature propagation. Finally, feature maps of multiple scales are output. The specific steps of step S2 are as follows: Step S2.1: Construct a parallel structure that integrates the convolutional branch and the star operation module, containing multiple layers of 3×3 ordinary convolutions and 1×1 convolutions for fusing and adjusting channels of multi-scale features; the star operation module processes multi-scale features separately, including H / 8×W / 8, H / 16×W / 16, and H / 32×W / 32; firstly, local features are extracted by 3×3 depthwise separable convolution, and then feature maps x1 and x2 are generated by two parallel 1×1 convolutions, where x1 is activated by ReLU6 and multiplied element-wise with x2. The resulting image is adjusted in dimension by 1×1 convolution and further processed by 3×3 depthwise separable convolution, and finally added to the input features to form a residual connection, thereby realizing the aggregation of global context features; Step S2.2: After adding and fusing the output features of the convolution branch and the star operation module, a dense descriptor with a dimension of 64 is generated by 1×1 convolution compression to represent local image features. At the same time, a nonlinear transformation is performed on the fused features and a sigmoid activation is applied to generate a heatmap, where each pixel value represents the confidence that a feature point exists at the location. In addition, by performing 8×8 neighborhood window feature extraction and regression processing on the input image, a tensor for predicting the offset of key point positions is generated. The three together constitute the output result, providing basic data for subsequent key point detection and feature matching.

2. The real-time feature point extraction and matching method based on star operation and GateMLP as described in claim 1, characterized in that, The specific steps of step S3 are as follows: Step S3.1: Perform non-maximum suppression on the heatmaps corresponding to the left and right views. Use a 5×5 neighborhood window to perform non-maximum suppression. Find local maximum points through maximum pooling. Only retain points in the heatmap that are equal to the local maximum and whose values ​​are higher than the detection threshold as candidate key points. Step S3.2: Combine the descriptors corresponding to the candidate key points with the heatmap scores to calculate the confidence level, filter out the candidate points with confidence levels higher than the set threshold, map their coordinates back to the original image size according to the scaling ratio recorded in the preprocessing stage, record the mapped coordinates as the final key points, and retain their corresponding descriptor vectors for subsequent matching.

3. The real-time feature point extraction and matching method based on star operation and GateMLP as described in claim 1, characterized in that, The specific steps of step S4 are as follows: Step S4.1: Calculate the cosine similarity between the key point descriptor of the left view and all key point descriptors of the right view, and select the right view key point with the highest similarity for each left view key point as a candidate match; Step S4.2: Perform nearest neighbor verification: If the best match of key point A in the left view is key point B in the right view, and the best match of key point B in the right view is also key point A in the left view, then retain the matching pair as the coarse matching result, and remove matching pairs with similarity below the set threshold.

4. The real-time feature point extraction and matching method based on star operation and GateMLP as described in claim 3, characterized in that, The specific steps of step S5 are as follows: Step S5.1: Concatenate the descriptors of the coarse matching pairs along the channel dimension into a 128-dimensional feature vector, and input it into the GateMLP network consisting of 3 layers of gated linear units. Each layer of gated linear units includes linear transformation, batch normalization and gate mechanism. Step S5.2: The GateMLP network outputs the sub-pixel level offset probability distribution within an 8×8 window. After normalization by the Softmax function, the offset with the highest probability is taken as the correction value. Step S5.3: Overlay the correction value onto the coarse matching key point coordinates to obtain accurate sub-pixel level coordinates, remove matching pairs whose distance exceeds the preset threshold after correction, and output the final matching point pairs.

Images