Image feature point matching method and system based on LightNet-SuperPoint

Abstract

The invention provides an image feature point matching method and system based on LightNet-SuperPoint, and relates to the technical field of image processing. According to the method, a lightweight coding network fused with an ECA attention mechanism is adopted to extract feature points and descriptors of continuous image frames in parallel, and high-quality target feature points are obtained through bilinear interpolation, confidence ranking and dynamic screening; a preliminary matching pair is formed by combining bidirectional matching and Euclidean distance constraint, a basic matrix is estimated through RANSAC, geometrically inconsistent point pairs are eliminated, finally a high-quality feature corresponding relation is output, matching precision and real-time performance are improved, and the method is suitable for tasks such as visual SLAM.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image feature point matching method and system based on LightNet-SuperPoint. Background Technology

[0002] Traditional algorithms based on handcrafted features, such as SIFT and ORB, can extract a large number of features, but they have drawbacks such as uneven spatial distribution, high redundancy, and poor real-time performance. In particular, in road scenarios, feature points are prone to dense clustering or empty areas, which seriously affect the subsequent matching results.

[0003] SuperPoint parallelizes feature point detection and descriptor generation, improving matching robustness. However, its encoding layer uses the VGG backbone, resulting in a large number of parameters and slow inference speed, which is not conducive to resource-constrained or real-time applications. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide an image feature point matching method and system based on LightNet-SuperPoint. By constructing a lightweight coding network and introducing multi-level filtering and geometric consistency constraints, the accuracy and real-time performance of image feature point matching are significantly improved, making it suitable for efficient visual localization and mapping tasks in resource-constrained scenarios.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] An image feature point matching method based on LightNet-SuperPoint includes:

[0007] Obtain at least one consecutive image frame to be matched;

[0008] The LightNet-SuperPoint network, which incorporates the ECA attention mechanism, performs feature point detection and descriptor decoding in parallel on the consecutive image frames to obtain an initial feature point set containing pixel-level confidence.

[0009] Bilinear interpolation subpixel refinement is performed on the initial feature point set, and the target feature points with the highest confidence are selected according to the normalized confidence scores.

[0010] Bidirectional matching is performed based on the descriptors of the target feature points. The intersection of the forward matching set and the reverse matching set is retained as a preliminary symmetric matching point pair set. The preliminary symmetric matching point pair set is then filtered according to a preset descriptor Euclidean distance threshold. Matching point pairs that do not meet the threshold are deleted to obtain a high-quality symmetric matching point pair set.

[0011] The RANSAC algorithm is applied to the set of high-quality symmetric matching point pairs to estimate the fundamental matrix, and mismatched point pairs that do not satisfy geometric consistency are removed to obtain the final high-quality feature correspondence.

[0012] The high-quality feature correspondence is output for use by visual SLAM or other pose estimation modules.

[0013] Preferably, the method for constructing the coding layer in the LightNet-SuperPoint network includes:

[0014] For an input image of size H×W×3, perform a 3×3 channel-wise convolution on each channel to generate a depth feature map of the same size; where H is the height of the image and W is the width of the image.

[0015] A 1×1 pointwise convolution is applied to the depth feature map, and combined with batch normalization and HS / RE nonlinear activation function, the channel information is re-integrated and the number of output channels is adjusted;

[0016] At least one Bneck module is sequentially input into the adjusted output channel. Each Bneck module uses a 3×3 or 5×5 depth-separable convolution and inserts an ECA attention mechanism based on one-dimensional convolution at the exit to learn cross-channel dependencies, improve feature representation ability and inference speed.

[0017] According to the preset number of output channels and step size parameters, the Bneck modules are stacked in series to form a complete LightNet encoding layer, and the final feature map is passed to the SuperPoint decoding branch for subsequent feature point and descriptor generation.

[0018] Preferably, bilinear interpolation sub-pixel refinement is performed on the initial feature point set, and the target feature points with the highest confidence scores are selected according to the normalized confidence scores, including:

[0019] A local pixel window of a preset size is cropped centered on each initial feature point of the initial feature point set. Within the local pixel window, bilinear interpolation is used to calculate the pixel-level confidence level, and the interpolated coordinates are updated to sub-pixel precision.

[0020] The pixel-level confidence scores are normalized and sorted from high to low to obtain a sequence of candidate feature points.

[0021] The feature points with the highest confidence are selected sequentially from the candidate feature point sequence until a preset target number is reached, forming a set of target feature points.

[0022] Preferably, bidirectional matching is performed based on the descriptors of the target feature points, retaining the intersection of the two matching results, and for each pair of feature points in the intersection, if the Euclidean distance is greater than twice the minimum matching distance of the consecutive image frames to be matched, the feature point pair is discarded, further including:

[0023] If the number of target feature points is less than the preset target number, the preset target number is reduced to supplement the target feature points.

[0024] Preferably, bidirectional matching is performed based on the descriptors of the target feature points. The intersection of the forward matching set and the reverse matching set is retained as a preliminary symmetric matching point pair set. The preliminary symmetric matching point pair set is then filtered according to a preset descriptor Euclidean distance threshold, and matching point pairs that do not meet the threshold are deleted to obtain a high-quality symmetric matching point pair set, including:

[0025] Using each target feature point descriptor in the first frame as a query, calculate the Euclidean distance between it and all target feature point descriptors in the second frame, and select the correspondence with the smallest distance to form a positive candidate matching set;

[0026] Using each target feature point descriptor in the second frame as a query, calculate the Euclidean distance between it and all target feature point descriptors in the first frame, and select the correspondence with the smallest distance to form a reverse candidate matching set;

[0027] In the positive candidate matching set and the negative candidate matching set, symmetrical matching items that appear in both matching results are retained to obtain a preliminary symmetrical matching set;

[0028] The minimum Euclidean distance L between all matching point pairs in the preliminary symmetric matching set is calculated. min And delete items with a distance greater than 2×L. min The matching point pairs are used to obtain the high-quality symmetric matching point pair set.

[0029] Preferably, the RANSAC algorithm is applied to the set of high-quality symmetric matching point pairs to estimate the fundamental matrix, and mismatched point pairs that do not satisfy geometric consistency are removed to obtain the final high-quality feature correspondence, including:

[0030] a) From the set of high-quality symmetric matching points, randomly select the minimum sample set that meets the requirements of the eight-point algorithm, and use the minimum sample set to estimate the initial basis matrix;

[0031] b) For each pair of matching points in the set of high-quality symmetric matching points, calculate the epipolar error according to the epipolar geometric constraint formula. If the epipolar error does not exceed a preset threshold, mark it as an inner point; otherwise, mark it as an outer point.

[0032] c) Set the confidence level and outlier rate, calculate the minimum number of iterations according to the formula derived from Bernoulli's experiment, and repeat sub-steps a) to b) until the minimum number of iterations is reached or the optimal model with the maximum number of interior points is obtained.

[0033] d) Using the pair of matching points with the most interior points obtained in substep c), re-estimate the final fundamental matrix using least squares.

[0034] e) Use the final base matrix to perform geometric consistency verification on all matching point pairs, delete matching point pairs that do not meet the epipolar error threshold, and output the final high-quality feature correspondence.

[0035] An image feature point matching system based on LightNet-SuperPoint includes:

[0036] The image frame acquisition unit is used to acquire at least one consecutive image frame to be matched;

[0037] The feature point extraction and description unit is used to perform feature point detection and descriptor decoding in parallel on the consecutive image frames using a LightNet-SuperPoint network that incorporates the ECA attention mechanism, so as to obtain an initial feature point set containing pixel-level confidence.

[0038] The feature point refinement and filtering unit is used to perform bilinear interpolation subpixel refinement on the initial feature point set, and sort them according to normalized confidence, and select the target feature point with the highest confidence.

[0039] The feature point bidirectional matching unit is used to perform bidirectional matching based on the descriptor of the target feature point, retain the intersection of the forward matching set and the reverse matching set as the initial symmetric matching point pair set, and filter the initial symmetric matching point pair set according to the preset descriptor Euclidean distance threshold, delete matching point pairs that do not meet the threshold, and obtain a high-quality symmetric matching point pair set.

[0040] The geometric consistency verification unit is used to apply the RANSAC algorithm to the set of high-quality symmetric matching point pairs to estimate the fundamental matrix and remove mismatched point pairs that do not meet the geometric consistency requirement, so as to obtain the final high-quality feature correspondence.

[0041] The feature relationship output unit is used to output the high-quality feature correspondence for use by visual SLAM or other pose estimation modules.

[0042] The present invention discloses the following technical effects:

[0043] (1) In the feature extraction network, the present invention uses LightNet, a lightweight encoder built on depthwise separable convolution and ECA attention mechanism, to replace the VGG encoding layer in the traditional SuperPoint, which reduces redundant convolution operations and the number of fully connected layers, and significantly improves the overall model inference speed. It is particularly suitable for edge devices or mobile platforms that are sensitive to computing resources.

[0044] (2) This invention combines bilinear interpolation with normalized confidence ranking and dynamic threshold adjustment mechanism to enhance the uniformity of target feature points in the image space, avoid concentrated clusters or empty areas, and improve the effectiveness and robustness of subsequent matching.

[0045] (3) This invention introduces a bidirectional matching intersection filtering mechanism and combines it with an Euclidean distance adaptive filtering strategy to eliminate low similarity matching pairs while ensuring matching accuracy, thereby effectively reducing false matching.

[0046] (4) This invention estimates the fundamental matrix using the RANSAC algorithm and removes point pairs that do not meet the epipolar geometric constraints, ensuring that the output feature correspondence has high accuracy and geometric consistency, providing more stable input data for subsequent pose estimation or 3D reconstruction.

[0047] (5) This invention significantly improves operating efficiency while maintaining matching accuracy. It can be flexibly deployed in SLAM front-end, image registration, AR / VR visual tracking and other application scenarios, and has good engineering practicality and promotion value. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart of the method provided in an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of the deep learning pose estimation process based on LightNet-SuperPoint provided in an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of the SuperPoint network structure provided in an embodiment of the present invention;

[0052] Figure 4 These are illustrations showing the feature point extraction results of different algorithms provided in embodiments of the present invention.

[0053] Figure 5 This is a schematic diagram of the bidirectional matching results provided in an embodiment of the present invention;

[0054] Figure 6 This is a schematic diagram of the RANSAC algorithm principle provided in an embodiment of the present invention;

[0055] Figure 7 This is a schematic diagram of the LightNet-SuperPoint feature extraction network structure provided in an embodiment of the present invention;

[0056] Figure 8 Channel-separable convolutional graphs provided in embodiments of the present invention;

[0057] Figure 9 A diagram illustrating the ECA attention mechanism provided in an embodiment of the present invention;

[0058] Figure 10 This is a flowchart of the improved model feature point extraction operation provided in an embodiment of the present invention;

[0059] Figure 11 This is a schematic diagram of illumination changes in the HPatches dataset provided in an embodiment of the present invention;

[0060] Figure 12 This is a schematic diagram illustrating the perspective changes of the HPatches dataset provided in an embodiment of the present invention.

[0061] Figure 13 This is a schematic diagram comparing the pose estimation of the KITTI-Odometry-02 sequence fragment algorithm provided in an embodiment of the present invention. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] The purpose of this invention is to provide an image feature point matching method and system based on LightNet-SuperPoint. By constructing a lightweight coding network and introducing multi-level filtering and geometric consistency constraints, the accuracy and real-time performance of image feature point matching are significantly improved, making it suitable for efficient visual localization and mapping tasks in resource-constrained scenarios.

[0064] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0065] Figure 1 The method flowchart provided in the embodiments of the present invention is as follows: Figure 1 As shown, this invention provides an image feature point matching method based on LightNet-SuperPoint, comprising:

[0066] Step 100: Obtain at least one consecutive image frame to be matched;

[0067] Step 200: Using the LightNet-SuperPoint network with ECA attention mechanism, feature point detection and descriptor decoding are performed in parallel on consecutive image frames to obtain an initial set of feature points containing pixel-level confidence.

[0068] Step 300: Perform bilinear interpolation subpixel refinement on the initial feature point set, and sort the target feature points according to their normalized confidence scores, selecting the target feature points with the highest confidence scores;

[0069] Step 400: Perform bidirectional matching based on the descriptors of the target feature points, retain the intersection of the forward matching set and the reverse matching set as the initial symmetric matching point pair set, and filter the initial symmetric matching point pair set according to the preset descriptor Euclidean distance threshold, delete the matching point pairs that do not meet the threshold, and obtain a high-quality symmetric matching point pair set.

[0070] Step 500: Apply the RANSAC algorithm to the set of high-quality symmetric matching point pairs to estimate the fundamental matrix, and remove mismatched point pairs that do not meet geometric consistency to obtain the final high-quality feature correspondence.

[0071] Step 600: Output high-quality feature correspondences for use by visual SLAM or other pose estimation modules.

[0072] This embodiment addresses the issues of excessive redundant information and long matching time in the original SuperPoint feature matching. It designs a feature point matching algorithm based on the LightNet-SuperPoint network, employing an unsupervised feature extraction network (LightNet) with a lightweight attention mechanism (ECA) as the encoding layer to achieve feature point detection and descriptor decoding of the image. Image features are downsampled to reduce computational complexity; high-quality feature points are extracted using LightNet's feature detection and classification layer, and bilinear interpolation is incorporated to calculate confidence levels, thus selecting a high-quality feature point set. Finally, a regional mismatch elimination strategy is employed to preserve locally optimal feature point pairs and establish stable and reliable feature association information. Its innovation lies in combining a feature extraction layer with a lightweight attention mechanism with an unsupervised feature extraction method, providing a new solution for front-end pose estimation in visual SLAM systems in dynamic scenes. The algorithm research roadmap is as follows: Figure 2 As shown.

[0073] Traditional methods typically require first extracting key feature points and then calculating local descriptors based on the selected key point locations. This process is step-by-step and computationally inefficient. In image dimensionality reduction, the SuperPoint algorithm simultaneously implements a parallel processing mechanism for feature point detection and local descriptor generation. This model achieves computational synergy between dimensionality compression and feature extraction by constructing a parameter-shared dual decoder. Therefore, this embodiment selects the SuperPoint algorithm as the baseline model for feature point extraction and matching.

[0074] SuperPoint belongs to the category of fully convolutional networks, and its structure mainly consists of three modules: an encoder, a feature point decoding module, and a descriptor decoding module. The encoder performs a non-linear dimensionality reduction operation from high to low dimensions on the input image. The two decoding modules are used for spatial localization of key image feature points and encoding and generating local descriptors. The specific flow of its model feature point extraction algorithm is as follows: Figure 3 As shown.

[0075] (1) For an image with an input size of W×H, the input data is generated by the feature encoding module of SuperPoint convolution, and then the nonlinear mapping transformation from the original features to the low-dimensional space is completed.

[0076] (2) The feature point decoding module receives the encoded feature image and maps it into a three-dimensional tensor with a specific dimension parameter of w×h×65. This tensor data is processed sequentially through a Softmax activation layer and a Reshape reshaping layer, and finally outputs a probability response matrix with a dimension of w×h×1. The three-dimensional matrix represents the confidence distribution characteristics of each pixel position in the original image as a key feature point.

[0077] (3) The descriptor decoder converts the single-frame image feature map output by the encoder into a tensor of dimension w×h×65. Then, it performs operations through a spatial interpolation layer and a normalization layer to finally generate a W×H×65 tensor. This three-dimensional matrix represents the local feature description information of each pixel position in the original image, where each vector element corresponds to the feature encoding of a specific spatial position. The rotation and translation of the camera position are solved using a matrix transformation algorithm between PnP point pairs, ultimately obtaining the pose estimate between the two frames of data.

[0078] To compare the feature extraction performance of traffic area images under different algorithms, this embodiment uses the SIFT feature point matching algorithm, the traditional ORB feature point matching algorithm, and the SuperPoint feature point extraction algorithm to extract image features on a self-collected dataset (a sequence of road scene data in a certain urban area, containing 425 image frames acquired from the front view of a vehicle-mounted monocular camera). A comparative experiment on the feature point extraction performance is designed.

[0079] Experimental results are as follows Figure 4 As shown in the diagram, comparative analysis reveals that, under the same experimental configuration, the SIFT algorithm detects the most densely packed feature points, but its spatial distribution exhibits significant clustering. More than two-thirds of the feature points are concentrated in the left half of a single frame image, with some areas even showing excessive overlap of feature points. This is related to its principle of using the difference of Gaussian pyramid for extreme value detection. The ORB algorithm, on the other hand, shows a non-uniform distribution of feature points in the vertical direction, with most corner points clustered in the upper half of the image. The smooth texture in the lower half makes it difficult for the corner detector to obtain pixels that meet the grayscale threshold conditions. The SuperPoint algorithm, based on a deep learning framework, demonstrates excellent distribution uniformity. This uniform distribution characteristic stems from the global receptive field mechanism used in the network, which can effectively capture image features of different scales and directions, providing more robust initial conditions for subsequent improvements to the feature matching algorithm.

[0080] The above three algorithms were used to perform unidirectional feature matching on images at different thresholds, and the test results are shown in Table 1. According to the matching results, it can be seen that as the threshold increases, the algorithm tends to match more feature points, but this operation sacrifices matching accuracy. SIFT achieves a significantly higher number of matches at high thresholds (1424 pairs at a threshold of 0.8), but its high-dimensional computation may sacrifice real-time performance. ORB performs poorly at low thresholds but increases rapidly at high thresholds (557% increase from 0.4 to 0.8), making it suitable for real-time scenarios, but subsequent mismatch filtering is required. SuperPoint has a moderate number of matches (466 pairs at a threshold of 0.8). Thanks to the robustness of deep learning to different viewpoints and lighting conditions, it is more stable in dynamic and complex environments. Therefore, this embodiment selects the SuperPoint algorithm as the baseline algorithm before improvement for subsequent experiments.

[0081] Table 1. Test results of one-way matching with different algorithms and thresholds.

[0082]

[0083] While the number of matches can reflect the breadth of matching to some extent, it cannot comprehensively measure the accuracy of the algorithm. In real-world scenarios, the original SuperPoint algorithm also suffers from uneven feature point distribution or a high false match rate. To address these issues, this embodiment improves the SuperPoint algorithm by proposing a match-then-optimize strategy. A feature point set is obtained through initial matching between image frames. Algorithms such as bidirectional matching are then used to optimize and filter the feature points, and the processed result is ultimately used as a high-quality matching point set. This improvement effectively alleviates the problem of uneven feature point distribution, significantly reduces the false match rate, and improves the overall performance of the matching algorithm.

[0084] (1) Bidirectional matching algorithm

[0085] The existing SuperPoint algorithm still has some matching errors during the matching process, which affects the overall modeling effect. To address this issue, this embodiment introduces a bidirectional matching mechanism, which involves two feature point matching processes in opposite directions. The left frame is matched against the right frame, and then the right frame is matched against the left frame. The common part of the two matching results is taken as the final set of matched feature points.

[0086] In the experiment, empirical constraints were introduced for the matching point pairs. The minimum and maximum distance values ​​between matching point pairs of all image keypoints were calculated. Matching point pairs whose distance exceeded twice the range of Lmin (the minimum distance in the set of matching point pairs in the current image pair) were removed from the set. This improvement effectively reduced the false matching rate and improved the overall accuracy of the matching results, providing a solution for subsequent experimental improvements and comparisons. The bidirectional matching experimental results are as follows: Figure 5 As shown.

[0087] (2) Selection of Region of Interest

[0088] Considering that in real-world applications, roads are typically straight or have minimal curvature, and the data consists of continuous frame sequences with relatively stable viewpoint changes, this embodiment proposes a feature point pairing method within a shared field of view. This method defines the upper third of the next frame as the shared field of view, and then performs feature point matching with the immediately following frame. The aim is to reduce computational complexity by decreasing the number of redundant feature points, thereby improving program efficiency and feature point matching accuracy. However, in actual experiments, especially in dynamic scenarios such as lane changes or cornering, this strategy did not meet expectations, and therefore this approach was ultimately abandoned.

[0089] (3) RANSAC algorithm

[0090] During feature matching, the high frequency of mismatched point pairs necessitates frequent sampling to eliminate outliers, significantly increasing the computational load. These erroneous matches can cause deviations in the fundamental matrix estimation, thereby affecting the pose estimation accuracy. To address this issue, this embodiment uses empirical constraints from the RANSAC algorithm to obtain a point set with a high matching rate and a small number of point pairs. The fundamental matrix is ​​then estimated, and outlier point pairs that do not meet the matrix constraints are filtered out, effectively reducing the number of mismatched point pairs.

[0091] like Figure 6 As shown, the RANSAC algorithm randomly selects a set of points from point pairs to estimate the model. A subset is pre-defined as the inlier set, and a preliminary association model is constructed based on this subset. Other points consistent with this model are considered inliers and categorized accordingly. When the number of inliers in the model reaches a certain threshold, the currently estimated model is considered reasonable.

[0092] In the estimation process of the fundamental matrix, the RANSAC algorithm iteratively selects the minimum sample set from the point set after empirical constraint filtering to find the minimum sample set that maximizes the number of local points. This optimal minimum sample set is then used to calculate the fundamental matrix F. Based on the obtained fundamental matrix F, mismatches in the image are further removed. The feature point set after empirical constraint preprocessing is used as input, and the filtering operation of mismatches is performed according to formula (1). Matching point pairs that fail to meet the conditions specified in formula (1) are removed from the entire matching point set.

[0093]

[0094] Where F represents a 3×3 matrix, [u' i v' i 1] and [u i v i 1] T These represent the homogeneous coordinates of the points selected through empirical constraints.

[0095] The above method optimizes the original SuperPoint algorithm, but experimental results show that the optimized algorithm still extracts redundant feature points on the image, increasing the computational load and reducing the efficiency of the detection layer decoding process. This embodiment designs and introduces a lightweight network, LightNet, to replace the original VGG encoding layer. This network features low computational burden, few parameters, and fast inference speed. Simultaneously, an ECA attention mechanism is introduced to compensate for the accuracy decrease caused by the small model parameters. Initial feature point screening is achieved through feature detection and classification layer operations, followed by updating the set of superior feature points according to their confidence within the feature region, while incorrect matching point pairs are removed.

[0096] The traditional SuperPoint network architecture's encoding layer consists of convolutional layers, pooling-based spatial downsampling, and nonlinear activation functions. Due to the large number of encoding layers, a large number of samples are required during training, significantly increasing computational cost and impacting the overall efficiency of the SLAM system. To address this issue, this embodiment designs a lightweight network—LightNet—with low computational complexity, small parameter size, and fast inference capabilities to replace the VGG encoding layer in the original architecture, improving the algorithm's accuracy and real-time performance. A classification layer is set up for image feature detection, using Softmax normalization to pre-screen extracted points, and bilinear interpolation is used to calculate confidence scores and rank them. The lightweight LightNet-SuperPoint feature extraction network structure is detailed below. Figure 7 .

[0097] When processing image features, the channel-separable convolutional structure first performs convolution operations on the image and outputs multiple channels. This design reduces the number of parameters and lowers computational costs. Through deep fusion of channel information and attention mechanisms, the coding layer network can fully preserve high-dimensional feature information while reducing computation time.

[0098] Table 2 Light-Net coding layer network structure

[0099]

[0100]

[0101] The network structure of the LightNet coding layer is shown in Table 2. The first column of the table shows the block structure to be processed in each feature layer traversal. It can be observed that the feature extraction process goes through many bneck modules. The second column shows the number of channels input to the bneck feature layer by the inverse residual structure. The third column, marked NL, represents the type of activation function, where HS and RE represent the activation function types. The fourth column, s, represents the stride used in each block structure. The fifth column, SE, indicates whether an attention mechanism is introduced in this network layer and its type.

[0102] The improved LightNet coding layer extracts features by combining channel-separable convolutions. The channel-separable convolution structure is as follows: Figure 8As shown, it mainly consists of two parts: channel-wise convolution and pointwise convolution. For an input image of size H×W×3, the same number of convolution kernels are used for each channel to perform convolution operations, generating feature map information for each channel. Pointwise convolution re-integrates these channels, effectively promoting information interaction between channels and having the ability to change the number of output feature channels, thereby significantly improving the efficiency of feature encoding and enabling more rapid achievement of feature encoding tasks.

[0103] This embodiment replaces part of the SE in the improved coding layer with the more efficient ECA. ECA is an improved version of SE, and its core principle is to learn the interaction between channels through one-dimensional convolution, using an adaptive convolution kernel size to capture cross-channel information, avoiding the use of fully connected layers, thereby significantly reducing computational complexity. It shows significant advantages in computer vision tasks such as image classification, object detection, and semantic segmentation, featuring small parameter count, high computational efficiency, and the ability to adapt to different input sizes.

[0104] In image processing, ECA first performs global average pooling on the input feature map, then generates channel weights through adaptive one-dimensional convolution, and finally weights the feature map using a sigmoid activation function to achieve efficient computation of channel attention. Its core function is to solve the loss problem caused by different channel weights during convolutional pooling. The processing flow includes global average pooling of the input feature map, downsampling, channel-dimensional feature transformation using a k-dimensional convolution kernel, calculating the attention weights for each channel using an activation function, multiplying the attention weights by corresponding elements of the original input feature map element by element, and finally generating an attention-weighted output feature map. An adaptive method is used in the processing flow to determine the threshold k (i.e., kernel size) and dynamically adjust the weight values ​​to generate image features in a more robust manner.

[0105] ECA attention mechanism structure as follows Figure 9 As shown, after inputting the H*W*C feature map into the ECA attention mechanism, a global pooling operation is first performed to obtain a 1×1×C feature map. Then, it passes through two fully connected layers to obtain a weighted 1×1×C feature map. By performing a channel-wise weighted multiplication of the original feature map and the processed feature map, a feature map H*W*C with different attention weights can be obtained.

[0106] In the process of feature matching using traditional algorithms, a large number of feature mismatches often occur due to experimental environment issues. To address this problem, this embodiment first uses bilinear interpolation to perform interpolation calculations in the local regions around the feature points detected by the LightNet algorithm, calculating the feature point positions with pixel-level precision. Then, the feature points are sorted and filtered according to the normalized scores. If the number of feature points in the set is insufficient, the detection threshold is lowered to supplement the number of feature points to ensure that the requirements are met.

[0107] The steps for removing false matches are as follows: Within a specified range, perform sampling consistency verification on the feature matching results, adjust the overlap threshold between adjacent regions to clearly define the matching associations within the region, and utilize orientation and scale feature information to perform in-depth sampling consistency verification on the identified matching pairs within the region, thereby ensuring that false matches can be effectively identified and eliminated. For detailed operational procedures, please refer to... Figure 10 As shown.

[0108] The theoretical derivation of the minimum number of iterations β required for model estimation is as follows: Given a confidence level parameter p (set to 0.9), let the outlier ratio in the dataset be ε, and let n sample points be independently drawn each time to form a subset. According to the Bernoulli trial principle, the probability of obtaining the entire interior set in a single sampling can be expressed as (1-ε). n The theoretical derivation of the minimum number of iterations β by constructing the cumulative distribution function is shown in equation (2):

[0109] 1-(1-(1-ε) n ) β =p (2)

[0110] After rearranging the formula, the number of iterations β can be obtained, as shown in equation (3):

[0111]

[0112] This embodiment's experiment is divided into two parts: The first part is a feature matching performance comparison experiment, in which feature extraction is compared on the publicly available dataset HPatches and a self-collected dataset, and quantitative and qualitative analyses are performed. The second part is a pose estimation experiment, in which the algorithm of this embodiment replaces the traditional ORB algorithm as a visual odometry algorithm for SLAM, and a trajectory comparison experiment is performed on a segment of the KITTI-06 sequence, a publicly available traffic scene dataset.

[0113] The experimental environment for this embodiment is as follows: CPU is 12th Gen Intel(R) Core(TM) i7-12700KF with a main frequency of 3.60GHz, memory is 32G, and operating system is Ubuntu 20.04.

[0114] This embodiment uses the HPatches dataset to evaluate the feature extraction and matching performance of the LightNet-SuperPoint algorithm. The HPatches dataset contains a large number of varied image sets with different lighting, size, and angle variations, making it highly suitable for studying image matching, recognition, and segmentation problems. The image dataset integrates image resources from natural scenes, image generation, and real-world photography, covering various core physical environment variables: the impact of lighting changes on object representation and geometric shape changes caused by shooting angle. The dataset contains 57 sets of scene image sequences with varying lighting intensity and 59 sets of image sequences with varying viewpoints, with each sequence containing 6 images. Some image samples are shown below. Figure 11 and Figure 12 As shown.

[0115] This example introduces another typical subsequence structure for a dataset, where each subsequence contains six images under the same type of variation. Geometric consistency verification is achieved through homography transformation matrix, and a baseline truth value for the spatial mapping relationship between image pairs is constructed.

[0116] The evaluation metrics for this embodiment are as follows:

[0117] (1) Feature point repetition rate

[0118] Feature point repetition rate, as a key indicator for evaluating the performance of feature detection algorithms, measures the probability that the same feature point can be accurately identified under different imaging environments or changing viewpoints. It can reflect the stability and robustness of the feature point detection algorithm to a certain extent, and is of great significance for ensuring the reliability and practicality of the algorithm.

[0119] Suppose that the point sets of graphs A and B after the feature detection algorithm are m respectively. points and n points m in Figure A points Each feature point set is multiplied by the transformation matrix H to generate a new coordinate point set m. points H, new set of coordinate points m points After H is filtered by the range constraint of image B, the set of remaining points is m' points , requires m points H cannot be less than 0 and must be greater than the width and height of image B. The constraints are as shown in equation (4), where m(x) i ,y i )∈m pointsH .

[0120] 0 <x i <B.width,0<y i <B.height (4)

[0121] Take n in image Bpoints The feature point set is multiplied by the transformation matrix H. -1 Generate a new set of coordinate points n points H,n points After filtering by the range constraint of image A, the set of remaining points is n' points Require n points H cannot be less than 0 and is greater than the width and height of image A, as constrained by equation (5), where n(x i ,y i )∈n pointsH .

[0122] 0 <x i <A.width,0<y i <A.height (5)

[0123] The specific calculation process of the feature point repetition rate is shown in equation (6).

[0124]

[0125] Where the molecule ∑dist(m) points ,n points ) m' points The point sets are respectively at n points Calculate the Euclidean distance. If the Euclidean distance threshold is less than a set value ε, then it is a duplicate point. The denominator is min(m). points ,n' points ) represents the set of coordinate points m pointsH and n pointsH After being constrained by formulas (4) and (5), m' is obtained. points and n' points The minimum value is used as the denominator to obtain the repetition rate of feature point pair matching.

[0126] (2) Accuracy

[0127] Accuracy is the core evaluation index for feature point matching. The percentage of correct cases among the predicted positive cases reflects the model's ability to correctly identify correct cases and demonstrates the effectiveness of feature point matching. The specific calculation process is shown in Equation (7).

[0128]

[0129] In the formula, TP refers to a sample that is actually a positive class and is correctly classified as a positive class (correct prediction), and FP refers to a sample that is actually a negative class but is incorrectly classified as a positive class (misclassification).

[0130] The algorithm performance analysis of this embodiment is as follows:

[0131] (1) Feature point extraction repetition rate comparison experiment

[0132] To verify the performance of the algorithm proposed in this embodiment, experiments were conducted on the HPatches dataset. Twenty sets of data sequences containing illumination variations, @30° variations, and @60° variations were used for validation. The feature point detection threshold was set to 300 candidate points, and the Non-Maximum Suppression (NMS) was set to 4. The experimental results are shown in Table 3.

[0133] As can be seen from the data in the table, in the test sequence of illumination change and viewpoint change, compared with the traditional feature extraction algorithm ORB, the average feature point extraction repetition rate of the other three algorithms all exceeded 50%. The algorithm proposed in this embodiment showed the best performance. In the test sequence, its repetition rate index was significantly improved compared with other algorithms, and the feature repetition rate was improved by 2.9% and 3.5% respectively compared with the original SuperPoint.

[0134] Table 3. Comparison of Feature Point Extraction Repeatability Rate Algorithm Results

[0135]

[0136] (2) Algorithm comparison experiment under 60° scene change

[0137] Table 4 presents the specific matching results of various algorithms, showing the comparison of evaluation metrics such as matching accuracy, runtime, number of coarse matching pairs, and number of correct matching pairs under the @60° rotation transformation dataset after the feature point matching algorithm integrates various optimized feature matching strategies.

[0138] As shown in Table 4, although the Superpoint+bidirectional matching algorithm and the Superpoint+RANSAC algorithm improve the matching accuracy at @60°, their runtime is nearly twice that of the algorithm in this embodiment. Furthermore, the Superpoint+bidirectional matching algorithm is inferior to the algorithm in this embodiment in terms of coarse matching point pairs, and the Superpoint+RANSAC algorithm is inferior to the algorithm in this embodiment in terms of the number of correct matching logs. The algorithm in this embodiment achieves an accuracy of 72.3% at @60° variations after optimization, which is slightly lower than the previous two improved algorithms. However, this algorithm generates a sufficient number of correct matching point pairs for SLAM pose estimation while significantly reducing runtime, thus exhibiting the best overall performance.

[0139] Table 4. Results of the improved feature point matching algorithm

[0140]

[0141] It should be noted that the bolded indicators in Tables 3 and 4 and the tables below represent algorithms with the best performance, while the underlined indicators represent algorithms with the second-best performance.

[0142] (3) Algorithm comparison experiment under scene changes

[0143] To further evaluate the performance of the algorithm in this embodiment in terms of feature matching accuracy, RANSAC geometric consistency verification was added to the original algorithm, and the number of correctly matched points was counted. This embodiment extracted the i_brooklyn dataset containing one illumination transformation sequence and the datasets v_artisans@30°, v_abstract@30°, v_artisans@60°, and v_abstract@60° from the HPatches dataset for testing and verification. The results are shown in Table 5.

[0144] As can be seen, the algorithm in this embodiment improves upon the ORB+RANSAC algorithm by approximately 11.2%, 16.0%, and 18.5% in data experiments involving illumination changes, 30° rotation angles, and 60° rotation angles, respectively. Compared to the SuperPoint+RANSAC algorithm, the improvements are 6.8%, 9.5%, and 4.9%, respectively. In sequences with significant changes in angle and viewpoint, especially in the abstract scene of v_abstract, the feature matching accuracy of the traditional ORB algorithm drops significantly, exhibiting obvious instability. The algorithm in this embodiment demonstrates a more significant advantage in feature matching accuracy compared to the ORB and SuperPoint methods, indicating that the matching accuracy of the proposed algorithm is higher than other feature matching algorithms.

[0145] Table 5 Comparison of feature matching accuracy under different perspectives and illumination changes.

[0146]

[0147] (4) Demonstration of algorithm visualization

[0148] To verify the performance of the proposed LightNet-SuperPoint algorithm compared to traditional feature matching methods, a comparative experiment was designed. ORB, the original SuperPoint, and the algorithm of this embodiment were selected and qualitative tests were conducted on the i_brooklyn, v_artisans, and v_abstract datasets. The experimental results show that in scenes involving lighting changes, the ORB algorithm performs poorly in terms of descriptor invariance, resulting in a large number of mismatches. After filtering out mismatches, the number of correctly matched point pairs is relatively small. The SuperPoint algorithm effectively extracts rich feature points and their descriptors, achieving a large number of matching pairs. However, compared to the algorithm proposed in this embodiment, the number of low-quality matches is still slightly higher. In the two scenes involving viewpoint changes, the ORB algorithm performs poorly, exhibiting the worst matching results. Although the SuperPoint algorithm can extract some matching results, it cannot effectively eliminate mismatches. The algorithm of this embodiment, however, utilizes the feature point set to remove mismatched point pairs, significantly increasing the number of correctly matched point pairs. Compared to the SuperPoint and ORB algorithms, the proposed algorithm shows a significantly increased number of successfully matched lines.

[0149] To further verify the feature extraction and matching performance of the LightNet-SuperPoint algorithm in real traffic scenarios, this embodiment visualizes the algorithm on a road self-collected dataset of a certain street. The experimental results show that the algorithm in this embodiment has the highest number of feature matches and the highest accuracy, which is significantly better than the other two algorithms and shows better robustness in real traffic scenarios.

[0150] The LightNet-SuperPoint feature extraction network was used as the visual odometry front-end for SLAM. Data matching was performed, and matrix calculations were used to determine vehicle localization changes. A segment of the KITTI-Odometry-02 sequence from the KITTI traffic scene dataset was selected for vehicle pose estimation experiments. The experimental results are as follows: Figure 13 As shown.

[0151] The left image shows the overall trajectory of the vehicle, while the right image shows partial trajectories of the vehicle along the x, y, and z axes in the world coordinate system. In the images, the blue lines represent the pose estimation results using the algorithm of this embodiment, and the orange lines represent the actual trajectories of the vehicle in the dataset.

[0152] It can be seen that the vehicle driving path map has initially achieved the experimental effect of pose estimation, indicating that the matrix result obtained by matching and solving the high-quality feature point information extracted by the algorithm in this embodiment can be used as the pose calculation method of the SLAM front-end odometry.

[0153] Traditional image matching algorithms, such as ORB, suffer from numerous mismatched point pairs, and the original SuperPoint network generates excessive redundant information during image feature extraction. Based on this research, this embodiment proposes a feature point matching algorithm based on LightNet-SuperPoint. The LightNet lightweight network replaces the VGG encoding layer in the original network, and a lightweight ECA attention mechanism is integrated to construct the encoding layer, enabling efficient feature detection and descriptor decoding in dynamic scenes. Downsampling and bilinear interpolation confidence filtering optimize computational complexity and feature quality. An error point pair elimination strategy is proposed, enhancing the robustness of associated features through local optimal feature pair selection, providing a high-precision, low-redundancy front-end pose estimation scheme for visual SLAM systems. Experimental results show that the improved algorithm significantly reduces computational complexity while maintaining high visual localization accuracy, exhibiting better real-time performance and robustness.

[0154] Corresponding to the above method, this embodiment also provides an image feature point matching system based on LightNet-SuperPoint, including:

[0155] The image frame acquisition unit is used to acquire at least one consecutive image frame to be matched;

[0156] The feature point extraction and description unit is used to perform feature point detection and descriptor decoding in parallel on the consecutive image frames using a LightNet-SuperPoint network that incorporates the ECA attention mechanism, so as to obtain an initial feature point set containing pixel-level confidence.

[0157] The feature point refinement and filtering unit is used to perform bilinear interpolation subpixel refinement on the initial feature point set, and sort them according to normalized confidence, and select the target feature point with the highest confidence.

[0158] The feature point bidirectional matching unit is used to perform bidirectional matching based on the descriptor of the target feature point, retain the intersection of the forward matching set and the reverse matching set as the initial symmetric matching point pair set, and filter the initial symmetric matching point pair set according to the preset descriptor Euclidean distance threshold, delete matching point pairs that do not meet the threshold, and obtain a high-quality symmetric matching point pair set.

[0159] The geometric consistency verification unit is used to apply the RANSAC algorithm to the set of high-quality symmetric matching point pairs to estimate the fundamental matrix and remove mismatched point pairs that do not meet the geometric consistency requirement, so as to obtain the final high-quality feature correspondence.

[0160] The feature relationship output unit is used to output the high-quality feature correspondence for use by visual SLAM or other pose estimation modules.

[0161] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0162] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An image feature point matching method based on LightNet-SuperPoint, characterized in that, include: Obtain at least one consecutive image frame to be matched; The LightNet-SuperPoint network, which incorporates the ECA attention mechanism, performs feature point detection and descriptor decoding in parallel on the consecutive image frames to obtain an initial feature point set containing pixel-level confidence. Bilinear interpolation subpixel refinement is performed on the initial feature point set, and the target feature points with the highest confidence are selected according to the normalized confidence scores. Bidirectional matching is performed based on the descriptors of the target feature points. The intersection of the forward matching set and the reverse matching set is retained as a preliminary symmetric matching point pair set. The preliminary symmetric matching point pair set is then filtered according to a preset descriptor Euclidean distance threshold. Matching point pairs that do not meet the threshold are deleted to obtain a high-quality symmetric matching point pair set. The RANSAC algorithm is applied to the set of high-quality symmetric matching point pairs to estimate the fundamental matrix, and mismatched point pairs that do not satisfy geometric consistency are removed to obtain the final high-quality feature correspondence. The high-quality feature correspondence is output for use by visual SLAM or other pose estimation modules.

2. The image feature point matching method based on LightNet-SuperPoint according to claim 1, characterized in that, The method for constructing the coding layer in the LightNet-SuperPoint network includes: For an input image of size H×W×3, perform a 3×3 channel-wise convolution on each channel to generate a depth feature map of the same size; where H is the height of the image and W is the width of the image. A 1×1 pointwise convolution is applied to the depth feature map, and combined with batch normalization and HS / RE nonlinear activation function, the channel information is re-integrated and the number of output channels is adjusted; At least one Bneck module is sequentially input into the adjusted output channel. Each Bneck module uses a 3×3 or 5×5 depth-separable convolution and inserts an ECA attention mechanism based on one-dimensional convolution at the exit to learn cross-channel dependencies, improve feature representation ability and inference speed. According to the preset number of output channels and step size parameters, the Bneck modules are stacked in series to form a complete LightNet encoding layer, and the final feature map is passed to the SuperPoint decoding branch for subsequent feature point and descriptor generation.

3. The image feature point matching method based on LightNet-SuperPoint according to claim 1, characterized in that, Bilinear interpolation subpixel refinement is performed on the initial feature point set, and the target feature points with the highest confidence scores are selected according to their normalized confidence scores, including: A local pixel window of a preset size is cropped centered on each initial feature point of the initial feature point set. Within the local pixel window, bilinear interpolation is used to calculate the pixel-level confidence level, and the interpolated coordinates are updated to sub-pixel precision. The pixel-level confidence scores are normalized and sorted from high to low to obtain a sequence of candidate feature points. The feature points with the highest confidence are selected sequentially from the candidate feature point sequence until a preset target number is reached, forming a set of target feature points.

4. The image feature point matching method based on LightNet-SuperPoint according to claim 3, characterized in that, Bidirectional matching is performed based on the descriptors of the target feature points, retaining the intersection of the two matching results. For each pair of feature points in the intersection, if the Euclidean distance is greater than twice the minimum matching distance of the consecutive image frames to be matched, the feature point pair is discarded. This also includes: If the number of target feature points is less than the preset target number, the preset target number is reduced to supplement the target feature points.

5. The image feature point matching method based on LightNet-SuperPoint according to claim 1, characterized in that, Bidirectional matching is performed based on the descriptors of the target feature points. The intersection of the forward matching set and the reverse matching set is retained as a preliminary symmetric matching point pair set. The preliminary symmetric matching point pair set is then filtered according to a preset descriptor Euclidean distance threshold, and matching point pairs that do not meet the threshold are deleted to obtain a high-quality symmetric matching point pair set, including: Using each target feature point descriptor in the first frame as a query, calculate the Euclidean distance between it and all target feature point descriptors in the second frame, and select the correspondence with the smallest distance to form a positive candidate matching set; Using each target feature point descriptor in the second frame as a query, calculate the Euclidean distance between it and all target feature point descriptors in the first frame, and select the correspondence with the smallest distance to form a reverse candidate matching set; In the positive candidate matching set and the negative candidate matching set, symmetrical matching items that appear in both matching results are retained to obtain a preliminary symmetrical matching set; The minimum Euclidean distance L between all matching point pairs in the preliminary symmetric matching set is calculated. min And delete items with a distance greater than 2×L. min The matching point pairs are used to obtain the high-quality symmetric matching point pair set.

6. The image feature point matching method based on LightNet-SuperPoint according to claim 1, characterized in that, The RANSAC algorithm is applied to the set of high-quality symmetric matching point pairs to estimate the fundamental matrix, and mismatched point pairs that do not satisfy geometric consistency are removed to obtain the final high-quality feature correspondence, including: a) From the set of high-quality symmetric matching points, randomly select the minimum sample set that meets the requirements of the eight-point algorithm, and use the minimum sample set to estimate the initial basis matrix; b) For each pair of matching points in the set of high-quality symmetric matching points, calculate the epipolar error according to the epipolar geometric constraint formula. If the epipolar error does not exceed a preset threshold, mark it as an inner point; otherwise, mark it as an outer point. c) Set the confidence level and outlier rate, calculate the minimum number of iterations according to the formula derived from Bernoulli's experiment, and repeat sub-steps a) to b) until the minimum number of iterations is reached or the optimal model with the maximum number of interior points is obtained. d) Using the pair of matching points with the most interior points obtained in substep c), re-estimate the final fundamental matrix using least squares. e) Use the final base matrix to perform geometric consistency verification on all matching point pairs, delete matching point pairs that do not meet the epipolar error threshold, and output the final high-quality feature correspondence.

7. An image feature point matching system based on LightNet-SuperPoint, characterized in that, include: The image frame acquisition unit is used to acquire at least one consecutive image frame to be matched; The feature point extraction and description unit is used to perform feature point detection and descriptor decoding in parallel on the consecutive image frames using a LightNet-SuperPoint network that incorporates the ECA attention mechanism, so as to obtain an initial feature point set containing pixel-level confidence. The feature point refinement and filtering unit is used to perform bilinear interpolation subpixel refinement on the initial feature point set, and sort them according to normalized confidence, and select the target feature point with the highest confidence. The feature point bidirectional matching unit is used to perform bidirectional matching based on the descriptor of the target feature point, retain the intersection of the forward matching set and the reverse matching set as the initial symmetric matching point pair set, and filter the initial symmetric matching point pair set according to the preset descriptor Euclidean distance threshold, delete matching point pairs that do not meet the threshold, and obtain a high-quality symmetric matching point pair set. The geometric consistency verification unit is used to apply the RANSAC algorithm to the set of high-quality symmetric matching point pairs to estimate the fundamental matrix and remove mismatched point pairs that do not meet the geometric consistency requirement, so as to obtain the final high-quality feature correspondence. The feature relationship output unit is used to output the high-quality feature correspondence for use by visual SLAM or other pose estimation modules.

Images