A resolution-adaptive end-to-end sub-pixel feature matching method

Abstract

This invention provides a resolution-adaptive end-to-end sub-pixel-level feature matching method, comprising the following steps: Step 1, performing joint detection and description of feature sub-functions based on single-head coupled output; Step 2, performing sub-pixel-level keypoint localization based on an improved differentiable keypoint detection algorithm; Step 3, implementing an adaptive mechanism for different resolutions or texture densities; Step 4, implementing end-to-end gradient closure based on backpropagation. This invention provides an effective solution to the core problems of traditional feature matching methods, such as insufficient sub-pixel-level localization accuracy, fragmented detection and description modules, and poor scene adaptability. This method significantly reduces keypoint localization error from the pixel level to the sub-pixel level, significantly improving the geometric accuracy of 3D reconstruction. The single-head coupled output structure achieves collaborative optimization of detection and description tasks, making feature representation more discriminative and reducing parameter redundancy through gradient sharing.

Description

Technical Field

[0001] This invention belongs to the field of 3D reconstruction technology, and particularly relates to a resolution-adaptive end-to-end subpixel-level feature matching method. Background Technology

[0002] 3D reconstruction is one of the core tasks in the field of computer vision, aiming to recover the 3D structure of a scene from multi-view images. It is widely used in fields such as autonomous driving, AR / VR, and digital twins. As the cornerstone of 3D reconstruction, feature matching plays a crucial role in data association, reconstruction accuracy, and efficiency. In recent years, feature matching algorithms have become a key path to improve the accuracy and efficiency of 3D reconstruction, with new demands for high precision, robustness, and versatility.

[0003] Current mainstream feature matching methods include those based on hand-designed approaches and those based on deep learning. Hand-designed feature matching methods perform well in terms of computational efficiency and interpretability, but they are less robust to complex scenes with varying viewpoints, lighting conditions, and missing textures. Furthermore, their reliance on extensive manual parameter tuning and heuristic rules limits their performance ceiling. Examples include SIFT, SURF, and ORB, which extract local feature points from images, compute their descriptors, and perform matching using nearest neighbor search or ratio testing. In contrast, deep learning-based methods significantly improve the robustness and accuracy of matching by automatically learning feature representations and matching strategies, leading to mainstream frameworks such as graph matching frameworks, self-supervised or unsupervised learning frameworks, detection-description joint learning frameworks, and matching optimization frameworks. However, deep learning methods typically require large amounts of labeled data and significant computational resources, and their high model complexity limits their application performance.

[0004] The Superglue feature matching method (reference: Sarlin, PE, DeTone, D., Malisiewicz, T., & Rabinovich A. Superglue: Learning feature matching with graph neural networks [C]. Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition, 2020: 4938-4947.) is a deep learning feature matching method based on graph neural networks (GNNs) and attention mechanisms to optimize matching. Its core process includes inputting feature points and descriptors from two images, modeling global contextual relationships using self-attention and cross-attention, generating optimal matching pairs by normalizing the matching score matrix using the Sinkhorn algorithm, and eliminating mismatches using geometric verification such as RANSAC. This mechanism makes it perform well in occluded, viewpoint-changing, and low-texture scenes, and it is widely used in visual localization, structure-of-motion (SfM), and augmented reality (AR). However, this method still has three main limitations: 1) It inherits the computational complexity problem of earlier Transformers, with the computational cost increasing quadratically with the number of keypoints; 2) It is highly dependent on labeled data, making training difficult; 3) The quality of the detector directly affects the matching performance, with matching accuracy only down to the pixel level, making it unsuitable for matching tasks in complex scenes, and its generalization stability needs improvement. Therefore, how to optimize the Superglue feature matching method while maintaining its advantages and solving the above problems has become an important challenge in current research.

[0005] LightGlue (reference: Lindenberger, P.-E. Sarlin, M. Pollefeys LightGlue: Local Feature Matching at Light Speed[C], 2023 IEEE / CVF International Conference on Computer Vision (ICCV), 2023:17581-17592.) and LoFTR (reference: Sun, Z. Shen, Y. Wang, H. Bao and X. Zhou, LoFTR: Detector-Free Local Feature Matching with Transformers[C], 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021: 8918-8927.) explore further in reducing computational complexity and improving the stability of feature matching in low-texture scenes. LightGlue significantly improves computational efficiency by introducing an adaptive depth and width mechanism while maintaining high matching accuracy. Its adaptive characteristics enable it to complete inference faster when processing simple image pairs. However, LightGlue is highly sensitive to detector quality. If the front-end feature point detection is unstable, it may lead to over-pruning and loss of effective matches. Furthermore, threshold tuning is difficult in extreme dynamic scenes. LoFTR, on the other hand, adopts a detector-free design. Through the global receptive field and positional encoding of the Transformer, combined with multi-scale feature pyramids and channel attention mechanisms, it can generate dense matches in low-texture regions, solving the problem of traditional detectors failing in repetitive patterns or weak-texture regions. However, although LoFTR outperforms SuperGlue in matching in low-texture regions, its computational complexity is high, requiring a fine-grained iterative correction strategy, resulting in lower real-time performance. Summary of the Invention

[0006] Purpose of the Invention: The problems of insufficient sub-pixel level localization accuracy, fragmented detection and description modules, and poor resolution adaptability in feature matching are the core bottlenecks restricting the improvement of feature matching performance, and are also the key technical challenges that this invention aims to overcome. These problems not only reduce the accuracy and robustness of feature matching, but also limit the practical application effect of the algorithm in complex scenarios. Specifically, this invention provides a resolution-adaptive end-to-end sub-pixel level feature matching method, including the following steps:

[0007] Step 1: Perform joint detection and description of features based on single-head coupling output;

[0008] Step 2: Perform sub-pixel level keypoint localization based on an improved differentiable keypoint detection algorithm;

[0009] Step 3: Implement an adaptive mechanism for different resolutions or texture densities;

[0010] Step 4: Perform end-to-end gradient closure based on backpropagation.

[0011] In step 1, a single-head coupled output structure is adopted, and key point confidence and descriptors are generated synchronously through a single set of convolution parameters. A gradient coupling mechanism is designed to enable the detection and description tasks to share common features and optimize collaboratively.

[0012] Step 1 includes:

[0013] First, a feature map F and a key point score map S are generated using a lightweight convolutional neural network (CNN). The CNN includes four depthwise separable convolutional blocks with a maximum receptive field of 204×204.

[0014] Let s represent the confidence that a key point exists at each pixel location after activation by the Sigmoid function. It is considered as a probability value during calculation, and s is used as the key point score.

[0015] Input feature map Through shared convolution kernel The process has D+1 channels, and the output channel tensor Z has D+1 coordinates. The mapping of the channel tensor Z at point Z is defined as:

[0016] (1),

[0017] in For convolution, Let be the weight matrix of the convolution kernel, and b be the bias term, the value of which is determined during training through backpropagation. b forms a shared convolution kernel The set of trainable parameters.

[0018] In step 1, record the coordinates. At this point, the normalized unit descriptor vector for:

[0019] (2),

[0020] in, This represents the first D channels of the output tensor Z at coordinates (x, y), and the length of the vector is calculated using the L2 norm (Euclidean norm). The dimension is D×1.

[0021] In step 1, record the coordinates. Key points scoring for:

[0022] (3),

[0023] in This represents the (D+1)th channel of the feature tensor Z; For the Sigmoid function;

[0024] The gradients of the descriptor loss and the detection loss are coupled to the shared convolutional kernel via a chain rule. The same set of trainable parameters is used to adjust The value of b is determined by... Gradient coupling formula:

[0025] (4),

[0026] in Represents partial differentials; This represents the gradient of the descriptor loss with respect to the descriptor d; This represents the gradient of the detection loss with respect to the score s; This indicates the descriptor normalization operation's response to the original descriptor. The gradient; This indicates the Sigmoid activation response to the original score. The gradient; The output Z of the convolutional layer represents the parameter set. The gradient;

[0027] Key Points With descriptor vector Derived from shared convolutional kernels It accepts gradients from both the descriptor loss and the keypoint loss.

[0028] Step 2 includes:

[0029] Step 2-1, Preliminary detection of integer pixel key points: A non-maximum suppression (NMS) strategy is employed, with a global threshold τ. The image coordinate system has its origin at the top left corner (0,0), where u represents the x-coordinate of a pixel in the image, and v represents the y-coordinate. The NMS strategy searches the entire score image for pixel coordinates (u,v) that simultaneously satisfy the following two conditions:

[0030] Score condition: The keypoint score s(i,j) at coordinate (i,j) is greater than the preset global threshold τ;

[0031] Local maximum condition: Within a local neighborhood N centered at pixel coordinates (u,v) with a radius of 2r+1, the keypoint score s(i,j) at coordinate (i,j) is the maximum among all keypoint scores of neighboring pixels.

[0032] The conditional formula is:

[0033] ,

[0034] (5),

[0035] Where s(u,v) represents the keypoint score at pixel coordinates (u,v), N represents a local neighborhood of size (2r+1)×(2r+1) centered at pixel coordinates (u,v); the global threshold τ is a configurable hyperparameter, the specific value of which is determined by cross-validation based on the training dataset, and the value range is [0.3, 0.7]; max represents the maximum value function;

[0036] The non-maximum suppression strategy only retains the pixel with the highest score, and in backpropagation, the selected pixel coordinates (u,v) and the corresponding keypoint score s(u,v) are treated as constants.

[0037] Step 2-2, the DKD (Different Keypoint Detection) strategy enables sub-pixel-level fine-tuning:

[0038] For the keypoint score map S obtained by the lightweight convolutional neural network CNN, with Centered on the keypoint score map S, a local window P of size N×N is cropped. A probability-weighted average soft-argmax function is applied to all scores within the local window P to transform the keypoint score map S into a spatial probability distribution. In coordinates Spatial probability distribution value at The calculation is as follows:

[0039] (6),

[0040] in The maximum score within the window. Let exp be the temperature coefficient, and let exp represent the natural exponential function. This indicates the coordinates within the local window P. Scoring based on key points; express The score of key points at other coordinates (p, q) within the same local window P;

[0041] Next, using coordinates Spatial probability distribution value at As weights, the expected coordinates of key points within the window are calculated. and subpixel position p:

[0042] (7),

[0043] Finally, the coordinates of the integrated center and subpixel position , the desired coordinates Mapping back to the entire image coordinate system, the final sub-pixel level keypoint coordinates for:

[0044] (8),

[0045] in It is the coordinate of the center of the local window in its own coordinate system.

[0046] Step 3 includes:

[0047] Define discrete candidate set , representing a predefined optional value for window size N; for the k-th candidate window currently being computed. During training, the logarithmic weights can be learned by updating them through gradient descent. And use Gumbel-Softmax to make the discrete sampling distribution continuous (Gumbel-Softmax is a discrete variable modeling technique used in deep learning that introduces Gumbel noise to achieve differentiable approximate sampling while preserving gradient information), and adjust the temperature parameter. Controlled sampling, selecting weights from the Gumbel distribution (extreme value distribution) and random noise in sampling as follows:

[0048] (9),

[0049] in This indicates that when the summation term in the denominator is traversed through all candidate windows, the j-th candidate window... The corresponding learnable log weights, This indicates that when the summation term in the denominator is traversed through all candidate windows, the j-th candidate window... Random noise sampled using a Gumbel distribution;

[0050] For each Calculate pixel-level soft coordinates Finally, the pixel-level soft coordinates are weighted and fused to update the sub-pixel position p, achieving differentiable selection:

[0051] (10)

[0052] (11),

[0053] Let the temperature coefficient of the probability-weighted average soft-argmax be... ,in, Used to control the concentration of the coordinate probability distribution. These are learnable parameters, initialized to 0; the softplus function is a flexible activation function, used... Will Mapped to , to make temperature parameters Always positive.

[0054] The neural network of the end-to-end subpixel feature matching model with full resolution adaptation learns through training. Automatic adjustment To adapt to the texture complexity of different image regions.

[0055] Step 4 includes:

[0056] Through end-to-end joint detection and description, each position i corresponds to a candidate coordinate. Mark the sub-pixel keypoint coordinates at position i as The corresponding key point score Redefining it using the Sigmoid function, denoted as... , indicating the position The probability of the existence of a true keypoint is used to calculate the Bernoulli loss, and the formula is as follows:

[0057] (12)

[0058] in, This indicates the sub-pixel keypoint coordinates. The integer pixel coordinates obtained after rounding.

[0059] Design a supervision signal based on the Bernoulli distribution. For position i, if position i falls within the geometric threshold neighborhood of a ground truth key point... Inside, make the label Resubmission is allowed; otherwise, the label will be removed. 0;

[0060] Confidence loss is calculated using Bernoulli negative log-likelihood. During the model training phase, continuous confidence is enforced. Fitting discrete labels :

[0061] (13)

[0062] To achieve collaborative optimization of detection and description tasks, a joint loss function is constructed. :

[0063] (14)

[0064] in, Represents reprojection loss The weighting coefficients, Indicates the peak dispersion loss The weighting coefficients, Describing the loss The weighting coefficients, Indicates reliability loss The weighting coefficients, Indicates confidence loss Weighting coefficients;

[0065] The combined formulas (4), (9) to (14) form a differentiable computational graph, and the gradient is completely backflowed through soft-argmax and Bernoulli, realizing the integrated learning of detection and description:

[0066] (15)

[0067] in, This represents the gradient propagated back by the descriptor d. Let L represent the gradient of the loss L with respect to the descriptor d; Describer d pairs The gradient; This represents the gradient of the keypoint score s backpropagated. This represents the gradient of the loss L with respect to the keypoint score s. Indicates the key point score s. gradient, This represents the summation of gradients over all spatial coordinates.

[0068] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0069] The present invention also provides a storage medium storing a computer program or instructions that, when the computer program or instructions are run on a computer, execute the steps of the method described.

[0070] Beneficial Effects: This invention provides an effective solution to the core problems of traditional feature matching methods, such as insufficient sub-pixel level localization accuracy, fragmented detection and description modules, and poor scene adaptability. By combining a differentiable sub-pixel localization mechanism with dynamic window selection, the model significantly reduces keypoint localization errors from the pixel level to the sub-pixel level, significantly improving the geometric accuracy of 3D reconstruction. The single-head coupled output structure achieves collaborative optimization of detection and description tasks, making feature representation more discriminative and reducing parameter redundancy through gradient sharing. The adaptive mechanism, through learnable window size and temperature parameters, enables the model to intelligently adapt to input scenes with different resolutions and texture densities, maintaining stable matching performance even in low-texture regions. The final output joint features combine sub-pixel level localization accuracy, high discriminative descriptors, and calibrated reliable confidence metrics, providing a higher quality data foundation for tasks such as 3D reconstruction and visual localization. Attached Figure Description

[0071] Figure 1 This is the overall flowchart of the present invention.

[0072] Figure 2 This is a rendering of the reconstructed model.

[0073] Figure 3 A detailed comparison image of the reconstructed model. Detailed Implementation

[0074] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0075] like Figure 1 As shown, this embodiment of the invention provides a resolution-adaptive end-to-end sub-pixel level feature matching method, including the following steps:

[0076] Step 1: Joint detection and description of features based on single-head coupling output;

[0077] In traditional keypoint detection and descriptor extraction tasks, a dual-branch structure (separation of detection and description) is often used, leading to inconsistencies in the features of the two tasks. The detection branch focuses on positional saliency, while the description branch focuses on matching discriminability. These two optimization objectives inherently conflict, and during backpropagation, the gradient is split across different parameter sets, weakening the model's convergence efficiency and feature alignment capability. To overcome these shortcomings, this invention proposes a single-head coupled output structure. Its core idea is to synchronously generate keypoint confidence and descriptors using a single set of convolutional parameters, and to design a gradient coupling mechanism that allows the detection and description tasks to share common features and perform collaborative optimization.

[0078] First, a lightweight convolutional neural network (CNN) is used to generate downsampled feature maps F and keypoint score maps S. This network consists of four depthwise separable convolutional blocks with a maximum receptive field of 204×204. The score s of each pixel coordinate is activated by a sigmoid function; the activated score s represents the confidence level that a keypoint exists at the coordinate.

[0079] Input feature map Through shared convolution kernel Process and output a (D+1) channel tensor. The mapping is defined as:

[0080] (1),

[0081] in For convolution;

[0082] Let the first D channel be:

[0083] (2),

[0084] Let the D+1th channel be:

[0085] (3),

[0086] The gradients of the descriptor loss and the detection loss are coupled to the same parameter set via a chain rule. :

[0087] (4),

[0088] Detection score With descriptor Originating from the same convolution kernel It simultaneously accepts gradients from both the descriptor loss and the keypoint loss. In this way, a single graph can simultaneously produce a descriptor graph for matching and a score graph for detection. Moreover, it avoids the problems of gradient splitting and potential direction conflicts in "two-headed" models, and it keeps the detection and description semantically consistent, enabling the model to share features and gradients, reducing overall parameters, and making end-to-end training more stable.

[0089] Step 2: Sub-pixel level keypoint localization based on an improved differentiable keypoint detection algorithm;

[0090] The core goal of Differentiable Keypoint Detection (DKD) algorithms is to achieve keypoint localization with sub-pixel accuracy. This process typically consists of two stages: initial screening of integer-pixel positions and fine-tuning of local sub-pixel positions.

[0091] (1) Preliminary detection of key points at whole pixels;

[0092] To select significant and unique candidate points, the algorithm employs Non-Maximum Suppression (NMS). The idea behind NMS is to select the best candidate box most likely representing a real target from a large number of overlapping candidate boxes, and remove all other redundant and highly overlapping candidate boxes. The selection steps and conditions are as follows: A global threshold τ (valued at 0.2) is set. NMS searches the entire score image for pixel coordinates (u, v) that simultaneously satisfy the following two conditions:

[0093] Score condition: The keypoint score s(i,j) at coordinate (i,j) is greater than the preset global threshold τ;

[0094] Local maximum condition: Within a local neighborhood N centered at pixel coordinates (u,v) with a radius of 2r+1, the keypoint score s(i,j) at coordinate (i,j) is the maximum among all keypoint scores of neighboring pixels.

[0095] The conditional formula is:

[0096] ,

[0097] (5),

[0098] Where s(u,v) represents the keypoint score at pixel coordinates (u,v), N represents a local neighborhood of size (2r+1)×(2r+1) centered at pixel coordinates (u,v); the global threshold τ is a configurable hyperparameter, the specific value of which is determined by cross-validation based on the training dataset, and the value range is [0.3, 0.7]; max represents the maximum value function;

[0099] The non-maximum suppression strategy only retains the pixel with the highest score, and in backpropagation, the selected pixel coordinates (u,v) and the corresponding keypoint score s(u,v) are treated as constants.

[0100] (2) Local soft-argmax achieves sub-pixel level fine-tuning;

[0101] Integer pixel localization inherently suffers from quantization error. To achieve higher geometric accuracy, DKD targets the score map S with... Centered on the original score map S, a local window P of size N×N is cropped. The soft-argmax function is applied to all scores within window P to transform them into a spatial probability distribution:

[0102] (6),

[0103] Next, using the generated probability distribution as weights, the expected coordinates of the key points within the window are calculated. :

[0104] (7),

[0105] Finally, the calculated local desired coordinates Map back to the entire image coordinate system. Because It is the offset relative to the local window center (u,v) (with the window center as the origin), and the final sub-pixel level keypoint coordinates. for:

[0106] (8),

[0107] in It is the coordinate of the center of the local window in its own coordinate system.

[0108] The improved sub-pixel keypoint localization based on a differentiable keypoint detection algorithm combines discrete NMS filtering with continuous soft-argmax regression. NMS efficiently provides reliable integer-pixel candidate points, while the learnable local soft-argmax algorithm finely adjusts the results within the small neighborhoods of these candidate points using the score distribution information, calculating sub-pixel coordinates with decimals. This significantly improves the geometric accuracy of keypoint localization while maintaining the end-to-end trainability of the model.

[0109] Step 3: Adaptive mechanism for different resolutions or texture densities;

[0110] In traditional subpixel localization, fixed window size N and the Softmax function are difficult to adapt to scenes with different resolutions or texture densities. Therefore, this invention proposes an adaptive mechanism for different resolutions or texture densities to achieve adaptive parameter optimization.

[0111] (1) Learnable window N and temperature parameter ;

[0112] Traditional methods using fixed windows often fail to adapt to changes in target scale. For example, densely textured scenes require smaller windows to avoid interference from neighboring points, while sparsely textured scenes require larger windows to capture information during high-altitude photography. Manually switching windows is labor-intensive, and multi-scale pyramid methods are computationally too demanding. The learnable window proposed in this invention achieves soft switching between different sizes through Gumbel-Softmax and can backpropagate gradients, solving two key problems in traditional methods: poor generalization and insufficient scene adaptability caused by fixed hyperparameters.

[0113] Define discrete candidate set , representing a predefined optional value for window size N, for each Maintaining learnable logarithmic weights And use Gumbel-Softmax to make the discrete sampling distribution continuous, and use Control sampling:

[0114] (9),

[0115] For each Calculate soft coordinates (soft-argmax of formula (2)), finally weighted fusion of coordinates to achieve differentiable selection:

[0116] (10)

[0117] (11);

[0118] Temperature of Soft-argmax , Control the concentration of the probability distribution of the coordinates. When under high temperature ( When the value is much greater than 1, the coordinate probability distribution is flat, the coordinates are smooth but easily deviate from the true peak value; when at low temperature ( When the value is much less than 1, the coordinate probability distribution is sharp, sensitive to noise, and the gradient is unstable. The learnable mechanism transforms parameter selection into an optimization problem, allowing the network to autonomously find the optimal balance point. Learnable and in the Epoch thawing. Softplus guarantees. Furthermore, the gradient is smooth. Faced with feature point sets with different resolutions or texture densities, the model can automatically select the most suitable window size N and soft-argmax temperature. And remain differentiable.

[0119] Step 4, end-to-end gradient loop closure based on backpropagation;

[0120] In keypoint detection tasks in computer vision, traditional methods often couple confidence scores with feature descriptors during training, making it difficult for confidence scores to accurately reflect the actual reliability of keypoints. To address this issue, this invention proposes an innovative confidence score normalization mechanism. Its core objective is to model whether a location is a ground truth keypoint as an explicit Bernoulli event, enabling the network to distinguish between "reliable keypoints" and "false keypoints" (such as texture noise or edge responses). Confidence scores only affect the weighted detection loss and threshold selection during inference, no longer directly influencing the descriptor learning process.

[0121] The network outputs a score at each candidate position. Redefine the mathematical meaning of this fraction. Bernoulli's success probability, interpreted as "the location contains a truth-critical point":

[0122] (12)

[0123] By constraining its value range to a probability space, the numerical normalization of the confidence score is achieved, which can be directly used for probability comparison and threshold decision-making, significantly improving the accuracy of the confidence score. To train this probabilistic representation, this invention designs a supervision signal based on a Bernoulli distribution. For position i, if it falls within the geometric threshold neighborhood of a ground truth key point... Inside, make the label Resubmission is allowed; otherwise, the label will be removed. 0. Calculated using Bernoulli negative log-likelihood:

[0124] (13)

[0125] The score output by the network The Bernoulli success probability is used as the basis for training, which means "this point is a true critical point". This allows us to learn a confidence index with calibrable probabilistic meaning, numerical normalization, and clear decision boundaries. This index is used to determine the detection loss weights and the selection criteria during inference, without affecting the descriptor itself.

[0126] To achieve collaborative optimization of detection and description tasks, a joint loss function is constructed:

[0127] (14)

[0128] Default weight .

[0129] The combined sub-modules of formulas (4), (9) to (14) constitute a differentiable computational graph. The gradient is completely backflowed through soft-argmax and Bernoulli, realizing the integrated learning of detection and description:

[0130] (15).

[0131] In one specific embodiment of the present invention, the method includes the following steps:

[0132] Step 1: Data preprocessing based on multi-scale feature extraction and enhancement;

[0133] Select an open-source dataset and input image pairs and First, Z-Score normalization and downsampling methods are used for preprocessing to generate multi-scale image pyramids. Where S represents the scale index. The purpose of this step is to provide a foundation for subsequent multi-scale feature extraction. Compared with traditional single-scale input methods, this invention is more robust to scale changes.

[0134] Step 2: Joint detection and description of features based on single-head coupling output;

[0135] First, a lightweight convolutional neural network (CNN) is used to generate a downsampled feature map F and a keypoint score map S. This network consists of four depthwise separable convolutional blocks with a maximum receptive field of 204×204. The feature map is input into a single convolutional head, and features are extracted synchronously through shared weights. The first D channels of the convolutional head output are L2 normalized to generate a descriptor map of size H×W×D. The D-dimensional vector at each spatial location in the map represents the features of the local region. The (D+1)th channel of the convolutional head output is activated by a sigmoid function to generate a score map S of size H×W×1. Each pixel value in the map represents the Bernoulli probability of a keypoint existing at that location, used for localization filtering and loss weighting. This design avoids the parameter redundancy of traditional two-branch structures (independent detection and descriptive mapping) and forces both tasks to share common underlying features.

[0136] Step 3: Subpixel-level keypoint detection, extraction, and adaptive optimization;

[0137] Non-maximum suppression (NMS) is performed on the fractional image S, setting the geometric neighborhood radius r and a global threshold. Only pixels that simultaneously satisfy the confidence condition (s>τ) and the local maximum condition (maximum value within a (2r+1)×(2r+1) window) are retained, resulting in integer pixel coordinates as the candidate keypoint set {(u,v)}. An adaptive window of size N×N (N∈{3,5,7}) is cropped on the fractional image S centered at each candidate point (u,v), with the window size dynamically selected using learnable weights. Within the window, a soft-argmax algorithm is applied to transform the scores into a spatial probability distribution, using a high-temperature parameter t. det By controlling the distribution smoothness and calculating the probability-weighted expected coordinates, sub-pixel level offsets are obtained. These offsets are then superimposed onto integer pixel coordinates to output the final sub-pixel coordinates p. For the descriptor d, bilinear interpolation is performed on the descriptor graph D based on the sub-pixel coordinates p. The extracted D-dimensional vector is then L2 normalized to ensure consistent feature scale. For the confidence score s... i The original score value at position p is directly obtained from the score map S as a key point reliability index, and its probabilistic calibration is supervised by Bernoulli loss.

[0138] Step 4, Feature matching;

[0139] Feature matching employs the Superglue matching framework. It receives keypoints p and their descriptors d from two images and encodes them into high-dimensional vectors, incorporating the keypoint locations to enhance spatial awareness. Next, a graph structure with two types of edges is constructed: self-edges between keypoints within the same image, used to aggregate local contextual information; and cross-edges between keypoints in different images, used to establish cross-image associations. By alternately executing self-attention and cross-attention mechanisms, the graph neural network progressively updates the feature representation of each keypoint and autonomously optimizes it through backpropagation loss function, finally outputting filtered high-precision matching pairs.

[0140] like Figure 3 As shown, sub-pixel-level keypoints are densely and precisely distributed in textured areas (such as edges and corners), while keypoints are significantly reduced but reasonably distributed in weakly textured areas (such as walls). The mean positioning error of sub-pixel coordinate p is ≤0.3 pixels (e.g., ...). Figure 2 As shown, compared to existing technologies (such as SuperPoint+superglue, which only studies pixel-level localization errors), this invention achieves more accurate edge alignment and outperforms existing technologies on open-source datasets. For example, it achieves a precision of 88.51%, a recall of 83.39%, and an F1 score of 85.82% on the open-source dataset Hpatches. Compared to existing technologies (such as SuperPoint+superglue), this invention achieves high keypoint accuracy, stronger robustness to scale changes, a large and uniform number of correct matching pairs, and very few mismatches through multi-scale feature enhancement, joint optimization of detection and description, adaptive sub-pixel localization, and graph matching fusion. It is particularly robust in repetitive textures and dynamically occluded regions.

[0141] This invention provides a resolution-adaptive end-to-end sub-pixel level feature matching method. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A resolution-adaptive end-to-end sub-pixel level feature matching method, characterized in that, Includes the following steps: Step 1: Perform joint detection and description of features based on single-head coupling output; Step 2 involves performing sub-pixel-level keypoint localization based on an improved differentiable keypoint detection algorithm; Step 2 includes: Step 2-1, Preliminary detection of integer pixel key points: A non-maximum suppression strategy is adopted, and a global threshold τ is set. The image coordinate system takes the upper left corner as the origin (0,0), u represents the horizontal coordinate of the pixel in the image, and v represents the vertical coordinate of the pixel in the image. The non-maximum suppression strategy searches for pixel coordinates (u,v) on the entire score image that simultaneously meet the following two conditions: Score condition: The key point score s(i,j) of coordinate (i,j) is greater than the preset global threshold τ; Local maximum condition: Within a local neighborhood N centered at pixel coordinates (u,v) with a radius of 2r+1, the keypoint score s(i,j) at coordinate (i,j) is the maximum among all keypoint scores of neighboring pixels. The conditional formula is: , （5）， Where s(u,v) represents the keypoint score at pixel coordinates (u,v), N represents a local neighborhood of size (2r+1)×(2r+1) centered at pixel coordinates (u,v), and max represents the maximum value function; The non-maximum suppression strategy only retains the pixel with the highest score, and in backpropagation, the selected pixel coordinates (u,v) and the corresponding keypoint score s(u,v) are treated as constants. Step 2-2, the DKD (Different Keypoint Detection) strategy enables sub-pixel-level fine-tuning: For the keypoint score map S obtained by the lightweight convolutional neural network CNN, with Centered on the keypoint score map S, a local window P of size N×N is cropped. A probability-weighted average soft-argmax function is applied to all scores within the local window P to transform the keypoint score map S into a spatial probability distribution. In coordinates Spatial probability distribution value at The calculation is as follows: （6）， in The maximum score within the window. Let exp be the temperature coefficient, and let exp represent the natural exponential function. This indicates the coordinates within the local window P. Scoring based on key points; express The score of key points at other coordinates (p, q) within the same local window P; Next, using coordinates Spatial probability distribution value at As weights, the expected coordinates of key points within the window are calculated. and subpixel position p: （7）， Finally, the coordinates of the integrated center and subpixel position , the desired coordinates Mapping back to the entire image coordinate system, the final sub-pixel level keypoint coordinates for: （8）， in It represents the coordinates of the center of the local window in its own coordinate system; Step 3: Implement an adaptive mechanism for different resolutions or texture densities; Step 4: Perform end-to-end gradient closure based on backpropagation.

2. The method according to claim 1, characterized in that, In step 1, a single-head coupled output structure is adopted, and key point confidence and descriptors are generated synchronously through a single set of convolution parameters. A gradient coupling mechanism is designed to enable the detection and description tasks to share common features and optimize collaboratively.

3. The method according to claim 2, characterized in that, Step 1 includes: First, a feature map F and a key point score map S are generated using a lightweight convolutional neural network (CNN). The CNN includes four depthwise separable convolutional blocks with a maximum receptive field of 204×204. Let s represent the confidence level of the existence of a keypoint after activation by the Sigmoid function at each pixel position, and use s as the keypoint score; Input feature map Through shared convolution kernel The process has D+1 channels, and the output channel tensor Z has D+1 coordinates. The mapping of the channel tensor Z at point Z is defined as: （1）， in For convolution, Let b be the weight matrix of the convolution kernel, and b be the bias term. b forms a shared convolution kernel The set of trainable parameters.

4. The method according to claim 3, characterized in that, In step 1, record the coordinates. At this point, the normalized unit descriptor vector for: （2）， in, This represents the first D channels of the output tensor Z at coordinates (x, y), and the length of the vector is calculated using the L2 norm. The dimension is D×1.

5. The method according to claim 4, characterized in that, In step 1, record the coordinates. Key points scoring for: （3）， in This represents the (D+1)th channel of the feature tensor Z; For the Sigmoid function; The gradients of the descriptor loss and the detection loss are coupled to the shared convolutional kernel via a chain rule. The same set of trainable parameters is used to adjust The value of b is determined by... Gradient coupling formula: （4）， in Represents partial differentials; This represents the gradient of the descriptor loss with respect to the descriptor d; This represents the gradient of the detection loss with respect to the score s; This indicates the descriptor normalization operation's response to the original descriptor. The gradient; This indicates the Sigmoid activation response to the original score. The gradient; The output Z of the convolutional layer represents the parameter set. The gradient; Key Points With descriptor vector Derived from shared convolutional kernels It accepts gradients from both the descriptor loss and the keypoint loss.

6. The method according to claim 5, characterized in that, Step 3 includes: Define discrete candidate set , representing a predefined optional value for window size N; for the k-th candidate window currently being computed. During training, the logarithmic weights can be learned by updating them through gradient descent. And use Gumbel-Softmax to make the discrete sampling distribution continuous, using temperature parameter adjustment Controlled sampling, selecting weights in the Gumbel distribution and random noise in sampling as follows: （9）， in This indicates that when the summation term in the denominator is traversed through all candidate windows, the j-th candidate window... The corresponding learnable log weights, This indicates that when the summation term in the denominator is traversed through all candidate windows, the j-th candidate window... Random noise sampled using a Gumbel distribution; For each Calculate pixel-level soft coordinates Finally, the pixel-level soft coordinates are weighted and fused to update the sub-pixel position p, achieving differentiable selection: （10）， （11）， Let the temperature coefficient of the probability-weighted average soft-argmax be... ,in, Used to control the concentration of the coordinate probability distribution. These are learnable parameters, initialized to 0; the softplus function is a flexible activation function, used... Will Mapped to , to make temperature parameters Always positive.

7. The method according to claim 6, characterized in that, Step 4 includes: Through end-to-end joint detection and description, each position i corresponds to a candidate coordinate. Mark the sub-pixel keypoint coordinates at position i as The corresponding key point score Redefining it using the Sigmoid function, denoted as... , indicating the position The probability of the existence of a true keypoint is used to calculate the Bernoulli loss, and the formula is as follows: （12）， in, This indicates the sub-pixel keypoint coordinates. The integer pixel coordinates obtained after rounding; Design a supervision signal based on the Bernoulli distribution. For position i, if position i falls within the geometric threshold neighborhood of a ground truth key point... Inside, make the label Otherwise, make the label 0; Confidence loss is calculated using Bernoulli negative log-likelihood. During the model training phase, continuous confidence is enforced. Fitting discrete labels : （13）， To achieve collaborative optimization of detection and description tasks, a joint loss function is constructed. : （14）， in, Represents reprojection loss The weighting coefficients, Indicates the peak dispersion loss The weighting coefficients, Describing the loss The weighting coefficients, Indicates reliability loss The weighting coefficients, Indicates confidence loss Weighting coefficients; The combined formulas (4), (9) to (14) form a differentiable computational graph, and the gradient is completely backflowed through soft-argmax and Bernoulli, realizing the integrated learning of detection and description: （15）， in, This represents the gradient propagated back by the descriptor d. Let L represent the gradient of the loss L with respect to the descriptor d; Describer d pairs The gradient; This represents the gradient of the keypoint score s backpropagated. This represents the gradient of the loss L with respect to the keypoint score s. Indicates the key point score s. gradient, This represents the summation of gradients over all spatial coordinates.

8. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, It stores a computer program or instructions that, when run on a computer, perform the steps of the method as described in any one of claims 1 to 7.

Images