Image matching method and device, equipment, storage medium and product

Abstract

The invention discloses an image matching method and device, equipment, a storage medium and a product, and relates to the technical field of image matching, and the image matching method comprises the steps: obtaining different-source to-be-matched images at different visual angles; sequentially performing feature extraction on the images to be matched through a preset feature point matching model to obtain key points and descriptors in each image to be matched, and performing matching on the key points based on the descriptors to obtain initial matching points obtained through initial matching, the number of channels in a convolutional layer of the preset feature point matching model is negatively correlated with the spatial resolution of the to-be-matched image, the change degree of the number of channels in a first stage is smaller than the change degree of the number of channels in a second stage, and the first stage and the second stage are determined based on the spatial resolution; and performing error filtering on the initial matching point to obtain a filtered target matching point. According to the invention, when an image with high spatial resolution is processed at an early stage, the calculation cost can be significantly reduced by reducing the number of channels.

Description

Technical Field

[0001] This application relates to the field of image matching technology, and in particular to an image matching method, apparatus, device, storage medium and product. Background Technology

[0002] In related technologies, in deep learning models, especially convolutional neural networks (CNNs), the initial layers are usually responsible for processing high spatial resolution information of the input image. These layers need to process a large amount of pixel data, so their computational load and memory consumption are relatively large. Especially in application scenarios with limited resources and complex environments, such computational burden is often unbearable. Summary of the Invention

[0003] The main purpose of this application is to provide an image matching method, apparatus, device, storage medium and product, which aims to solve the technical problem of excessive computational burden of deep learning models in resource-constrained and complex application scenarios such as missile terminal guidance.

[0004] To achieve the above objectives, this application proposes an image matching method, which includes: Obtain heterogeneous source images to be matched from different perspectives; By using a preset feature point matching model, features are extracted sequentially from the images to be matched to obtain key points and descriptors in each image. Based on the descriptors, the key points are matched to obtain initial matching points. The number of channels in the convolutional layer of the preset feature point matching model is negatively correlated with the spatial resolution of the images to be matched. The change in the number of channels in the first stage is less than the change in the second stage. The first stage and the second stage are determined based on the spatial resolution. Error filtering is performed on the initial matching points to obtain the filtered target matching points.

[0005] In one embodiment, the step of sequentially extracting features from the images to be matched using a preset feature point matching model to obtain key points and descriptors in each image to be matched, and then matching the key points based on the descriptors to obtain initial matching points includes: The images to be matched are sequentially input into a preset feature point matching model. Features are extracted from the images to be matched through a convolutional layer to obtain an intermediate feature vector. The number of channels in the convolutional layer increases layer by layer. The intermediate feature vector is input into the key point head for convolution to obtain the key point vector, and the key point vector is converted into an image to obtain a key point heatmap including key point location information. The intermediate feature vector is input into the descriptor head for multi-scale feature extraction to obtain descriptor information. Based on the descriptor information, the descriptor corresponding to the key point is obtained. Based on the descriptor, the key points are matched to obtain the initial matching points obtained from the preliminary matching.

[0006] In one embodiment, the step of inputting the intermediate feature vector into the keypoint header for convolution to obtain a keypoint vector, and converting the keypoint vector into an image to obtain a keypoint heatmap including keypoint location information includes: The intermediate feature vector is divided into grid units of preset pixels; Convert the grid cells into preset dimension vectors; The preset dimension vector is convolved to obtain the key point vector and key point position after convolution and regression. Based on the key point locations, the key point vectors are converted into images to obtain a key point heatmap that includes key point location information.

[0007] In one embodiment, the step of inputting the intermediate feature vector into the descriptor head for multi-scale feature extraction to obtain descriptor information, and obtaining the descriptor corresponding to the key point based on the descriptor information, includes: The intermediate feature vectors at different scales are input into the descriptor header, and the intermediate feature vectors at different scales are unified to the same scale to obtain the unified intermediate feature vector. The unified intermediate feature vector is summed element by element to obtain the summed intermediate feature vector; The intermediate feature vectors after addition are convolved and fused to obtain descriptor information, wherein the descriptor information includes a descriptor graph; The key points in the key point heatmap are mapped to the descriptor graph to obtain the descriptor corresponding to the key point.

[0008] In one embodiment, the step of performing error filtering on the initial matching point to obtain the filtered target matching point includes: A preset number of initial matching points are randomly selected, and the homography matrix corresponding to the preset number of initial matching points is calculated. Based on the homography matrix, the projection error of the preset number of initial matching points is calculated respectively; Based on the projection error, the homography matrix is ​​iteratively optimized to obtain the optimized homography matrix; Based on the optimized homography matrix, the target matching point after error filtering is determined.

[0009] In one embodiment, the descriptor information includes a reliability map, and the step of matching the key points based on the descriptor to obtain initial matching points includes: A preset number of candidate matching points are selected from the reliability map corresponding to the image to be matched; Based on the descriptor, preliminary matching is performed on the candidate matching points in different images to be matched to obtain initial matching points; Predict the offset of each matching point in the initial matching points, and correct the position of the initial matching points based on the offsets to obtain the corrected initial matching points.

[0010] Furthermore, to achieve the above objectives, this application also proposes an image matching device, which includes: The acquisition module 10 is used to acquire images of heterogeneous sources to be matched from different perspectives; The matching module 20 is used to sequentially extract features from the images to be matched using a preset feature point matching model to obtain key points and descriptors in each image to be matched, and to match the key points based on the descriptors to obtain initial matching points. The number of channels in the convolutional layer of the preset feature point matching model is negatively correlated with the spatial resolution of the images to be matched. The change in the number of channels in the first stage is less than the change in the second stage. The first stage and the second stage are determined based on the spatial resolution. The filtering module 30 is used to perform error filtering on the initial matching point to obtain the filtered target matching point.

[0011] In addition, to achieve the above objectives, this application also proposes an image matching device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image matching method as described above.

[0012] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the image matching method described above.

[0013] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the image matching method described above.

[0014] One or more technical solutions proposed in this application have at least the following technical effects: In related technologies, in deep learning models, especially convolutional neural networks (CNNs), the initial layers are typically responsible for processing high spatial resolution information of the input image. These layers need to process a large amount of pixel data, resulting in relatively high computational and memory consumption. This computational burden is often unbearable, especially in resource-constrained and complex environments such as missile terminal guidance. In contrast, this application obtains heterogeneous images to be matched from different perspectives; through a preset feature point matching model, features are extracted from the images to be matched sequentially to obtain key points and descriptors in each image; and based on the descriptors, the key points are matched to obtain initial matching points. The number of channels in the convolutional layers of the preset feature point matching model is negatively correlated with the spatial resolution of the image to be matched. The change in the number of channels in the first stage is less than the change in the second stage, and the first and second stages are determined based on the spatial resolution. Error filtering is applied to the initial matching points to obtain filtered target matching points. This application uses a preset feature point matching model to extract features from heterogeneous images to be matched from different perspectives, obtaining key points and descriptors in each image. Based on the above information, matching is performed to obtain initial matching points. Error filtering is then applied to the matching pairs to obtain filtered target matching points. This application sets the number of channels in the convolutional layer of the preset feature point matching model to be negatively correlated with the spatial resolution of the image to be matched. In the early processing of high spatial resolution images, reducing the number of channels can significantly reduce computational costs. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating an embodiment of the image matching method of this application. Figure 2 This is a schematic diagram of the overall technical solution of the image matching method of this application; Figure 3 This is a schematic diagram of the network structure of the image matching method in this application; Figure 4 This is a flowchart illustrating Embodiment 2 of the image matching method of this application; Figure 5 This is a schematic diagram of the module structure of the image matching device according to an embodiment of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the image matching method in the embodiments of this application.

[0018] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0020] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0021] The main solution of this application embodiment is as follows: acquiring heterogeneous images to be matched from different perspectives; sequentially extracting features from the images to be matched using a preset feature point matching model to obtain key points and descriptors in each image to be matched, and matching the key points based on the descriptors to obtain initial matching points obtained from preliminary matching, wherein the number of channels in the convolutional layer of the preset feature point matching model is negatively correlated with the spatial resolution of the images to be matched, and the change in the number of channels in the first stage is less than the change in the second stage, and the first stage and the second stage are determined based on the spatial resolution; and performing error filtering on the initial matching points to obtain filtered target matching points.

[0022] In related technologies, in deep learning models, especially convolutional neural networks (CNNs), the initial layers are usually responsible for processing high spatial resolution information of the input image. These layers need to process a large amount of pixel data, so their computational load and memory consumption are relatively large. Especially in application scenarios with limited resources and complex environments, such computational burden is often unbearable.

[0023] This application uses a preset feature point matching model to extract features from heterogeneous images to be matched from different perspectives, obtaining key points and descriptors in each image. Based on the above information, matching is performed to obtain initial matching points. Error filtering is then applied to the matching pairs to obtain filtered target matching points. This application sets the number of channels in the convolutional layer of the preset feature point matching model to be negatively correlated with the spatial resolution of the image to be matched. In the early processing of high spatial resolution images, reducing the number of channels can significantly reduce computational costs.

[0024] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or image matching device capable of performing the above functions. The following description uses an image matching device as an example to illustrate this embodiment and the subsequent embodiments.

[0025] Based on this, embodiments of this application provide an image matching method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the image matching method of this application.

[0026] In this embodiment, the image matching method includes steps S10 to S30: Step S10: Obtain the images of the heterogeneous source to be matched from different perspectives; It should be noted that the execution entity in this embodiment is an image matching device. Different viewpoints refer to images acquired from different observation angles or directions. For example, in a missile terminal guidance scenario, target images are acquired from multiple directions (such as left, right, up, and down) by the aircraft or sensors. The range of viewpoint differences can cover 0° to 20° or greater to simulate dynamic observation conditions in a real battlefield environment. Heterogeneous sources refer to image data from different physical sensors or imaging modalities, such as visible light images and infrared images. By acquiring images to be matched from different viewpoints and heterogeneous sources, the image matching device provides diverse input data for the subsequent lightweight deep learning matching network. This effectively simulates the multimodal image matching challenges in complex battlefield environments, laying a data foundation for improving the robustness and accuracy of the algorithm under conditions of heterogeneous viewpoints, large-scale differences, and interference.

[0027] For example, in missile terminal guidance applications, the image matching device can acquire visible light reference images and infrared observation images from different flight paths as images to be matched, such as... Figure 2 As shown, Figure 2 A schematic diagram of the overall technical solution is provided. Figure 1 , Figure 2 The input is fed into the feature point detection-matching network module, and after processing by the XFeat algorithm, high-precision matching is finally achieved to support target recognition and locking.

[0028] Step S20: Using a preset feature point matching model, features are extracted sequentially from the images to be matched to obtain key points and descriptors in each image to be matched. Based on the descriptors, the key points are matched to obtain initial matching points. The number of channels in the convolutional layer of the preset feature point matching model is negatively correlated with the spatial resolution of the images to be matched. The change in the number of channels in the first stage is less than the change in the second stage. The first stage and the second stage are determined based on the spatial resolution. Understandably, the preset feature point matching model employs the lightweight deep learning network XFeat, which uses a feathered CNN architecture, including a descriptor head and a keypoint head, to extract local features from the image. Keypoints are pixels in an image with significant structural features, such as corner points and edge points, used to represent the main content of the image. Descriptors are feature representations of the region surrounding each keypoint, typically high-dimensional vectors, used to measure the similarity between keypoints. The image matching device automatically learns and outputs keypoint locations and descriptor vectors from the input image using the preset feature point matching model. Through feature extraction and initial matching using the preset lightweight feature point matching model, it achieves efficient and accurate acquisition of keypoint correspondences even in resource-constrained environments.

[0029] Furthermore, the number of channels in the convolutional layer of the feature point matching model changes less in the first stage than in the second stage, and the first and second stages are determined based on the spatial resolution of the image to be matched.

[0030] Specifically, in traditional deep convolutional networks, such as VGG and ResNet, the initial convolutional layers are usually set with a high number of channels, such as 64, in order to ensure the expressive power of shallow features. However, in the resource-constrained scenario of missile terminal guidance, the input image resolution is high. If this conventional design is adopted, the computational cost and memory usage of the activation values ​​in the first layer alone will exceed the hardware's carrying capacity. To address this, this application proposes the design strategy of the XFeat network. When processing high spatial resolution images in the early convolutional layers, the number of channels is reduced as much as possible to reduce computational costs. As the spatial resolution decreases, the number of channels is increased to maintain feature expressive power, thus balancing computational efficiency and feature representation capability.

[0031] Specifically, the XFeat network employs a unique channel distribution strategy: an asymmetric, efficiency-prioritized channel allocation strategy. As the spatial resolution is halved, the number of channels increases significantly. Starting from the input layer, the number of channels in each stage follows an increasing sequence from 4, 8, 24, 64, 64 to 128. In the first two stages of processing the original high-resolution image, the number of channels is compressed to single or double digits. This contrasts sharply with general backbone networks. The number of channels is only significantly increased as the spatial resolution is halved step by step through stride convolution (e.g., from H×W to H / 32×W / 32), ensuring that the deep network has sufficient semantic expressive power.

[0032] Furthermore, although the extreme channel compression in the early stages of the XFeat network sacrificed some shallow details, it effectively compensated for the accuracy loss through mechanisms such as multi-scale feature fusion (in the descriptor head) and reliability-based matching refinement (using reliability maps to predict offsets). Finally, residual mismatches were eliminated through geometric consistency constraints such as RANSAC, thus achieving highly robust matching performance with extremely low resource consumption. This end-to-end, application-specific systematic optimization is not available in general segmentation models or standard CNN architectures.

[0033] Furthermore, the number of channels in the convolutional layers of the feature point matching model varies across different regions in the first stage based on the image complexity of the input image.

[0034] Furthermore, the XFeat network is equipped with pre-configured, computationally inexpensive edge detection operators (such as the Sobel operator). Before processing the image through convolutional layers at half the spatial resolution, the input image is quickly scanned to generate a gradient magnitude map reflecting the richness of texture or edge intensity in various local regions of the image. Based on this gradient magnitude map, the image matching device dynamically allocates different computational resources to different spatial locations of the image. Specifically, in high-complexity regions identified as rich in texture and containing significant edges or corners, the device activates all channels of the convolutional layer (e.g., all four channels of the first input layer) to ensure that the detailed information of these key regions is fully captured. Conversely, in low-complexity regions identified as large areas of smoothness and lacking significant structure, the device selectively activates only a portion of the channels of the convolutional layer (e.g., only two channels, or even fewer), while completely skipping or zeroing out the computation of the remaining channels. In this way, when the entire network processes an image, the actual computational cost of its early convolutional layers is no longer fixed, but is intelligently and dynamically scaled according to the image content itself. This greatly saves the ineffective computational cost in irrelevant smooth areas while ensuring the quality of key region features.

[0035] For example, in a missile terminal guidance scenario, a visible light reference image and an infrared observation image are input into the XFeat model, such as... Figure 2 As shown, keypoints and descriptors are extracted through the descriptor header and keypoint header, respectively.

[0036] Step S30: Perform error filtering on the initial matching points to obtain the filtered target matching points.

[0037] It should be noted that the initial matching points refer to the set of key matching point pairs initially obtained through the feature point matching model. This set may include erroneous matching pairs caused by factors such as image noise, viewpoint changes, modal differences, or repetitive textures. The image matching device identifies and removes erroneous matching pairs from the initial matching points, obtaining high-precision, high-reliability target matching points that satisfy geometric consistency constraints after error filtering.

[0038] In one feasible implementation, step S20 includes: The images to be matched are sequentially input into a preset feature point matching model. Features are extracted from the images to be matched through a convolutional layer to obtain an intermediate feature vector. The number of channels in the convolutional layer increases layer by layer. As can be understood, convolutional layers are the basic building blocks in the XFeat network. Each basic layer contains 2D convolutions with kernel sizes ranging from 1 to 3, combined with ReLU activation functions and BatchNorm normalization layers, and uses convolutions with a stride of 2 for resolution reduction. The image matching device systematically increases the feature dimension while gradually reducing spatial resolution by employing a convolutional layer structure with progressively increasing channel count. This allows the network to retain rich spatial detail information in shallow layers and capture higher-level semantic features in deeper layers, thereby obtaining intermediate feature vectors.

[0039] The intermediate feature vector is input into the key point head for convolution to obtain the key point vector, and the key point vector is converted into an image to obtain a key point heatmap including key point location information. It's important to note that the keypoint head refers to the parallel branch in the XFeat network specifically responsible for keypoint detection. It uses 1x1 convolutions to rapidly process the image after transforming the 8x8 tensor blocks. The keypoint vector is the feature representation obtained after processing by the keypoint head, encoding the location information of keypoints within each 8x8 grid cell. The keypoint heatmap converts the keypoint vector into a two-dimensional image representation, where each pixel value reflects the confidence or probability of a keypoint being present at the corresponding location. The image matching device generates the keypoint heatmap through the dedicated keypoint head, decoupling the keypoint detection and description processes. This allows the network to quickly locate salient feature points in the image while maintaining spatial resolution, effectively improving the accuracy and robustness of keypoint detection in a compact network architecture and providing reliable keypoint location information for subsequent feature matching.

[0040] Specifically, refer to Figure 3 , Figure 3 A schematic diagram of the network structure is provided. After the intermediate feature vector is input to the keypoint head, it is processed by four convolutional layers to obtain the keypoint vector. This vector is reinterpreted as an 8x8 grid representation, and finally a keypoint heatmap is generated. The highlighted areas represent the high-confidence locations of the keypoints. These keypoints will be used in the subsequent feature matching process.

[0041] The intermediate feature vector is input into the descriptor head for multi-scale feature extraction to obtain descriptor information. Based on the descriptor information, the descriptor corresponding to the key point is obtained. Understandably, the descriptor head refers to the parallel branch in the XFeat network specifically responsible for generating feature descriptors, which fuses multi-scale features through a feature pyramid strategy. By extracting multi-scale features from the descriptor head, the image matching device can fully utilize semantic information at different scales to generate compact descriptors with strong discriminative power. These descriptors exhibit good robustness to viewpoint changes, scale differences, and modal differences, providing a reliable similarity measurement basis for subsequent accurate feature matching.

[0042] For example, such as Figure 3 As shown, after the intermediate feature vector is input into the descriptor head, the feature representations of the three scales are fused through the feature pyramid strategy. After convolutional fusion block, the final 64-dimensional dense feature map F is generated. Based on the position determined by the key point heatmap, the 64-dimensional descriptor corresponding to each key point is sampled from F. These descriptors will be used for subsequent initial matching point generation.

[0043] Based on the descriptor, the key points are matched to obtain the initial matching points obtained from the preliminary matching.

[0044] It should be noted that the image matching device uses descriptors extracted from two images to be matched to find the most similar corresponding key points in another image, and obtains the initial matching point through the above matching process.

[0045] In one feasible implementation, the steps of inputting the intermediate feature vector into the keypoint header for convolution to obtain a keypoint vector, and converting the keypoint vector into an image to obtain a keypoint heatmap including keypoint location information include: The intermediate feature vector is divided into grid units of preset pixels; It is understandable that the preset pixel grid unit refers to dividing the feature map into 8×8 pixel grid units in spatial dimension. The image matching device reorganizes the spatial dimension H×W of the intermediate feature vector into (H / 8)×(W / 8) grid units, with each unit containing features corresponding to 8×8 spatial locations. By dividing the intermediate feature vector into 8×8 pixel grid units, the feature representation can be effectively organized while maintaining spatial granularity, providing a structured input format for subsequent keypoint detection. This partitioning method not only preserves the spatial detail information within the unit but also improves computational efficiency through unit-level processing, which is an important prerequisite for achieving efficient keypoint detection.

[0046] Convert the grid cells into preset dimension vectors; It should be noted that the preset dimension vector refers to a 64-dimensional feature vector. The image matching device reorganizes the features of the 64 spatial locations within each 8×8 grid cell into a 64-dimensional vector. By converting the grid cell into a preset dimension vector, it can generate a compact and structured feature representation while preserving the spatial details within the cell. This conversion facilitates the rapid processing of key point distribution, improves the efficiency and accuracy of key point detection, and reduces computational complexity, making it suitable for resource-constrained deployment environments.

[0047] The preset dimension vector is convolved to obtain the key point vector and key point position after convolution and regression. Understandably, the image matching device uses a 1×1 convolutional layer to perform fast linear transformation and non-linear activation processing on a preset dimension vector to obtain a key point vector, and then analyzes the key point position of each obtained key point in the image.

[0048] Based on the key point locations, the key point vectors are converted into images to obtain a key point heatmap that includes key point location information.

[0049] It should be noted that the image matching device processes the positional information in the keypoint vectors through the softmax function, then reinterprets and maps it into a two-dimensional image space to obtain the converted keypoint heatmap.

[0050] In one feasible implementation, the intermediate feature vector is input into the descriptor head for multi-scale feature extraction to obtain descriptor information. Based on the descriptor information, the step of obtaining the descriptor corresponding to the key point includes: The intermediate feature vectors at different scales are input into the descriptor header, and the intermediate feature vectors at different scales are unified to the same scale to obtain the unified intermediate feature vector. Understandably, different scales refer to the feature representations extracted from multiple convolutional layers of the XFeat network. These features have different spatial resolutions, such as 1 / 8, 1 / 16, and 1 / 32 of the original input image size, capturing different levels of semantic information. The image matching device upscales the lower-resolution feature maps to higher resolutions through upsampling operations (such as bilinear interpolation), ensuring that all feature maps have the same spatial size (e.g., H / 8 × W / 8) to obtain a unified intermediate feature vector.

[0051] The unified intermediate feature vector is summed element by element to obtain the summed intermediate feature vector; It should be noted that the image matching device adds the unified intermediate feature vector element by element, which can effectively fuse feature information from different scales to obtain a single feature representation.

[0052] The intermediate feature vectors after addition are convolved and fused to obtain descriptor information, wherein the descriptor information includes a descriptor graph; Understandably, image matching devices, through convolutional fusion processing, can further optimize and integrate the features after multi-scale fusion, generating descriptive sub-images with stronger discriminative power and better consistency.

[0053] The key points in the key point heatmap are mapped to the descriptor graph to obtain the descriptor corresponding to the key point.

[0054] It should be noted that the image matching device can assign a suitable descriptor to each detected key point by spatially aligning the key point heatmap with the descriptor map and sampling its features.

[0055] In one feasible implementation, the step of matching the key points based on the descriptor to obtain the initial matching points includes: A preset number of candidate matching points are selected from the reliability map corresponding to the image to be matched; As can be understood, the reliability map refers to the two-dimensional probability map regressed from the descriptor head in the XFeat network. Each pixel value represents the unconditional probability that the corresponding local feature can be confidently matched, and it is used to evaluate the matching reliability of feature points. The preset number refers to a pre-set integer K, used to control the number of candidate matching points selected to balance computational efficiency and matching accuracy. The image matching device selects the top K high-confidence feature points based on the reliability map scores, and these points will be used as candidate matching points.

[0056] Based on the descriptor, preliminary matching is performed on the candidate matching points in different images to be matched to obtain initial matching points; It should be noted that the image matching device uses the nearest neighbor search algorithm. For each candidate matching point descriptor in the reference image, it searches for the most similar one among all candidate matching point descriptors in the observed image to form a preliminary correspondence, that is, to obtain the initial matching point.

[0057] Predict the offset of each matching point in the initial matching points, and correct the position of the initial matching points based on the offsets to obtain the corrected initial matching points.

[0058] As can be understood, the offset refers to the pixel-level position correction value predicted by the multilayer perceptron (MLP) in the matching refinement module, used to correct coarse keypoint matches to more precise pixel positions. The image matching device adds the predicted offset to the coordinates of the initial match, thereby obtaining a high-precision corrected match pair. A matching refinement module based on multilayer perceptron (MLP) is proposed, which can recover pixel-level offsets from coarse local descriptors without the need for high-resolution feature maps, thus efficiently achieving semi-dense matching.

[0059] In one feasible implementation, the XFeat model is trained using supervised learning and utilizes various loss functions to optimize network parameters. This process primarily relies on pixel-level ground truth correspondences to ensure the model can learn high-precision feature matching capabilities. The following is a detailed analysis: 1. Training Data Preparation Dataset composition: Includes real-world datasets (such as Megadepth) and synthetic data (generated by applying geometric transformations, radiometric variations, etc., to COCO images). These data are mixed in a certain ratio (e.g., 6:4) and the image sizes are adjusted to a uniform size (e.g., 800x600).

[0060] Annotation information: Provides pixel-level true correspondences for supervised training.

[0061] 2. Loss Function Design To optimize the different components of the XFeat network (keypoint detection head, descriptor head, reliability map, and matching refinement module), a combination of multiple loss functions was employed. Each loss function targets a specific task objective, working together to improve overall performance.

[0062] a. Descriptor learning loss (L_desc) Objective: Maximize the descriptor similarity of matching point pairs and minimize the similarity of non-matching point pairs.

[0063] Method: Local feature embedding F is supervised using negative log-likelihood (NLL) loss. Specifically, descriptor pairs are sampled from the dense descriptor maps of two images, and a similarity matrix S is constructed. After applying double softmax, the values ​​on the main diagonal represent the probability of a correct match, which is optimized using NLL loss.

[0064] b. Reliability learning loss (L_rel) Objective: To predict the match confidence (i.e. reliability) for each descriptor.

[0065] Methods: The double softmax probabilities are interpreted as confidence metrics, and the reliability plot R is directly supervised. L1 loss is used to measure the difference between the predicted reliability and the target reliability calculated based on the double softmax scores.

[0066] c. Pixel offset learning loss (L_refine) Objective: Improve matching accuracy to the sub-pixel level.

[0067] Method: The matching refinement module supervises the pixel-level offsets obtained from the ground truth correspondence of the original input image resolution. Similarly, NLL loss is used to calculate logits o, where o represents the predicted offset distribution.

[0068] d. Keypoint detection loss (L_kp) Objective: To learn high-quality key point locations.

[0069] Methods: Knowledge distillation is employed to learn from a stronger teacher network (such as ALIKE). ALIKE keypoints are selected as the soft-label supervised model, and NLL loss is used to optimize the keypoint heatmap.

[0070] 3. Total Loss Function The total loss is a linear combination of all the losses mentioned above: L=αLdesc+βLrel+γLrefine+δLkp Here, α, β, γ, and δ are hyperparameters used to balance the importance of different tasks.

[0071] In this embodiment, the XFeat featherweight CNN backbone network is proposed and implemented. Its innovation lies in the strategy of significantly reducing the number of channels in the early layers and tripling the number of channels in the later layers. While maintaining high spatial resolution, it significantly reduces the computational cost and memory usage. This architecture design enables the algorithm to achieve efficient visual correspondence on resource-constrained devices, breaking the limitations of traditional deep learning models in terms of computational resources.

[0072] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 Step S30, the image matching method further includes steps S01~S04: Step S01: Randomly select a preset number of initial matching points and calculate the homography matrix corresponding to the preset number of initial matching points; It should be noted that the preset number refers to the minimum number of sample point pairs required to calculate the homography matrix. For homography matrix estimation, this number is usually 4 pairs of matching points. The image matching device uses the random sampling strategy in the Random Sample Consensus (RANSAC) algorithm to randomly select 4 pairs of matching points from the initial set of matching points and calculate the homography matrix H, which describes the projection transformation relationship between two image planes and can characterize geometric transformations such as rotation, translation, scaling, and projection.

[0073] Step S02: Based on the homography matrix, calculate the projection error of the preset number of initial matching points respectively; As can be understood, projection error refers to the Euclidean distance between the coordinates of the projected point and its true matching point after projecting the matching point in the source image onto the target image plane through the homography matrix H. The source image and target image correspond to the images in the image to be matched. The image matching device projects all initial matching points in the source image onto the target image coordinate system through the currently estimated homography matrix H, calculates the Euclidean distance between each projected point and its corresponding true matching point as the projection error, and marks the matching point as a mismatch and removes it when the error exceeds a preset threshold (e.g., 3 pixels).

[0074] Step S03: Based on the projection error, iteratively optimize the homography matrix to obtain the optimized homography matrix; It should be noted that the image matching device repeatedly performs the cyclic process of "randomly selecting matching points - calculating homography matrix - calculating projection error - filtering interior points". Each iteration recalculates a more accurate homography matrix based on the current set of interior points. After multiple iterations, the final homography matrix calculated from the largest set of interior points best reflects the true geometric transformation relationship between the two images.

[0075] For example, the image matching device performs iterative optimization using the RANSAC algorithm: each time, four pairs of matching points are randomly selected to calculate the initial homography matrix, then the projection error of all matching points is calculated, and points with errors less than a threshold are classified as inliers. After repeating this process multiple times, the homography matrix with the most inliers is selected as the basis, and finally the optimized homography matrix is ​​calculated using the least squares method with all inliers.

[0076] Step S04: Based on the optimized homography matrix, determine the target matching point after error filtering.

[0077] Understandably, the image matching device uses the optimized homography matrix H to calculate the final projection error of all initial matching points, and only retains the matching points with errors less than a set threshold (such as 3 pixels) as target matching points. These high-quality matching points will be used for the final image transformation or target localization.

[0078] In this embodiment, the final geometric verification based on the optimized homography matrix can effectively identify and eliminate the remaining mismatched points, ensuring that the final set of matching points has high geometric consistency and positional accuracy. This is crucial for subsequent tasks such as image registration and target localization, especially in application scenarios with extremely high matching accuracy requirements, such as missile terminal guidance.

[0079] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the image matching method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0080] This application also provides an image matching device, please refer to... Figure 5 The image matching device includes: The acquisition module 10 is used to acquire images of heterogeneous sources to be matched from different perspectives; The matching module 20 is used to sequentially extract features from the images to be matched using a preset feature point matching model to obtain key points and descriptors in each image to be matched, and to match the key points based on the descriptors to obtain initial matching points. The number of channels in the convolutional layer of the preset feature point matching model is negatively correlated with the spatial resolution of the images to be matched. The change in the number of channels in the first stage is less than the change in the second stage. The first stage and the second stage are determined based on the spatial resolution. The filtering module 30 is used to perform error filtering on the initial matching point to obtain the filtered target matching point.

[0081] Optionally, the matching module includes: The feature extraction submodule is used to sequentially input the image to be matched into a preset feature point matching model, extract features from the image through convolutional layers to obtain intermediate feature vectors, wherein the number of channels in the convolutional layers increases layer by layer; input the intermediate feature vectors into the keypoint header for convolution to obtain keypoint vectors, and convert the keypoint vectors into an image to obtain a keypoint heatmap including keypoint location information; input the intermediate feature vectors into the descriptor header for multi-scale feature extraction to obtain descriptor information, obtain descriptors corresponding to keypoints based on the descriptor information; and match the keypoints based on the descriptors to obtain initial matching points obtained from preliminary matching.

[0082] Optionally, the feature extraction submodule includes: A conversion unit is used to divide the intermediate feature vector into grid cells of preset pixels; convert the grid cells into vectors of preset dimensions; convolve the preset dimension vectors to obtain the regressed keypoint vectors and keypoint positions; and based on the keypoint positions, convert the keypoint vectors into an image to obtain a keypoint heatmap including keypoint position information.

[0083] The fusion unit is used to input the intermediate feature vectors of different scales into the descriptor header, unify the intermediate feature vectors of different scales to the same scale, and obtain the unified intermediate feature vector; add the unified intermediate feature vector element by element to obtain the summed intermediate feature vector; perform convolution fusion on the summed intermediate feature vector to obtain descriptor information, wherein the descriptor information includes a descriptor graph; and map the key points in the key point heatmap to the descriptor graph to obtain the descriptor corresponding to the key point.

[0084] The prediction unit is configured to select a preset number of candidate matching points in the reliability map corresponding to the image to be matched; perform preliminary matching on the candidate matching points in different images to be matched based on the descriptor to obtain initial matching points; predict the offset of each matching point in the initial matching points; and perform position correction on the initial matching points based on the offset to obtain the corrected initial matching points.

[0085] Optionally, the filtering module includes: An optimization submodule is used to randomly select a preset number of initial matching points, calculate the homography matrix corresponding to the preset number of initial matching points; calculate the projection error of the preset number of initial matching points based on the homography matrix; iteratively optimize the homography matrix based on the projection error to obtain the optimized homography matrix; and determine the target matching point after error filtering based on the optimized homography matrix.

[0086] The image matching apparatus provided in this application, employing the image matching method described in the above embodiments, can solve the technical problem of image matching. Compared with the prior art, the beneficial effects of the image matching apparatus provided in this application are the same as those of the image matching method described in the above embodiments, and other technical features in the image matching apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0087] This application provides an image matching device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the image matching method in Embodiment 1 above.

[0088] The following is for reference. Figure 6 The diagram illustrates a structural schematic of an image matching device suitable for implementing embodiments of this application. The image matching device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, tablets, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The image matching device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0089] like Figure 6As shown, the image matching device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the image matching device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the image matching device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows image matching devices with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0090] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0091] The image matching device provided in this application, employing the image matching method described in the above embodiments, can solve the technical problem of image matching. Compared with the prior art, the beneficial effects of the image matching device provided in this application are the same as those of the image matching method described in the above embodiments, and other technical features of the image matching device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0092] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0093] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0094] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the image matching method in the above embodiments.

[0095] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0096] The aforementioned computer-readable storage medium may be included in the image matching device; or it may exist independently and not be assembled into the image matching device.

[0097] The aforementioned computer-readable storage medium carries one or more programs. When the one or more programs are executed by the image matching device, the image matching device performs the following actions: acquires heterogeneous images to be matched from different viewpoints; sequentially extracts features from the images to be matched using a preset feature point matching model to obtain key points and descriptors in each image to be matched, and matches the key points based on the descriptors to obtain initial matching points, wherein the number of channels in the convolutional layer of the preset feature point matching model is negatively correlated with the spatial resolution of the images to be matched, and the change in the number of channels in the first stage is less than the change in the second stage, and the first and second stages are determined based on the spatial resolution; and performs error filtering on the initial matching points to obtain filtered target matching points.

[0098] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0100] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0101] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described image matching method, thereby solving the technical problem of image matching. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the image matching method provided in the above embodiments, and will not be repeated here.

[0102] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image matching method described above.

[0103] The computer program product provided in this application can solve the technical problem of image matching. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the image matching method provided in the above embodiments, and will not be repeated here.

[0104] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. An image matching method, characterized in that, The image matching method includes: Obtain heterogeneous source images to be matched from different perspectives; By using a preset feature point matching model, features are extracted sequentially from the images to be matched to obtain key points and descriptors in each image. Based on the descriptors, the key points are matched to obtain initial matching points. The number of channels in the convolutional layer of the preset feature point matching model is negatively correlated with the spatial resolution of the images to be matched. The change in the number of channels in the first stage is less than the change in the second stage. The first stage and the second stage are determined based on the spatial resolution. Error filtering is performed on the initial matching points to obtain the filtered target matching points.

2. The image matching method as described in claim 1, characterized in that, The steps of sequentially extracting features from the images to be matched using a preset feature point matching model to obtain key points and descriptors in each image, and then matching the key points based on the descriptors to obtain initial matching points include: The images to be matched are sequentially input into a preset feature point matching model. Features are extracted from the images to be matched through a convolutional layer to obtain an intermediate feature vector. The number of channels in the convolutional layer increases layer by layer. The intermediate feature vector is input into the key point head for convolution to obtain the key point vector, and the key point vector is converted into an image to obtain a key point heatmap including key point location information. The intermediate feature vector is input into the descriptor head for multi-scale feature extraction to obtain descriptor information. Based on the descriptor information, the descriptor corresponding to the key point is obtained. Based on the descriptor, the key points are matched to obtain the initial matching points obtained from the preliminary matching.

3. The image matching method as described in claim 2, characterized in that, The step of inputting the intermediate feature vector into the keypoint header for convolution to obtain the keypoint vector, and converting the keypoint vector into an image to obtain a keypoint heatmap including keypoint location information includes: The intermediate feature vector is divided into grid units of preset pixels; Convert the grid cells into preset dimension vectors; The preset dimension vector is convolved to obtain the key point vector and key point position after convolution and regression. Based on the key point locations, the key point vectors are converted into images to obtain a key point heatmap that includes key point location information.

4. The image matching method as described in claim 2, characterized in that, The step of inputting the intermediate feature vector into the descriptor head for multi-scale feature extraction to obtain descriptor information, and obtaining the descriptor corresponding to the key point based on the descriptor information, includes: The intermediate feature vectors at different scales are input into the descriptor header, and the intermediate feature vectors at different scales are unified to the same scale to obtain the unified intermediate feature vector. The unified intermediate feature vector is summed element by element to obtain the summed intermediate feature vector; The intermediate feature vectors after addition are convolved and fused to obtain descriptor information, wherein the descriptor information includes a descriptor graph; The key points in the key point heatmap are mapped to the descriptor graph to obtain the descriptor corresponding to the key point.

5. The image matching method as described in claim 1, characterized in that, The step of performing error filtering on the initial matching point to obtain the filtered target matching point includes: A preset number of initial matching points are randomly selected, and the homography matrix corresponding to the preset number of initial matching points is calculated. Based on the homography matrix, the projection error of the preset number of initial matching points is calculated respectively; Based on the projection error, the homography matrix is ​​iteratively optimized to obtain the optimized homography matrix; Based on the optimized homography matrix, the target matching point after error filtering is determined.

6. The image matching method as described in claim 2, characterized in that, The descriptor information includes a reliability map, and the step of matching the key points based on the descriptor to obtain the initial matching points includes: A preset number of candidate matching points are selected from the reliability map corresponding to the image to be matched; Based on the descriptor, preliminary matching is performed on the candidate matching points in different images to be matched to obtain initial matching points; Predict the offset of each matching point in the initial matching points, and correct the position of the initial matching points based on the offsets to obtain the corrected initial matching points.

7. An image matching device, characterized in that, The device includes: The acquisition module is used to acquire heterogeneous source images to be matched from different perspectives; The matching module is used to sequentially extract features from the images to be matched using a preset feature point matching model, obtain key points and descriptors in each image to be matched, and match the key points based on the descriptors to obtain initial matching points. The number of channels in the convolutional layer of the preset feature point matching model is negatively correlated with the spatial resolution of the images to be matched. The change in the number of channels in the first stage is less than the change in the second stage. The first stage and the second stage are determined based on the spatial resolution. The filtering module is used to perform error filtering on the initial matching points to obtain the filtered target matching points.

8. An image matching device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image matching method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the image matching method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the image matching method as described in any one of claims 1 to 6.

Images