A real-time image matching method based on a hybrid MLP structure

Abstract

The application discloses a real-time image matching method based on a hybrid MLP structure, comprising: acquiring image data and extracting a feature map; constructing a hybrid MLP architecture to divide the obtained feature map into windows, fuse information at different spatial positions, and output features of a spatial mixing layer and a channel mixing layer; and inputting the features into a matching model, using a differentiable matching layer of the matching model to match the features into a confidence matrix, selecting matching items in the confidence matrix according to a confidence threshold and a mutual nearest neighbor criterion, obtaining a coarse level matching result; inputting the coarse level matching result into a Mixer-WMLP module for several times of transformation, finally obtaining matching with sub-pixel accuracy, and obtaining a final detection matching result through correction and softmax. The application designs an image matching model based on a hybrid MLP architecture, reduces the calculation complexity and running time of image matching, can be applied to low-level visual tasks of image matching, has good matching effect, and reduces the calculation cost.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image matching, and in particular to a real-time image matching method based on a hybrid MLP structure. Background Art

[0002] Image matching, a fundamental and critical process in visual perception, connects two image objects with identical or similar attributes. It serves as a bridge from low-level vision to high-level vision, and is an effective means of recognizing and integrating information, as well as recovering high-dimensional structure from low-dimensional images. Image matching technology, a key research direction in computer vision, aims to quickly and accurately establish correspondences between similar objects or scenes, transferring information from known images to new images through estimated correspondences. Feature matching techniques can establish spatial correspondences between images or objects, providing prior knowledge for subsequent specific visual tasks such as object tracking, 3D geometric reconstruction, autonomous ground navigation, augmented reality, and structure from motion, thereby improving the efficiency and accuracy of related tasks. Recent advances in multilayer perceptron (MLP) models have provided new and effective network architecture designs for computer vision tasks. Compared with convolutional neural networks (CNNs) and visual transformers, MLP-based visual backbones exhibit less induced bias, improve sample efficiency, and reduce computational costs.

[0003] The drawback of existing techniques is that convolutional neural networks (CNNs) have limited receptive fields and may not be able to distinguish subtle areas. The global receptive field and positional encoding of the Transformer attention mechanism make the transformed feature representation contextual and position-dependent, but at the cost of very high computational and memory costs. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology. To achieve the above purpose, a real-time image matching method based on a hybrid MLP structure is adopted to solve the problems raised in the above background technology.

[0005] A real-time image matching method based on a hybrid MLP structure includes the following steps:

[0006] Step S1: Acquire image data and perform feature extraction on the image data through local feature extraction to obtain a feature map;

[0007] Step S2: construct an image matching model based on a hybrid MLP architecture to divide the obtained feature map into windows according to a preset size, and pass it through the MLP module to fuse information from different spatial positions, and output the features of the spatial mixing layer and the channel mixing layer;

[0008] Step S3: input the obtained features into the matching model, use the differentiable matching layer of the matching model to match the features into a confidence matrix, and select matching items in the confidence matrix according to the confidence threshold and the mutual nearest neighbor criterion to obtain a coarse level matching result;

[0009] Step S4: Input the coarse level matching result into the Mixer-WMLP module for several transformations to finally obtain a match with sub-pixel accuracy, and obtain the final detection matching result through correction and softmax.

[0010] As a further solution of the present invention: the specific steps in step S2 include:

[0011] Build an image matching model based on a hybrid MLP architecture;

[0012] The Mixer-WMLP module of the hybrid MLP architecture divides the tensor between windows of preset sizes through window partitioning, and obtains the feature map at the coarse level as (N,H,W,C), where the partition is (num_windows,window_size,window_size,C), where N is the number of feature maps;

[0013] After two MLP modules, the information from different spatial positions is integrated to form N C Spatial mixing layer and channel mixing layer.

[0014] As a further solution of the present invention: the specific steps in the Mixer-WMLP module include:

[0015] The feature map is evenly divided into non-overlapping windows through the Mixer-WMLP module, and each window is expanded as a token for token information exchange between spatial locations;

[0016] The MLP module is used to map columns and rows in sequence to integrate the information in the spatial domain and channel domain. The input feature map is defined as H×W×C, and the window size is M. Then there are H / M×W / M windows. The calculation complexity is as follows:

[0017] Ω(MLP)=2α(HW) 2 C;

[0018]

[0019] Among them, α is the expansion factor of the first MLP node, and C represents the feature dimension.

[0020] As a further solution of the present invention: the specific steps in step S3 include:

[0021] The obtained features are input into a matching model, and a differentiable matching layer is used to match the converted features into a confidence matrix P C ;

[0022] Then, according to a confidence threshold and a mutual nearest neighbor criterion, a matching item in P C is selected to obtain a coarse level matching result M C .

[0023] As a further scheme of the present application, the specific steps in step S4 include:

[0024] For each pair of coarse level matching results obtained, first, the position of the coarse level matching result is located Then, it is divided into a local window of WxW.

[0025] Through the Mixer-WMLP module, the features cropped in each window are transformed N f times to obtain and center vectors and are associated with each other to generate a heat map heatmap.

[0026] Then, the expectation on the probability distribution thereof is calculated to obtain a final matching M f with sub-pixel accuracy, and the final detection matching result is obtained through correction and softmax

[0027] Compared with the prior art, the present application has the following technical effects:

[0028] By adopting the above technical scheme, by designing the Mixer-WMLP module based on the mixed MLP architecture, a sampling area close to the global is obtained by dividing the feature map into non-overlapping windows. The image matching method is obtained, which reduces the calculation complexity and running time of image matching. It can be applied to low-level vision tasks of image matching. Compared with the image matching method based on the transformer in the prior art, the MAIM algorithm based on Mixer-WMLP reduces the GPU calculation amount and has a global domain comparable to the transformer. Since the applicability of MAIM represents the subsequent research target, we will consider applying MAIM to visual tasks such as visual odometry. The present application not only has good matching effect, but also reduces the calculation cost, and is more suitable for image matching in complex traffic scenes. BRIEF DESCRIPTION OF DRAWINGS

[0029] The specific embodiments of the present application will be described in detail below with reference to the accompanying drawings:

[0030] Figure 1 It is a step schematic diagram of the image matching method of the disclosed embodiment of the present application.

[0031] Figure 2 This is a diagram of the overall network structure of the image matching method disclosed in the embodiment of this application;

[0032] Figure 3 A schematic diagram of a Mixer-WMLP network according to an embodiment disclosed in this application;

[0033] Figure 4 A schematic diagram of sampling positions of the image matching method according to the embodiment disclosed in this application;

[0034] Figure 5 This is a schematic diagram of the indoor visualization results of the embodiment disclosed in this application;

[0035] Figure 6 This is a schematic diagram of the outdoor visualization results of the embodiment disclosed in this application. DETAILED DESCRIPTION

[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0037] Please refer to Figure 1 and Figure 2 In an embodiment of the present invention, a real-time image matching method based on a hybrid MLP structure includes the following steps:

[0038] Step S1: Acquire image data and perform feature extraction on the image data through local feature extraction to obtain a feature map;

[0039] In this embodiment, a data set of image data can be obtained according to actual conditions, and then a feature map can be extracted through local features;

[0040] Step S2: Construct an image matching model based on a hybrid MLP architecture to divide the obtained feature map into windows according to a preset size, and pass it through the MLP module to fuse information from different spatial positions, and output the features of the spatial mixing layer and the channel mixing layer. The specific steps include:

[0041] Step S21: constructing an image matching model based on a hybrid MLP architecture;

[0042] In this embodiment, Figure 3The figure shows a schematic diagram of the Mixer-WMLP network. A hybrid MLP architecture called Mixer-WMLP is constructed, which evenly divides the feature map into non-overlapping windows and expands each window as a token to achieve token information exchange between spatial locations. MLP maps columns and rows in sequence to integrate information from spatial and channel regions. Unlike traditional convolution, the spatial domain and channel domain are operated separately. The former allows communication between different spatial tokens operating on a channel, treating each column as input, while the latter allows communication between different channels. The input and output sizes of each layer are equivalent, preventing the loss of resolution as the layer depth increases.

[0043] Step S22: using the Mixer-WMLP module of the hybrid MLP architecture to divide the tensor between windows of a preset size through window partitioning, the coarse-level feature map is obtained as (N, H, W, C), where the partition is (num_windows, window_size, window_size, C), where N is the number of feature maps;

[0044] After two MLP modules, the information from different spatial positions is integrated to form N C Spatial mixing layer and channel mixing layer.

[0045] In this embodiment, the input feature map is defined as H×W×C, and the window size is M. Then there are H / M×W / M windows. The calculation complexity is calculated as follows:

[0046] Ω(MLP)=2α(HW) 2 C;

[0047]

[0048] Among them, α is the expansion factor of the first MLP node, and C represents the feature dimension.

[0049] In this embodiment, the size of the sampling position of our method is as follows: Figure 2 As shown in the figure, a schematic diagram of sampling locations is shown. We introduce cross-window connections to maintain efficient computation of non-overlapping windows, essentially achieving global sampling.

[0050] The convolution operation has a local receptive field, so it is more suitable for extracting features with local dependencies.

[0051]

[0052] Among them, W conv ∈R k×k×C are learnable weights, h and w are the height and width of X respectively.

[0053] Transform: embed the input X from the image sequence, get the Q, K, V matrices, and output Y as the feature attention combination:

[0054]

[0055] Our method introduces a window into the MLP-based architecture. It first transposes the input X from the window and then appends a token mixing MLP within the window to obtain the output

[0056]

[0057] Where W is the learnable weight in the MLP. This method uses matrix transposition and the MLP to perceive global information. The introduction of locality further improves the performance of the MLP-based architecture and reduces computational complexity.

[0058] Step S3: Input the obtained features into the matching model, use the differentiable matching layer of the matching model to match the features into a confidence matrix, and select matching items in the confidence matrix according to the confidence threshold and the mutual nearest neighbor criterion to obtain a coarse level matching result. The specific steps include:

[0059] The obtained features are input into the matching model, and the converted features are matched into the confidence matrix P using the differentiable matching layer. C ;

[0060] Then, P is selected based on the confidence threshold and the mutual nearest neighbor criterion. C The matching items in the , get the coarse level matching result M C .

[0061] Step S4: Input the coarse level matching result into the Mixer-WMLP module for several transformations to finally obtain a match with sub-pixel accuracy. The final detection matching result is obtained through correction and softmax. The specific steps include:

[0062] For each pair of coarse level matching results obtained, first locate the position of the coarse level matching result Then divide it into W×W local windows;

[0063] The features cropped in each window are N-thrown through the Mixer-WMLP module f Second transformation, we get and Center vector and Correlate with each other to generate a heatmap;

[0064] Then calculate the expectation on its probability distribution to obtain the final matching M with sub-pixel accuracyf , the final detection and matching results are obtained through correction and softmax

[0065] The following two different experiments are set up to verify the effect of the method of the present invention:

[0066] (1) Indoor pose estimation

[0067] Relative indoor poses were tested using the ScanNet dataset, using only ground truth pose and depth, and selecting images based on the overlap score computed over all possible image pairs in a given sequence. Following the same procedure as Superglue [8], 1500 test pairs with an overlap score between 0.4 and 0.8 were selected. This dataset contains image pairs with a wide baseline and many textureless regions.

[0068] Evaluation scheme: We report the AUC for position error thresholds of (5°, 10°, and 20°), where the position error is the maximum of the rotation and translation errors. Relative poses are obtained by fundamental matrix estimation and RANSAC. We also report the matching accuracy and scores [2, 4], where matches based on the distance are considered correct.

[0069] The following local attributes are also evaluated as references: the hand-crafted method ORB; the CNN-based image matching methods GMS, D2-Net, Superpoint and ContextDesc, which use public access training models; the CNN-based detector-free matching method DRC-Net and the transformer-based image matching method LoFTR, and train the model on the MegaDepth dataset, as shown in Table 1 below, for evaluation in indoor pose estimation;

[0070] Table 1 Evaluation in indoor pose estimation

[0071]

[0072] In this example, the extracted point pairs are not as dense as those in LoFTR, but our method runs faster. Compared to Superpoint+SuperGlue, our method can extract more correct matches, which gives it a significant advantage in solving 3D vision tasks such as visual odometry. It runs in 72 milliseconds for a 640×480 image pair on an RTX 2080. This figure is significantly lower than the 118 milliseconds required by the Transformer-based LoFTR method and close to the 69 milliseconds required by the CNN-based Superpoint. These results demonstrate the effectiveness of our proposed method.

[0073] Even in fuzzy areas with low texture or repeated patterns, the present invention can produce high-quality matches and achieve good matching results, such as Figure 5 As shown in the figure, it is a schematic diagram of indoor visualization results.

[0074] (2) Outdoor pose estimation

[0075] Following the process used by SuperGlue, all learned methods are trained on the MegaDepth dataset, which contains depth maps for multi-view stereo computation. The overlap score is used to compute the SfM co-occurrence [2, 51]. We adopt the same method as DISK, only considering the "Sacred Heart" and "St. Peter's Square" scenes for validation. The images are resized so that their long dimension is set to 640. A total of 1800 images are selected for the test phase, as shown in Table 2. Evaluation scheme: We report the AUC of the three-dimensional position error.

[0076] Table 2 Evaluation in outdoor pose estimation

[0077]

[0078] like Figure 6 As shown in FIG, the diagram shows the visualization result of outdoor image matching. The present invention can produce high-quality matching and achieve good matching effect.

[0079] Although the embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents, and all should be included within the scope of protection of the present invention.

Claims

1. A real-time image matching method based on a hybrid MLP structure, characterized in that: The following steps are involved: Step S1: Acquire image data and perform feature extraction on the image data through local feature extraction to obtain a feature map; Step S2: Construct an image matching model based on a hybrid MLP architecture to divide the obtained feature map into windows according to a preset size, and pass it through the MLP module to fuse information from different spatial positions, and output the features of the spatial mixing layer and the channel mixing layer. The specific steps include: Build an image matching model based on a hybrid MLP architecture; The Mixer-WMLP module of the hybrid MLP architecture divides the tensor between windows of preset sizes through window partitioning, and obtains the feature map at the coarse level as , which is divided into (num_windows,window_size,window_size,C), is the number of feature maps; After two MLP modules, the information from different spatial positions is integrated to form spatial mixing layer and channel mixing layer; The specific steps in the Mixer-WMLP module include: The feature map is evenly divided into non-overlapping windows through the Mixer-WMLP module, and each window is expanded as a token for token information exchange between spatial locations; The MLP module is used to map columns and rows in sequence to integrate the information of the spatial domain and channel domain, and the input feature map is defined as , the window size is , then The calculation complexity is: ； ； in, The first MLP node expansion factor, represents the feature dimension; Step S3: input the obtained features into the matching model, use the differentiable matching layer of the matching model to match the features into a confidence matrix, and select matching items in the confidence matrix according to the confidence threshold and the mutual nearest neighbor criterion to obtain a coarse level matching result; Step S4: Input the coarse level matching result into the Mixer-WMLP module for several transformations to finally obtain a match with sub-pixel accuracy, and obtain the final detection matching result through correction and softmax.

2. The real-time image matching method based on the hybrid MLP structure according to claim 1, characterized in that: The specific steps in step S3 include: The obtained features are input into the matching model, and the converted features are matched into a confidence matrix using a differentiable matching layer. ; Then, according to the confidence threshold and the mutual nearest neighbor criterion, The matching items in get the coarse level matching results .

3. The real-time image matching method based on a hybrid MLP structure according to claim 1, characterized in that: The specific steps in step S4 include: For each pair of coarse level matching results obtained, first locate the position of the coarse level matching result , and then divided into The local window; The features cropped in each window are processed by the Mixer-WMLP module. Transformation, get Center vector and Correlate with each other to generate a heatmap; Then calculate the expectation on its probability distribution to obtain the final match with sub-pixel accuracy , the final detection and matching results are obtained through correction and softmax .

Images