Multi-scale image matching method and system

Abstract

The invention relates to a multi-scale image matching method and system. The method comprises the following steps: S1, acquiring a data set for image matching training; s2, an image matching model is constructed, the image matching model comprises a visual space cue extraction module, a visual space fusion module and a visual space potential image transformation module, the visual space cue extraction module extracts visual cues and space cues from the input image pair, the visual space fusion module fuses the extracted visual cues and space cues into the same space, and the visual space potential image transformation module carries out image matching on the visual cues and the space cues; the visual space potential graph transformation module extracts local features from visual space fusion features by constructing a feature graph and fuses the local features and global features, so that multi-scale context information corresponding to the visual space is effectively captured; training the image matching model through the data set; and S3, carrying out image matching on the to-be-matched image pair through the trained image matching model. According to the method and the system, the image matching speed and accuracy can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image matching, and in particular to a multi-scale image matching method and system. Background Art

[0002] Image matching is a technology for aligning different images. It aims to align and fuse multiple images from different times, sensors, perspectives or modes. The goal of matching is to align the corresponding positions of the same object in all images so that more effective analysis and processing can be performed. It occupies an important position in the field of computer vision and pattern recognition. Traditional methods often extract image feature points through SIFT and SURF and use algorithms such as RANSAC for image feature matching, but these methods often perform poorly when there is obvious noise and image deformation, and the computational complexity is high. With the development of deep learning, deep learning-based methods such as SuperGlue have made significant progress in image feature matching. These methods improve the accuracy and robustness of matching through graph neural networks and attention mechanisms.

[0003] Image feature matching has a wide range of uses in practical applications. In simultaneous localization and mapping (SLAM), by matching image features between consecutive frames, it is not only possible to build and update a three-dimensional map of the environment in real time, but also to achieve real-time perception and response to the dynamic environment. In terms of robot navigation, image feature matching can help the robot accurately understand the structure and characteristics of the surrounding environment, so as to formulate accurate path planning and obstacle avoidance strategies. For unmanned driving technology, image feature matching technology is the key to achieving high-precision positioning and real-time environmental perception. By analyzing and processing dual-view images, unmanned driving systems can quickly and accurately identify roads, traffic signs and obstacles, thereby ensuring driving safety and efficiency.

[0004] Some models aim to achieve robust visual localization by learning geometric consistency. This method is based on graph neural networks and significantly improves the reliability and accuracy of image feature matching by utilizing geometric information in images. For example, the key to OANet is its ability to accurately identify and match feature points in complex scenes, thereby achieving high-precision pose estimation.

[0005] On the other hand, ACNet and MS 2 DG-Net is a method based on attention mechanism and multi-scale feature extraction. These two methods introduce attention mechanism to improve the accuracy of visual position recognition by focusing on key areas in the image. ACNet uses attention mechanism to enhance the information aggregation ability in the feature extraction process, so that the network can more effectively identify and match important features in the image. In this way, ACNet not only improves the positioning accuracy, but also improves the efficiency of the algorithm when processing large-scale data. MS2 DG-Net was proposed by Dai et al. It is a multi-scale and multi-depth geometric network that aims to achieve robust visual positioning through multi-level feature extraction and geometric information fusion. 2 DG-Net effectively captures rich geometric information in the image by extracting features at different scales and depths, thereby enhancing the accuracy and robustness of image feature matching. This method is particularly suitable for complex and dynamic scenes and can achieve high-precision pose estimation in a variety of environments. ACNet's poor performance in some low-level network layers, including its asymmetric convolution, may lead to a slight performance degradation. In addition, although there is no additional computational burden at inference time, the performance may be affected due to different gradient flows and random initialization during training. MS 2 The main disadvantage of DG-Net is the high computational cost caused by its complex graph construction and dynamic update mechanism, which makes it less efficient in real-time applications. At the same time, on some datasets (especially those with fewer outliers), its performance may not be significantly better than that of simple models. Summary of the invention

[0006] The purpose of the present invention is to provide a multi-scale image matching method and system, which can improve the speed and accuracy of image matching.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is: a multi-scale image matching method, comprising the following steps: S1. Obtain a data set for image matching training; S2. Construct an image matching model, wherein the image matching model includes a visual space clue extraction module, a visual space fusion module, and a visual space latent map transformation module. The visual space clue extraction module extracts visual clues and spatial clues from the input image pair to reduce the impact of occlusion, large viewing angle change, and illumination change between the two images. The visual space fusion module fuses the extracted visual clues and spatial clues into the same space and projects them together into the original space. The visual space latent map transformation module extracts local features from the visual space fusion features by constructing a feature map and fuses the local features with the global features, thereby effectively capturing the multi-scale context information corresponding to the visual space; train the image matching model using the data set; S3. Perform image matching on the image pairs to be matched using the trained image matching model.

[0008] Furthermore, the data sets obtained for image matching training include the YFCC100M data set and the SUN3D data set.

[0009] Furthermore, the visual space clue extraction module includes a convolutional neural network, a feature correspondence module, a cross attention layer and a multi-layer perceptron MLP. The visual space clue extraction module extracts the image from the input image pair {I A , I B} to extract high-dimensional local features {F A , F B}, then the extracted high-dimensional local features are flattened into one-dimensional vectors through the feature correspondence module and passed to the cross attention layer to generate the initial visual cues of the scene; then, the initial visual cues are embedded into the MLP to obtain the visual cues F for fusion v In addition, the input image pair is preprocessed using a feature detector, and then the initial corresponding set I is established through the nearest neighbor matching strategy C ; The initial corresponding set I C is embedded into another MLP to extract deep features and serve as spatial cues F s .

[0010] Furthermore, the visual space fusion module models the interaction between visual cues and spatial cues through the Transformer layer, and combines difference pooling and difference de-pooling operations to generate fused visual space features; the specific implementation method is: Input visual clues F v and spatial clues F s ; Reduce the dimension of spatial clues and extract compressed spatial features through difference pooling: S pooled = DiffPool(F s ) Among them, S pooled is the spatial feature after dimensionality reduction, DiffPool() represents the difference pooling operation; Concatenate the visual cues and the reduced-dimensional spatial features to form a joint feature VS: VS = Concat(S pooled , F v ) Among them, Concat() means concatenation in the feature dimension, and VS is the joint feature after concatenation; Then, the Transformer layer is used to globally model the joint features of visual and spatial cues to generate the interacted features: VS' = Transformer(VS) Among them, Transformer () represents the Transformer function, which uses a multi-head self-attention mechanism to model the relationship between feature points, and VS' is the feature after interaction; Then, the interactive feature VS' is decomposed into updated visual cues and spatial cues: VS' = {S' updated , V' updated} Among them, S' updated represents the updated spatial clue, V' updated Visual cues to indicate updates; Merge the initial clue with the updated clue: S updated = F s + S' updated V updated = F v + V' updated The ResNet encoder is used to extract features from the updated fused spatial and visual cues: S encoded = ResNetBlock(S updated ) V encoded = ResNetBlock(V updated ) Among them, S encoded is the encoded spatial feature, V encoded is the encoded visual feature; Add the encoded spatial features and visual features to generate joint features: VS encoded = S encoded + V encoded Map the joint features back to the original space through unpooling: F VS = DiffUnpool(S cues , VS encoded ) Among them, DiffPool() represents the difference pooling operation, which is used to map the joint features back to the original spatial resolution; F VS It is the visual spatial fusion feature finally output by the visual spatial fusion module.

[0011] Furthermore, the visual space latent graph transformation module includes a graph construction module, a graph boundary node extraction module BDB, an aggregation module, an encoding module, a multi-head self-attention mechanism module MHSA and an FFN network; wherein the graph construction module constructs a feature graph by calculating the relationship between feature points and captures the local geometric relationship between nodes; the BDB module extracts multi-scale context information by fusing local and global features; the aggregation module fuses the local information and global information in the features to enhance the overall feature expression; the encoding module performs deep encoding on the input features and extracts high-level semantic features while retaining important input information; the MHSA module is used to capture the complex relationship between global context information and feature points; the FFN network is used to further extract and nonlinearly transform the channel dimension information of each feature point.

[0012] Furthermore, the implementation method of the graph construction module is: Construct a KNN-based graph based on the Euclidean distance between feature points in the joint visual space; First, determine the K neighbors of the feature point: for each feature point in the joint visual space f i , select the K feature points with the closest Euclidean distance as its nearest neighbor feature points, and its nearest neighbor feature set is expressed as follows: V i = { f i1 ,f i2 ,…,f ik} in, V i Representing feature points f i The nearest neighbor feature set of f ik Representing feature points f i The kth nearest neighbor feature point; Then calculate the edges between each feature point and its neighbors: Through the edge e ij Connect feature points f i Its nearest neighbor feature point f ij , and define the content of the edge by calculating the difference between two feature points: e ij = [ f i , f ij -f i ] in, f ij It is a feature point f i The jth nearest neighbor feature point of f ij - f i is the difference vector between two feature points; Through the obtained edges and feature points, a global graph G is constructed in , expressed as: G in = {G 1 , ..., G i , ..., G N} Among them, G 1 , ..., G i , ..., G N They are the local feature maps corresponding to N feature points, which contain the connection between each feature point and its nearest neighbor feature point.

[0013] Furthermore, the implementation method of the BDB module is: The BDB module includes a boundary enhancement module, a KNN projection branch, a local attention branch, a global attention branch and a cross-layer fusion module; The boundary enhancement module is used for global feature extraction, and its input is G in , the output is x 1 ; The processing process of the boundary enhancement module is expressed as:

[0014] in, Indicates that 1×1 convolution is used to map the number of channels from C to C, BN( ) indicates batch normalization, ReLU( ) indicates the ReLU activation function, and x 1 Represents the global characteristics of the output; The KNN projection branch is used for local feature extraction, and its input is x 1 , the output is x local ; The processing process of the KNN projection branch is expressed as:

[0015] in, Indicates that 1×1 convolution is used to map the number of channels from C to C / 2, BN( ) indicates batch normalization, ReLU( ) indicates the ReLU activation function, and x local Represents the local features of the output; The local attention branch is used for local graph convolution feature extraction, and its input is x 1 , the output is x localatt ; The processing process of the local attention branch is expressed as: x localatt = DGCNN Layer (G(x 1 )) Among them, G( ) represents the construction of local graph features and the extraction of local structural information; DGCNN Layer ( ) indicates that the DGCNN layer is used for graph convolution operation; The global attention branch is used for global context feature extraction, and its input is x 1 , the output is x globalatt ; The processing process of the global attention branch is expressed as:

[0016] First, adaptive average pooling is performed to compress the spatial size to 1×1, and then 1×1 convolution is used to map the number of channels from C to C / r; then batch normalization is performed, and then the ReLU activation function is used, and then 1×1 convolution is used to map the number of channels from C / r back to C / 2. After batch normalization, the output is x globalatt ; The cross-layer fusion module is used for feature fusion, which combines the three features x globalatt 、x localatt and x local Perform element-by-element addition operation and output the fused feature map x lg , expressed as: x lg = x globalatt + x localatt + x local Then for the fused feature map x lg Apply Sigmoid, expressed as: G out = σ·x lg Among them, σ represents the Sigmoid function, G out It is the output of the BDB module.

[0017] Furthermore, the implementation method of the aggregation module, the encoding module, the MHSA module and the FFN network is: The aggregation module performs the feature G output by the BDB module. out Perform neighborhood aggregation operations to capture more geometric and semantic information: F Agg = Agg(G out ) Among them, F Agg Represents the features output by the aggregation module, and Agg( ) represents the neighborhood aggregation operation; the features aggregated by the aggregation module contain the relationship between the current feature and its neighbors; The encoding module outputs the feature F of the aggregation module. Agg Perform deep encoding to extract high-level semantic features and maintain the integrity of the input information: F Encoded = Encoder(F Agg ) Among them, F Encoded Represents the features output by the encoding module, and Encoder( ) represents the encoding operation; the encoder consists of multiple ResNet blocks, each of which performs a convolution operation to extract more feature information; The MHSA module introduces a length similarity matrix to calculate the geometric relationship between feature points to ensure the accuracy of feature matching: F MHSA = MHSA(F Encoded , M ls ) Among them, F MHSA represents the output features of the MHSA module, MHSA( ) represents the multi-head self-attention mechanism operation, M ls Represents a length similarity matrix; the length similarity matrix M ls The calculation method is: For the input image pair {I A , I B} Use feature detector for preprocessing, and then establish the initial corresponding set I through the nearest neighbor matching strategy C ={c 1 , c 2 , ..., c i , ..., c j ,..., c N}, where c i 、c j They represent the initial corresponding set I C The i-th and j-th feature point pairs in N represent the initial corresponding set I C The number of feature point pairs in c i =(p i A , p i B ), c i =(p j A , p j B ), where pi A 、p i B Represents the input image I A , input image I B The length similarity matrix M ls Element m i,j Calculated according to the following formula:

[0018] Among them, || || represents norm calculation, | | represents taking the absolute value; The features output by the MHSA module are divided into two paths, one path is input into the FFN network, and the other path is spliced ​​with the features output by the FFN network to obtain the final output features; The FFN network receives the features output by the MHSA module and performs a series of transformations to further extract features:

[0019] Among them, Linear 1 is the first linear transformation layer, which maps the features output by the MHSA module to a higher dimensional space to enhance the representation capability; ReLU ( ) is the ReLU activation function, which is used for Linear 1 The output features are processed to enhance the nonlinear characteristics and get notification after activation; Linear 2 It is the second linear transformation layer, which is used to map the activated features back to the final output space.

[0020] Furthermore, a hybrid loss function is adopted To supervise the training process of the image matching model:

[0021] in, represents classification loss; α is a hyperparameter that balances classification loss and essential matrix loss; It represents the essential matrix loss, which is used to measure the difference between the predicted essential matrix and the true essential matrix. After feature extraction through the image matching model, the predicted essential matrix is ​​calculated based on the matched feature point pairs, and then the essential matrix loss is calculated; Classification Loss It is expressed as:

[0022] Where H( ) represents the binary cross entropy loss function; t represents the relevant weight of the tth iteration; y t represents a weakly supervised label; ωt is the adaptive temperature vector, ⊙ represents the Hadamard product, and λ is the number of iterations.

[0023] The present invention also provides a multi-scale image matching system, comprising a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above method can be implemented.

[0024] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides a multi-scale image matching method and system. The method and system can extract and fuse visual cues and spatial cues for input image pairs by constructing an image matching model combining visual space clue extraction, visual space fusion module and visual space latent map transformation module. On this basis, local features are further extracted from the visual space fusion features and the local features are fused with the global features, thereby effectively capturing the multi-scale context information corresponding to the visual space, making full use of the local geometric information and global context information in the image, solving the matching error problem of traditional methods when processing complex scenes, and improving the robustness and accuracy of the method in complex environments; in addition, through local and global cross-layer multi-branch fusion, the computational complexity is effectively reduced, making the model more efficient when processing large-scale image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a schematic diagram of an image matching model in an embodiment of the present invention; Figure 2 is a specific implementation structure diagram of the BDB module in an embodiment of the present invention; Figure 3 It is a visualization comparison diagram of an embodiment of the present invention in an outdoor scene of YFCC100M. DETAILED DESCRIPTION

[0026] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0027] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0028] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0029] This embodiment provides a multi-scale image matching method, comprising the following steps: S1. Obtain a data set for image matching training; S2, build Figure 1 The image matching model shown in the figure includes a visual-spatial clue extraction module, a visual-spatial fusion module and a visual-spatial latent graph transformation module. The visual-spatial clue extraction module extracts visual clues and spatial clues from an input image pair to reduce the impact of occlusion, large viewing angle change and illumination change between the two images. The visual-spatial fusion module fuses the extracted visual clues and spatial clues into the same space and projects them into the original space together. The visual-spatial latent graph transformation module (VSLG Former, Visual-Spatial Latent Graph Transformer) extracts local features from visual-spatial fusion features by constructing a feature map and fuses the local features with the global features, thereby effectively capturing multi-scale contextual information corresponding to the visual space. The image matching model is trained using a data set. S3. Perform image matching on the image pairs to be matched using the trained image matching model.

[0030] In this embodiment, the data sets obtained for image matching training include the YFCC100M data set and the SUN3D data set. The scenes included in the data sets are divided into two categories: a known scene group and an unknown scene group. In this embodiment, evaluation is performed on both unknown scenes and known scenes.

[0031] The visual space clue extraction module includes a convolutional neural network, a feature correspondence module (Correspondence Feature Module), a cross attention layer and a multi-layer perceptron MLP. The visual space clue extraction module extracts the image from the input image pair {I A , I B} to extract high-dimensional local features {F A , F B}, then the extracted high-dimensional local features are flattened into one-dimensional vectors through the feature correspondence module and passed to the cross attention layer to generate the initial visual cues of the scene; then, the initial visual cues are embedded into the MLP to obtain the visual cues F for fusion v In addition, the input image pair is preprocessed using feature detectors (such as SIFT and SuperPoint), and then the initial corresponding set I is established through the nearest neighbor matching strategy C ; The initial corresponding set I C is embedded into another MLP to extract deep features and serve as spatial cues F sIn this embodiment, the convolutional neural network adopts the ResNet34 convolutional neural network architecture.

[0032] The visual space fusion module models the interaction between visual cues and spatial cues through the Transformer layer, and combines difference pooling and difference de-pooling operations to generate fused visual space features. The specific implementation method of the visual space fusion module is as follows.

[0033] Input Visual Cues and spatial clues , where B is the batch size, D v is the dimension of visual features, D s is the dimension of the spatial feature.

[0034] Reduce the dimensionality of spatial clues and extract compressed spatial features through difference pooling: S pooled = DiffPool(F s ) Among them, S pooled is the spatial feature after dimensionality reduction, and DiffPool() represents the difference pooling operation.

[0035] Concatenate the visual cues and the reduced-dimensional spatial features to form a joint feature VS: VS = Concat(S pooled , F v ) Among them, Concat() means concatenation on the feature dimension. is the joint feature after splicing.

[0036] Then, the Transformer layer is used to globally model the joint features of visual and spatial cues to generate the interacted features: VS' = Transformer(VS) Among them, Transformer () represents the Transformer function, which uses a multi-head self-attention mechanism to model the relationship between feature points. are the features after interaction.

[0037] Then, the interactive feature VS' is decomposed into updated visual cues and spatial cues: VS' = {S' updated , V' updated} Among them, S' updated represents the updated spatial clue, V' updated Visual cues that indicate updates.

[0038] Merge the initial clue with the updated clue: S updated = F s + S' updated V updated = F v + V' updated The ResNet encoder is used to extract features from the updated fused spatial and visual cues: S encoded = ResNetBlock(S updated ) V encoded = ResNetBlock(V updated ) Among them, S encoded is the encoded spatial feature, V encoded is the encoded visual feature.

[0039] Add the encoded spatial features and visual features to generate joint features: VS encoded = S encoded + V encoded Map the joint features back to the original space through unpooling: F VS = DiffUnpool(S cues , VS encoded ) Among them, DiffPool() represents the difference pooling operation, which is used to map the joint features back to the original spatial resolution; is the visual space fusion feature finally output by the visual space fusion module, and D is the dimension after fusion.

[0040] The visual space potential graph transformation module includes a graph construction module, a graph boundary node extraction module BDB (Boundary Detection and Extraction Block), an aggregation module, an encoding module, a multi-head self-attention mechanism (MHSA) module and an FFN network. Among them, the graph construction module constructs a feature graph by calculating the relationship between feature points to capture the local geometric relationship between nodes. The BDB module extracts multi-scale context information by fusing local and global features. The aggregation module fuses local information and global information in the features to enhance the overall feature expression. The encoding module performs deep encoding of the input features, extracts high-level semantic features, and retains important input information. The MHSA module is used to capture the complex relationship between global context information and feature points. The FFN network is used to further extract and nonlinearly transform the channel dimension information of each feature point.

[0041] The specific implementation method of the graph construction module is as follows.

[0042] A KNN-based graph is constructed based on the Euclidean distance between feature points in the joint visual space.

[0043] a) First determine the K neighbors of the feature point: For each feature point in the joint visual space f i , select the K feature points with the closest Euclidean distance as its nearest neighbor feature points, and its nearest neighbor feature set is expressed as follows: V i = { f i1 ,f i2 ,…,f ik} in, V i Representing feature points f i The nearest neighbor feature set of f ik Representing feature points f i The kth nearest neighbor feature point.

[0044] b) Then calculate the edges between each feature point and its neighbors: Through the edge e ij Connect feature points f i Its nearest neighbor feature point f ij , and define the content of the edge by calculating the difference between two feature points: e ij = [ f i , f ij - f i ] in, f ij It is a feature point f i The jth nearest neighbor feature point of f ij - f i is the difference vector between two feature points.

[0045] Through the obtained edges and feature points, a global graph G is constructed in , expressed as: G in = {G 1 , ..., G i , ..., G N} Among them, G 1 , ..., G i , ..., G N They are the local feature maps corresponding to N feature points, which contain the connection between each feature point and its nearest neighbor feature point.

[0046] like Figure 2 As shown, the BDB module includes a boundary enhancement module (BE, Boundary Extraction Grid), a KNN projection branch, a local attention branch, a global attention branch and a cross-layer fusion module. The specific implementation method of the BDB module is as follows.

[0047] The boundary enhancement module is used for global feature extraction, and its input is G in ∈R B×C×H×W , the output is x 1 ∈R B ×C×H×W , where B is the batch size, C is the number of channels, H and W are the height and width of the image; the processing process of the boundary enhancement module is expressed as:

[0048] in, Indicates that 1×1 convolution is used to map the number of channels from C to C, BN( ) indicates batch normalization, ReLU( ) indicates the ReLU activation function, and x 1 Represents the global characteristics of the output.

[0049] The KNN projection branch is used for local feature extraction, and its input is x 1 , the output is x local ; The processing process of the KNN projection branch is expressed as:

[0050] in, Indicates that 1×1 convolution is used to map the number of channels from C to C / 2, BN( ) indicates batch normalization, ReLU( ) indicates the ReLU activation function, and x local Represents the local features of the output.

[0051] The local attention branch is used for local graph convolution feature extraction, and its input is x 1 , the output is x localatt ∈R B ×C / 2×H×W ; The processing process of the local attention branch is expressed as: x localatt = DGCNN Layer (G(x 1 )) Among them, G( ) means constructing local graph features through the get_graph_feature function to extract local structural information; DGCNN Layer ( ) indicates that the DGCNN layer is used for graph convolution operation.

[0052] The global attention branch is used for global context feature extraction, and its input is x 1 , the output is x globalatt ∈R B ×C / 2×1×1 ; The processing process of the global attention branch is expressed as:

[0053] First, adaptive average pooling is performed to compress the spatial size to 1×1, and then 1×1 convolution is used to map the number of channels from C to C / r; then batch normalization is performed, and then the ReLU activation function is used, and then 1×1 convolution is used to map the number of channels from C / r back to C / 2. After batch normalization, the output is x globalatt .

[0054] The cross-layer fusion module is used for feature fusion, which combines the three features x globalatt 、x localatt and x local Perform element-by-element addition operation and output the fused feature map x lg ∈R B×C / 2×1×1 , expressed as: x lg = x globalatt+ x localatt + x local Then for the fused feature map x lg Apply Sigmoid, expressed as: G out = σ·x lg Among them, σ represents the Sigmoid function, G out It is the output of the BDB module.

[0055] The specific implementation methods of the aggregation module, encoding module, MHSA module and FFN network are respectively described as follows.

[0056] The aggregation module performs the feature G output by the BDB module. out Perform neighborhood aggregation operations to capture more geometric and semantic information: F Agg = Agg(G out ) Among them, F Agg Represents the features output by the aggregation module, and Agg( ) represents the neighborhood aggregation operation; the features aggregated by the aggregation module contain the relationship between the current feature and its neighbors.

[0057] The encoding module outputs the feature F of the aggregation module. Agg Perform deep encoding to extract high-level semantic features and maintain the integrity of the input information: F Encoded = Encoder(F Agg ) Among them, F Encoded Represents the features of the encoding module output, and Encoder( ) represents the encoding operation; the encoder consists of multiple ResNet blocks, each of which performs a convolution operation to extract more feature information.

[0058] This embodiment designs a multi-head self-attention mechanism (MHSA) module for local and global integration to each correspondence. The MHSA module introduces a length similarity matrix to calculate the geometric relationship between feature points to ensure the accuracy of feature matching: F MHSA = MHSA(F Encoded , M ls ) Among them, F MHSA represents the output features of the MHSA module, MHSA( ) represents the multi-head self-attention mechanism operation, M ls Represents the length similarity matrix.

[0059] The length similarity matrix Mls The calculation method is: For the input image pair {I A , I B} Use feature detectors (such as SIFT and SuperPoint) for preprocessing, and then use the nearest neighbor matching strategy to establish the initial corresponding set I C ={c 1 , c 2 , ..., c i , ..., c j ,..., c N}, where c i 、c j They represent the initial corresponding set I C The i-th and j-th feature point pairs in N represent the initial corresponding set I C The number of feature point pairs in c i =(p i A ,p i B ), c i =(p j A , p j B ), where p i A 、p i B Represents the input image I A , input image I B The length similarity matrix M ls Element m i,j Calculated according to the following formula:

[0060] Among them, || || represents norm calculation, that is, calculating the length of a vector or the Euclidean distance between two points, and | | represents taking the absolute value to ensure that the calculated value is non-negative.

[0061] The features output by the MHSA module are divided into two paths, one path is input into the FFN network, and the other path is concatenated with the features output by the FFN network to obtain the final output features.

[0062] The FFN network receives the features output by the MHSA module and performs a series of transformations to further extract features:

[0063] Among them, Linear 1is the first linear transformation layer, which maps the features output by the MHSA module to a higher dimensional space to enhance the representation capability; ReLU ( ) is the ReLU activation function, which is used for Linear 1 The output features are processed to enhance the nonlinear characteristics and get notification after activation; Linear 2 It is the second linear transformation layer, which is used to map the activated features back to the final output space.

[0064] In this embodiment, a hybrid loss function is used To supervise the training process of the image matching model:

[0065] in, represents classification loss; α is a hyperparameter that balances classification loss and essential matrix loss; It represents the essential matrix loss, which is used to measure the difference between the predicted essential matrix and the true essential matrix. After feature extraction through the image matching model, the predicted essential matrix is ​​calculated based on the matched feature point pairs, and then the essential matrix loss is calculated.

[0066] Classification Loss It is expressed as:

[0067] Where H( ) represents the binary cross entropy loss function; t represents the relevant weight of the tth iteration; y t Indicates weakly supervised labels, at the pole distance threshold of 10 −4 The next choice is positive; ω t is the adaptive temperature vector, ⊙ represents the Hadamard product, and λ is the number of iterations.

[0068] In this embodiment, this method is compared with existing deep learning methods. This embodiment selects multiple benchmark models, including the traditional RANSAC algorithm, graph-based methods such as MS 2DG-Net, and a series of deep learning-based models such as PointNet++, PointCN, and MSA-Net. Our goal is to evaluate the performance of this method compared with existing methods in different scenarios, especially the performance in known and unknown scenarios. Through testing on the YFCC100M and SUN3D datasets, it can be seen that this method outperforms existing methods in terms of the number of parameters and accuracy. The comparison results are shown in Table 1. In the YFCC100M dataset, this method is significantly higher than other methods in both known and unknown scenarios. In contrast, the traditional RANSAC algorithm is not as good as this model in terms of the number of parameters and accuracy. For example, the accuracy of RANSAC in the YFCC100M dataset is only 9.07%. Graph-based methods, such as MS 2 DG-Net, although it has a reduced number of parameters, still fails to surpass it in terms of accuracy. This method shows strong performance in both known and unknown scenarios. Specifically, this method achieves 47.95% and 61.85% accuracy in YFCC100M known and unknown scenarios, respectively. This method also shows sub-optimal performance on the SUN3D dataset. The efficient performance of this method is due to the aggregation strategies in the global and local fusion modules. These strategies not only improve the efficiency of the sampling process, but also improve the computational efficiency by identifying multi-branch connections in the graph, and effectively mitigate the impact of pseudo outliers.

[0069] Table 1 Performance comparison between this method and existing methods

[0070] This example also gives the area under the cumulative error curve (AUC) at the maximum rotation and displacement errors, and sets thresholds of 5°, 10°, and 20°. We choose the nearest neighbor (NN) method as the baseline method, which retains all hypothesized correspondences. Then, this method is compared with several classic outlier removal methods. In addition, learning-based methods that directly classify correspondences are also included, as well as other methods that consider consistency and smoothness. During the evaluation process, we tested different robust model estimators, including the weighted eight-point algorithm and the RANSAC algorithm, which are used to calculate the pose relationship of the predicted inliers. It should be noted that the weighted eight-point algorithm requires the calculation of the inlier probability for each correspondence, so it is not suitable for the classic method. All results are shown in Table 2. Considering that the accuracy of camera pose estimation is affected by the quality of the initial correspondence, when RANSAC post-processing is performed under the same conditions, the accuracy of this method is better than all other methods in the same AUC parameter comparison.

[0071] Table 2 Results of relative pose estimation using the weighted eight-point algorithm / RANSAC (AUC scores at 5°, 10°, and 20° thresholds in the YFCC100M unknown scene)

[0072] Figure 3 This is a visualization comparison diagram of the YFCC100M outdoor scene in this embodiment. Figure 3 In the figure, (a), (b), and (c) are the results of OANet, VSformer, and our method, respectively. If the correspondence represents a true positive, it is drawn in green, and if it represents a false positive, it is drawn in red.

[0073] like Figure 3 As shown, the visualization of the proposed method achieves the best performance in this comparison. This is because the advantages of the global and local strategies of the proposed method make the proposed method more effective in processing details.

[0074] This embodiment also provides a multi-scale image matching system, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the above method can be implemented.

[0075] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0076] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0077] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0078] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0079] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A multi-scale image matching method, characterized in that: The following steps are involved: S1. Obtain a data set for image matching training; S2. Construct an image matching model, wherein the image matching model includes a visual space clue extraction module, a visual space fusion module, and a visual space latent map transformation module. The visual space clue extraction module extracts visual clues and spatial clues from the input image pair to reduce the impact of occlusion, large viewing angle change, and illumination change between the two images. The visual space fusion module fuses the extracted visual clues and spatial clues into the same space and projects them together into the original space. The visual space latent map transformation module extracts local features from the visual space fusion features by constructing a feature map and fuses the local features with the global features, thereby effectively capturing the multi-scale context information corresponding to the visual space; train the image matching model using the data set; S3, performing image matching on the image pairs to be matched using the trained image matching model; The visual space potential graph transformation module includes a graph construction module, a graph boundary node extraction module BDB, an aggregation module, an encoding module, a multi-head self-attention mechanism module MHSA and an FFN network; wherein the graph construction module constructs a feature graph by calculating the relationship between feature points and captures the local geometric relationship between nodes; the BDB module extracts multi-scale context information by fusing local and global features; the aggregation module fuses local information and global information in the features to enhance the overall feature expression; the encoding module performs deep encoding on the input features, extracts high-level semantic features, and retains important input information; the MHSA module is used to capture the complex relationship between global context information and feature points; the FFN network is used to further extract and nonlinearly transform the channel dimension information of each feature point; The implementation method of the BDB module is: The BDB module includes a boundary enhancement module, a KNN projection branch, a local attention branch, a global attention branch and a cross-layer fusion module; The boundary enhancement module is used for global feature extraction, and its input is G in , the output is x1; the processing process of the boundary enhancement module is expressed as: in, Indicates that 1×1 convolution is used to map the number of channels from C to C, BN( ) indicates batch normalization, ReLU( ) indicates the ReLU activation function, and x1 indicates the global feature of the output; The KNN projection branch is used for local feature extraction, with its input being x1 and its output being x local ; The processing process of the KNN projection branch is expressed as: in, Indicates that 1×1 convolution is used to map the number of channels from C to C / 2, BN( ) indicates batch normalization, ReLU( ) indicates the ReLU activation function, and x local Represents the local features of the output; The local attention branch is used for local graph convolution feature extraction, with input x1 and output x localatt ; The processing process of the local attention branch is expressed as: x localatt = DGCNN Layer (G(x1)) Among them, G( ) represents the construction of local graph features and the extraction of local structural information; DGCNN Layer ( ) indicates that the DGCNN layer is used for graph convolution operation; The global attention branch is used for global context feature extraction, with input x1 and output x globalatt ; The processing process of the global attention branch is expressed as: First, adaptive average pooling is performed to compress the spatial size to 1×1, and then 1×1 convolution is used to map the number of channels from C to C / r; then batch normalization is performed, and then the ReLU activation function is used, and then 1×1 convolution is used to map the number of channels from C / r back to C / 2. After batch normalization, the output is x globalatt ; The cross-layer fusion module is used for feature fusion, which combines the three features x globalatt 、x localatt and x local Perform element-by-element addition operation and output the fused feature map x lg , expressed as: x lg = x globalatt + x localatt + x local Then for the fused feature map x lg Apply Sigmoid, expressed as: G out = σ·x lg Among them, σ represents the Sigmoid function, G out It is the output of the BDB module.

2. A multi-scale image matching method according to claim 1, characterized in that: The datasets obtained for image matching training include the YFCC100M dataset and the SUN3D dataset.

3. The multi-scale image matching method according to claim 1, characterized in that: The visual space clue extraction module includes a convolutional neural network, a feature correspondence module, a cross attention layer and a multi-layer perceptron MLP. The visual space clue extraction module extracts the image from the input image pair {I A , I B } to extract high-dimensional local features {F A , F B }, then the extracted high-dimensional local features are flattened into one-dimensional vectors through the feature correspondence module and passed to the cross attention layer to generate the initial visual cues of the scene; then, the initial visual cues are embedded into the MLP to obtain the visual cues F for fusion v In addition, the input image pair is preprocessed using a feature detector, and then the initial corresponding set I is established through the nearest neighbor matching strategy C ; The initial corresponding set I C is embedded into another MLP to extract deep features and serve as spatial cues F s .

4. The multi-scale image matching method according to claim 1, characterized in that: The visual space fusion module models the interaction between visual cues and spatial cues through the Transformer layer, and combines difference pooling and difference de-pooling operations to generate fused visual space features; its specific implementation method is: Input visual clues F v and spatial clues F s ; Reduce the dimension of spatial clues and extract compressed spatial features through difference pooling: S pooled = DiffPool(F s ) Among them, S pooled is the spatial feature after dimensionality reduction, DiffPool() represents the difference pooling operation; Concatenate the visual cues and the reduced-dimensional spatial features to form a joint feature VS: VS = Concat(S pooled , F v ) Among them, Concat() means concatenation in the feature dimension, and VS is the joint feature after concatenation; Then, the Transformer layer is used to globally model the joint features of visual and spatial cues to generate the interacted features: VS' = Transformer(VS) Among them, Transformer () represents the Transformer function, which uses a multi-head self-attention mechanism to model the relationship between feature points, and VS' is the feature after interaction; Then, the interactive feature VS' is decomposed into updated visual cues and spatial cues: VS' = {S' updated , V' updated } Among them, S' updated represents the updated spatial clue, V' updated Visual cues to indicate updates; Merge the initial clue with the updated clue: S updated = F s + S' updated V updated = F v + V' updated The ResNet encoder is used to extract features from the updated fused spatial and visual cues: S encoded = ResNetBlock(S updated ) In encoded = ResNetBlock(V updated ) Among them, S encoded is the encoded spatial feature, V encoded is the encoded visual feature; Add the encoded spatial features and visual features to generate joint features: VS encoded = S encoded + V encoded Map the joint features back to the original space through unpooling: F VS = DiffUnpool(S cues , VS encoded ) Among them, DiffPool() represents the difference pooling operation, which is used to map the joint features back to the original spatial resolution; F VS It is the visual spatial fusion feature finally output by the visual spatial fusion module.

5. The multi-scale image matching method according to claim 1, characterized in that: The implementation method of the graph construction module is: Construct a KNN-based graph based on the Euclidean distance between feature points in the joint visual space; First, determine the K neighbors of the feature point: for each feature point in the joint visual space f i , select the K feature points with the closest Euclidean distance as its nearest neighbor feature points, and its nearest neighbor feature set is expressed as follows: V i = { f i1 ,f i2 ,…,f ik} in, V i Representing feature points f i The nearest neighbor feature set of f ik Representing feature points f i The kth nearest neighbor feature point; Then calculate the edges between each feature point and its neighbors: Through the edge e ij Connect feature points f i Its nearest neighbor feature point f ij , and define the content of the edge by calculating the difference between two feature points: e ij = [ f i , f ij － f i ] in, f ij It is a feature point f i The jth nearest neighbor feature point of f ij - f i is the difference vector between two feature points; Through the obtained edges and feature points, a global graph G is constructed in , expressed as: G in = {G1, ..., G i , ..., G N } Among them, G1, ..., G i , ..., G N They are the local feature maps corresponding to N feature points, which contain the connection between each feature point and its nearest neighbor feature point.

6. The multi-scale image matching method according to claim 1, characterized in that: The implementation method of the aggregation module, encoding module, MHSA module and FFN network is: The aggregation module performs the feature G output by the BDB module. out Perform neighborhood aggregation operations to capture more geometric and semantic information: F Agg = Agg(G out ) Among them, F Agg Represents the features output by the aggregation module, and Agg( ) represents the neighborhood aggregation operation; the features aggregated by the aggregation module contain the relationship between the current feature and its neighbors; The encoding module outputs the feature F of the aggregation module. Agg Perform deep encoding to extract high-level semantic features and maintain the integrity of the input information: F Encoded = Encoder(F Agg ) Among them, F Encoded Represents the features output by the encoding module, and Encoder( ) represents the encoding operation; the encoder consists of multiple ResNet blocks, each of which performs a convolution operation to extract more feature information; The MHSA module introduces a length similarity matrix to calculate the geometric relationship between feature points to ensure the accuracy of feature matching: F MHSA = MHSA(F Encoded , M ls ) Among them, F MHSA represents the output features of the MHSA module, MHSA( ) represents the multi-head self-attention mechanism operation, M ls Represents a length similarity matrix; the length similarity matrix M ls The calculation method is: For the input image pair {I A , I B } Use feature detector for preprocessing, and then establish the initial corresponding set I through the nearest neighbor matching strategy C ={c1, c2, ..., c i , ..., c j ,..., c N }, where c i 、c j They represent the initial corresponding set I C The i-th and j-th feature point pairs in N represent the initial corresponding set I C The number of feature point pairs in c i =(p i A , p i B ), c i =(p j A , p j B ), where p i A 、p i B Represents the input image I A , input image I B The length similarity matrix M ls Element m i,j Calculated according to the following formula: Among them, || || represents norm calculation, | | represents taking the absolute value; The features output by the MHSA module are divided into two paths, one path is input into the FFN network, and the other path is spliced ​​with the features output by the FFN network to obtain the final output features; The FFN network receives the features output by the MHSA module and performs a series of transformations to further extract features: Among them, Linear1 is the first linear transformation layer, which maps the features output by the MHSA module to a higher dimensional space to enhance the representation capability; ReLU () is the ReLU activation function, which processes the features output by Linear1 to enhance the nonlinear characteristics and obtain the activated notification; Linear2 is the second linear transformation layer, which is used to map the activated features back to the final output space.

7. The multi-scale image matching method according to claim 1, characterized in that: Using a mixed loss function To supervise the training process of the image matching model: in, represents classification loss; α is a hyperparameter that balances classification loss and essential matrix loss; It represents the essential matrix loss, which is used to measure the difference between the predicted essential matrix and the true essential matrix. After feature extraction through the image matching model, the predicted essential matrix is ​​calculated based on the matched feature point pairs, and then the essential matrix loss is calculated; Classification Loss It is expressed as: Where H( ) represents the binary cross entropy loss function; t represents the relevant weight of the tth iteration; y t represents a weakly supervised label; ω t is the adaptive temperature vector, ⊙ represents the Hadamard product, and λ is the number of iterations.

8. A multi-scale image matching system, characterized in that: The method comprises a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the method according to any one of claims 1 to 7 can be implemented.

Images