SeViMatch feature matching method based on semantic-visual fusion

Abstract

The invention discloses a SeViMatch feature matching method based on semantic-visual fusion, and belongs to the technical field of image matching. An image matching method based on a detector is widely applied due to high calculation efficiency, but redundant background features are easily extracted under low-texture and complex scenes, and the matching precision and robustness are reduced due to lack of perception and fusion of image semantics. Therefore, the invention provides a SeViMatch feature matching method based on semantic-visual fusion, and the method comprises three core modules: a self-adaptive feature optimization module focuses on a foreground region and suppresses background noise; the semantic perception module is used for modeling scene semantic distribution through an auto-encoder-decoder structure to realize cross-image semantic alignment; and the dynamic semantic fusion module adaptively adjusts the fusion proportion of semantic and visual features according to the content to realize deep semantic-visual collaborative representation. Multiple groups of experimental results show that the matching precision and robustness are remarkably improved, and the effectiveness of complementary fusion of semantic and visual features is verified.

Description

Technical Field

[0001] This invention relates to the field of image matching technology, and specifically to a SeViMatch feature matching method based on semantic-visual fusion. Background Technology

[0002] With the widespread application of computer vision technology in fields such as autonomous driving, augmented reality, and 3D reconstruction, image matching, as one of the core technologies, is becoming increasingly important. Image matching aims to establish relationships between different images by identifying and matching similar feature points, thereby enabling tasks such as scene understanding and target localization. However, existing image matching methods still face many challenges in terms of accuracy and efficiency.

[0003] Based on whether or not they rely on keypoint detectors, existing image matching methods can be divided into detector-free methods and detector-based methods. Detector-free image matching methods first learn dense feature representations at the pixel or region level across the entire image, and then perform dense local feature matching. Even in scenes with low texture or large viewpoint differences, they can achieve high matching accuracy, but their high computational cost limits their application in real-time tasks. In contrast, detector-based image matching methods detect salient keypoints and extract and match descriptors from these keypoints, offering advantages such as low computational cost and high speed, making them more suitable for scenarios with high real-time requirements. Currently, detector-based image matching methods face two key challenges: 1. Difficulty in effectively distinguishing foreground and background features in an image: Traditional detector-based methods often rely on local features of the image, such as low-level visual information like edges and corners, during the feature extraction process. In complex backgrounds or low-texture areas, redundant or even misleading features are easily extracted, resulting in the foreground being weakened and the matching accuracy being reduced.

[0004] 2. Lack of perception and fusion of semantic information in images: Semantic information is crucial for understanding image content and building stable cross-image correspondences; however, traditional methods lack perception of semantic information, which makes it impossible to fully utilize the high-level information contained in images to guide the matching process, limiting the feature expression ability of the model and making it difficult to adapt to diverse and complex scenarios. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a SeViMatch feature matching method based on semantic-visual fusion. The method uses an adaptive feature optimization module to focus on the foreground region and suppress background noise; a semantic perception module models the semantic distribution of the scene through an autoencoder-decoder structure to achieve cross-image semantic alignment; and a dynamic semantic fusion module adaptively adjusts the fusion ratio of semantic and visual features according to the content to achieve deep semantic-visual collaborative representation, which significantly improves matching accuracy and robustness.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: a SeViMatch feature matching method (SeViMatch) based on semantic-visual fusion, comprising the following steps: S1. Construct a multi-scale feature extraction module: After preprocessing the input original image pair, it is downsampled three times through three deformable convolutional blocks of the CNN backbone network, and then multi-scale features are extracted through residual structure connection, outputting 1 / 4 scale feature map, 1 / 8 scale feature map and 1 / 16 scale feature map. The preprocessing operation involves converting the original image to grayscale, unifying the scale to 640×640 pixels, enhancing contrast, and removing noise. During training, data augmentation operations such as random cropping, rotation, and flipping are performed. The backbone network includes three deformable convolutional blocks; This invention preprocesses the original image pairs and then inputs them into a CNN backbone network to extract multi-scale feature maps of the image pairs. The backbone network can capture feature information of the image at different scales, providing basic feature representations for subsequent key point detection and descriptor generation. In this invention, the multi-scale feature extraction module first pairs the preprocessed image data. The input is fed into the backbone network for feature extraction. The CNN backbone network consists of three deformable convolutional blocks, each containing a deformable convolutional layer, a BatchNorm layer, and a ReLU activation layer. Each time the image passes through a convolutional block, its height and width are halved, and the number of channels is doubled. After being connected by a residual structure, the final output is feature maps at three scales. , , .

[0007] S2. Construct a keypoint detection module: Input a 1 / 8 scale feature map, generate a keypoint location probability map through a keypoint detection head, and output a set of keypoints through non-maximum suppression and a pre-set confidence threshold; simultaneously input the original image, output a teacher network heatmap through a pre-trained teacher network; based on the keypoint location probability map and the teacher network heatmap, perform distillation loss function calculation to supervise keypoint location.

[0008] S3. Constructing the descriptor generation module: Inputting 1 / 4 scale feature maps, 1 / 8 scale feature maps, and 1 / 16 scale feature maps, the adaptive feature optimization module outputs optimized feature maps; based on the 1 / 8 scale feature map, the semantic perception module outputs aligned semantic features; inputting the aligned semantic feature map and the optimized feature map, the dynamic semantic fusion module outputs feature maps containing semantic information, and then performing local feature extraction through the descriptor header to generate descriptors for key points.

[0009] S4. Based on the key point descriptor, after similarity calculation and matching confidence weight calculation and weight adjustment, the matching threshold is filtered to output a stable set of matching point pairs. As a preferred embodiment of the present invention, step S1 includes the following steps: S1.1 Input the original image pair, perform preprocessing operations on the original image pair, and output the preprocessed image pair; This invention adjusts the pixel values ​​of the original image to [0, 1] through normalization, then unifies the image scale to a specific resolution, such as 640×640 pixels, enhances the image contrast using histogram equalization, removes noise using Gaussian filtering, and finally uses random cropping, rotation, flipping and other methods to perform data augmentation during training to improve the generalization ability of the model. S1.2 Input the preprocessed image pair, perform three downsampling operations through three deformable convolutional blocks of the CNN backbone network, extract multi-scale features through residual structure connections, and output 1 / 4 scale feature map, 1 / 8 scale feature map and 1 / 16 scale feature map. The deformable convolutional block includes a deformable convolutional layer, an average pooling layer, and an activation layer; Deformable convolutional blocks are the core components of the feature extraction module. Each block consists of a deformable convolutional layer, an average pooling layer, and an activation layer. By introducing a spatial offset, the deformable convolutional layer can adaptively adjust the receptive field, thereby capturing key local features in the image more flexibly. The specific calculation expression for deformable convolutional layers is as follows: Where F is the input feature map, Δp is the generated offset, and K is the convolution kernel. It is the feature map after deformed convolution; As a preferred embodiment of the present invention, step S2 includes the following steps: S2.1 Keypoint Detection: Input a 1 / 8 scale feature map, generate a keypoint location probability map through the keypoint detection head, and output a set of keypoints through non-maximum suppression and pre-set confidence threshold filtering; The key point detection head consists of a convolutional layer and an activation layer; This invention uses 1 / 8 scale feature maps The input is fed into a keypoint detection head consisting of a set of 3×3 convolutions. Convolution operations are used to perform deep feature extraction and dimension transformation on the feature map, generating a keypoint location probability map. The corresponding formula for this process is: in, This represents the 1 / 8 scale feature map of the input. This represents the computational process of a keypoint detection head consisting of a set of 3×3 convolutions. This results in a probability map of the generated keypoint locations; Subsequently, non-maximum suppression (NMS) and thresholding are applied to the keypoint location probability map: NMS eliminates redundant candidate points in the local neighborhood, retaining only the keypoints with the highest probability values, as shown in the formula: ,in To suppress the radius, Given the candidate keypoint set after NMS processing, a confidence threshold is then set to filter out those that meet the requirements. The high-confidence keypoint locations are used to obtain the final keypoint set. .

[0010] S2.2 Keypoint Location Supervision: Input the original image, pass through the pre-trained teacher network, and output the teacher network heatmap; based on the keypoint location probability map and the teacher network heatmap, perform distillation loss function calculation to supervise keypoint location. Key point location supervision: Simultaneously, this invention introduces ALIKE as a teacher network, inputting the original image to generate a teacher key point heatmap. As a monitoring signal, the key point heatmap generated by the student network is calculated. Learning is guided by the loss function of the teacher network heatmap, and the loss formula is expressed as follows: in, It is the total number of pixels. and These are heatmaps generated by the student network and the teacher network, respectively, at pixel levels. The value at the location; through this monitoring mechanism, the accuracy of the key point locations generated by the student network is ensured. This is the index of the total number of pixels.

[0011] As a preferred embodiment of the present invention, step S3 includes the following steps: S3.1 Input 1 / 4 scale feature map, 1 / 8 scale feature map and 1 / 16 scale feature map, and output the optimized feature map through the adaptive feature optimization module; The adaptive feature optimization module includes: a microscopic attention mechanism, a focused attention mechanism, a background suppression mechanism, and feature fusion; Specifically, the following steps are included: S3.1.1 Based on the 1 / 4 scale feature map, an enhanced detail feature map is output through a microscopic attention mechanism; Microscopic attention mechanism: A 3×3 convolution kernel is used to convolve the input 1 / 4 scale feature map to generate a detail attention map. The feature map is then normalized using the Sigmoid activation function, and then the detail attention map is multiplied element-wise with the input feature map to enhance the detail information. The specific expression is as follows: in, To enhance the detailed feature map, This represents a detail attention map. Represents element-wise product. This represents the Sigmoid activation function. A 1 / 4 scale feature map; The microscopic attention mechanism effectively enhances local details in images, especially minute features around object boundaries and key points, enabling the model to better capture these precise details.

[0012] S3.1.2 Input a 1 / 8 scale feature map, and output a key focus attention map through a focus attention mechanism; Focused attention mechanism: Adaptive average pooling is used to scale the input 1 / 8 scale feature map. Compressed to a 1×1 spatial dimension, a focus attention map is generated through two convolutional layers. The input feature map is then normalized using the Sigmoid activation function. Finally, the channel attention map is multiplied element-wise along the channel dimension with the input feature map to highlight key regions. The specific expression is as follows: in, To focus on attention maps, For key focus attention maps, A 1 / 8 scale feature map; Focused attention mechanisms can enhance key information about objects or regions of interest in an image, while avoiding excessive attention to background areas. S3.1.3 Input a 1 / 16 scale feature map, and output a suppressed background feature map through a background suppression mechanism; Background suppression mechanism: Global adaptive average pooling is used to generate a background weight map, which is then used to suppress the background weight map at a 1 / 16 scale from the input feature map. A background weight map is generated through convolutional layers. The background weight map is then normalized using the Sigmoid activation function, and then multiplied element-wise with the input feature map to suppress the background region. The specific expression is as follows: in, For background weighting, To suppress background feature maps, A feature map at a 1 / 16 scale; The background suppression mechanism branch enables the feature map to... The background region becomes smooth, while preserving the high-frequency features of the object and avoiding interference from irrelevant noise on subsequent semantic fusion and feature matching; S3.1.4. Based on the enhanced detail feature map, the key focus attention map, and the suppressed background feature map, perform feature fusion and output the optimized feature map; Feature fusion: , and The features are sampled to a uniform scale using bilinear interpolation, then concatenated along the channel dimension, and finally fused using a 1×1 convolutional layer to obtain the optimized feature map. The specific expression is as follows: in, These are, respectively, a uniform-scale enhanced detail feature map, a key focus attention map, and a background-suppressed feature map obtained through bilinear interpolation sampling. This is the optimized feature map.

[0013] S3.2 Based on the 1 / 8 scale feature map, the semantic perception module outputs the aligned semantic features; Specifically, the following steps are included: S3.2.1 Input a 1 / 8 scale feature map, and obtain a low-dimensional semantic feature map through the encoder module; Input 1 / 8 scale feature map After processing by the encoder module, it is mapped into a low-dimensional semantic feature map. The specific formula for obtaining the low-dimensional semantic feature map is as follows: in, Using 3×3 convolutional layers, the spatial resolution is reduced from... Compress to At the same time, the number of channels was adjusted to 128. For a 1×1 convolutional layer, the number of channels is compressed to 64; S3.2.2 Input a low-dimensional semantic feature map. After passing through the semantic alignment module, the decoder restores the low-dimensional semantic feature map to a feature map of the same scale as the input 1 / 8 scale feature map and outputs the aligned semantic feature map. The semantic alignment module includes a self-attention mechanism and a cross-attention mechanism; Semantic alignment module: aligns low-dimensional semantic feature maps Its unfolding yields semantic tokens To further enhance the expressive power of semantic features and their alignment ability across images, self-attention and cross-attention mechanisms are introduced, enabling the module to adaptively adjust the attention paid to different regions according to the specific semantic content of the image. The specific formula is as follows: Where MHSA represents multi-head self-attention mechanism, MHCA represents multi-head cross-attention mechanism, Q is the query feature, K is the key feature, V is the value feature, and d is the dimension of the feature; Finally, the decoder restores the low-dimensional semantic feature map to a 1 / 8 scale feature map. By using feature maps of the same scale, this restoration process ensures the consistency between the spatial structure and semantic content of the image, thereby further enhancing the semantic expressive power of the feature maps and resulting in aligned semantic feature maps: in, For transposed convolutional layers, the resolution is changed from... Restore to , For the aligned semantic feature map S3.2.3. Based on the low-dimensional semantic feature map and the aligned semantic feature map, perform the reconstruction loss function calculation to obtain the reconstruction loss value; The reconstruction loss of this invention calculates the difference between the input feature map and the reconstructed feature map, ensuring that key semantic information is preserved during semantic extraction and alignment, while suppressing irrelevant background information. The reconstruction loss function is expressed as follows: in, Represents a low-dimensional semantic feature map. The aligned semantic feature map Indicates the total number of pixels. An index representing the total number of pixels.

[0014] S3.3 Input the aligned semantic feature map and the optimized feature map, and output a feature map containing semantic information through the dynamic semantic fusion module; Specifically, the following steps are included: S3.3.1 Based on the aligned semantic features, use the K-Means algorithm for joint clustering to initialize key semantic features and output key semantic features; Key semantic feature initialization: using the K-Means algorithm to initialize the semantic features of image pairs. and Joint clustering is performed on the two images. This strategy forces the model to extract shared semantic prototypes (such as cross-scene common semantics like "sky" and "building") from the two images, thereby eliminating semantic ambiguity caused by differences in viewpoint or occlusion (such as local deformation or partial visibility of the same object in different images). The key semantic features generated after clustering It contains 10 representative semantic cluster centers, each cluster center corresponding to a common semantic information across images: in, As the first semantic feature, As a second semantic feature, The number of cluster centers is 10 in this invention; S3.3.2 Based on key semantic features, the updated key semantic features are output through semantic-visual cross-attention; Dynamic update of key semantic features: Key semantic features extracted through K-Means clustering Essentially, this is a static compression of semantic features, lacking dynamic perception of actual visual content; therefore, semantic-visual cross-attention is used to optimize the image's visual features. As a key value, For queries, dynamically filter semantic information that is strongly related to the visual content. : Here, MHCA stands for Multi-Head Cross-Attention Mechanism, and Concat represents the concatenation operation, used to concatenate and integrate the output features of multiple attention heads. This represents the query weight matrix. Represents the key weight matrix. Represents the value weight matrix. represents the output projection weight matrix, d represents the total dimension of the input features, and h is the number of attention heads; Subsequently, the module further optimizes the relationship between key semantic features through self-attention and cross-attention mechanisms. The self-attention mechanism can enhance the correlation of key semantic features within each image, better capturing the semantic structure within the image: The cross-attention mechanism effectively promotes the fusion of semantic information across images by aligning key semantic features between image pairs. This ensures that shared semantic features between image pairs are transmitted and integrated. This interactive process effectively eliminates semantic differences between image pairs. The calculation expression for the updated key semantic features is as follows: in, This represents the key semantic features within image A after enhancement. This represents the key semantic features within image B after enhancement. This represents the key semantic features of image A after the update. This represents the key semantic features of image B after the update. Meanwhile, using key semantic features as intermediate representations for information transmission can significantly reduce computational overhead while ensuring the accuracy of information delivery; specifically, traditional attention mechanisms require calculating the interactions between all features, with a complexity of O(n log n). By using sparse representation of key semantic features, the complexity is reduced to [missing value]. , where k is the number of key semantic features, and is much smaller than n.

[0015] S3.3.3 Based on the updated key semantic features as keys and values, and the optimized feature map of the image as the query, semantic fusion is performed through a multi-head cross-attention mechanism to generate a feature map containing semantic information. The computational expression for generating a feature map containing semantic information is: In the formula, Key semantic features The optimized feature map is shown below, where Q represents the query, K represents the key, and V represents the value.

[0016] S3.4 Based on the feature map containing semantic information, local features are extracted through the descriptor header to generate descriptors for key points; This invention uses a lightweight descriptor header to extract local features from the feature map after fusing semantic information, generating descriptors for key points. ; in, A feature map containing semantic information. It consists of multiple 3×3 convolutional layers; S3.5. Based on key point descriptors, descriptor supervision is performed through scalar loss calculation, and scalar loss values ​​are output. This invention supervises the generated descriptors using a dual softmax strategy, thereby ensuring that descriptors of matching points are similar, while descriptors of non-matching points are significantly different. The scalar loss function can be expressed as: in, In feature map A, the first... The first feature and the first feature map B Similarity between feature descriptors Index A for feature map A Index B for feature map.

[0017] As a preferred embodiment of the present invention, step S4 includes the following steps: The formula for calculating similarity is: In the formula, The first feature point in each of the two images Second feature point The descriptor; The expression for calculating the matching confidence weight is: In the formula, The response intensity at the feature point For neighborhood feature consistency, The preset balance coefficient; The weighted adjustment calculation expression is: Combining the weighted similarity matrix with the preset matching threshold Filter out The candidate matching pairs are then subjected to two-way verification, requiring... and Finally, a stable set of matching point pairs is obtained. .

[0018] Compared with existing technologies, this invention provides a SeViMatch feature matching method based on semantic-visual fusion, which has the following beneficial effects: This invention significantly improves the accuracy and robustness of matching by deeply fusing semantic information and visual features, especially performing well in complex scenarios such as changes in lighting, viewpoint, and occlusion. Simultaneously, it employs a self-learning approach to extract semantic features, eliminating the need for external semantic segmentation networks or large amounts of labeled data, effectively reducing computational costs. Furthermore, the lightweight feature extraction module, constructed using multi-scale shallow networks and deformable convolution techniques, reduces computational load and increases feature extraction speed, meeting the demands of applications with high real-time requirements. This invention is applicable to various image matching tasks, such as environmental perception in autonomous driving and scene reconstruction in augmented reality, demonstrating strong versatility and adaptability.

[0019] This invention trains on the publicly available image dataset Megadepth, fully utilizing its large-scale outdoor scene data and multi-view image information to learn rich geometric and semantic feature representations. Testing on the Megadepth test set, the method achieved excellent matching performance, verifying the model's effectiveness in handling large-scale scenes and complex lighting conditions. Furthermore, by directly applying the pre-trained weights to the Scannet dataset, experimental results show that this invention also outperforms existing image matching methods on the Scannet test set. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the feature matching method for semantic-visual fusion according to the present invention; Figure 2 This is a schematic diagram of the image feature matching method based on semantic fusion feature detection and description of the present invention; Figure 3 This is a schematic diagram of the multi-scale feature extraction module structure of the present invention; Figure 4 This is a schematic diagram of the adaptive feature optimization module structure of the present invention; Figure 5 This is a schematic diagram of the semantic perception module structure of the present invention; Figure 6 This is a schematic diagram of the dynamic semantic fusion module structure of the present invention; Figure 7 The figures show the MMA variation curves of different methods in this invention under thresholds of 1 to 10 pixels. Figure (a) is the overall figure, Figure (b) is the figure under illumination changes, and Figure (c) is the figure under viewpoint changes. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0022] The network was trained using the open-source PyTorch deep learning framework and trained with the Adam optimizer. Due to hardware limitations, the training image size was set to 640×640, the batch size to 5, and the weight decay to 3×10⁻⁶. 4. The training on a single GTX 1080Ti GPU converged after 24 hours.

[0023] Please see Figures 1-2 A SeViMatch feature matching method (SeViMatch) based on semantic-visual fusion specifically includes the following steps: S1. Construct a multi-scale feature extraction module: After preprocessing the input original image pair, it is downsampled three times through three deformable convolutional blocks of the CNN backbone network, and then multi-scale features are extracted through residual structure connection, outputting 1 / 4 scale feature map, 1 / 8 scale feature map and 1 / 16 scale feature map. The backbone network consists of three deformable convolutional blocks; This invention preprocesses the original image pairs and then inputs them into a CNN backbone network to extract multi-scale feature maps of the image pairs. The backbone network can capture feature information of the image at different scales, providing basic feature representations for subsequent key point detection and descriptor generation. like Figure 3 As shown, the multi-scale feature extraction module first processes the preprocessed image... The input is fed into the backbone network for feature extraction. The CNN backbone network consists of three deformable convolutional blocks, each containing a deformable convolutional layer, a BatchNorm layer, and a ReLU activation layer. Each time the image passes through a convolutional block, its height and width are halved, and the number of channels is doubled. After being connected through a residual structure, the final output features at three scales are obtained. Figure 1 / 4 scale feature map 1 / 8 scale feature map and 1 / 16 scale feature map ; Specifically, the following steps are included: S1.1 Input the original image pair, perform preprocessing operations on the original image pair, and output the preprocessed image pair; This invention adjusts the pixel values ​​of the original image to [0, 1] through normalization, performs grayscale processing, unifies the image scale to a specific resolution, such as 640×640 pixels, enhances image contrast through histogram equalization, removes noise through Gaussian filtering, and finally uses random cropping, rotation, flipping and other methods to perform data augmentation during training to improve the generalization ability of the model. S1.2 Input the preprocessed image pair, perform three downsampling operations through three deformable convolutional blocks of the CNN backbone network, extract multi-scale features through residual structure connections, and output 1 / 4 scale feature map, 1 / 8 scale feature map and 1 / 16 scale feature map. In this invention, deformable convolutional blocks are the core components of the feature extraction module. Each block consists of deformable convolutional layers, average pooling layers, and activation layers. By introducing spatial offsets, deformable convolutional layers can adaptively adjust the receptive field, thereby capturing key local features in the image more flexibly. The specific calculation expression for deformable convolutional layers is as follows: Where F is the input feature map, Δp is the generated offset, and K is the convolution kernel. It is the feature map after deformed convolution.

[0024] S2. Construct a keypoint detection module: Input a 1 / 8 scale feature map, generate a keypoint location probability map through a keypoint detection head, and output a set of keypoints through non-maximum suppression and a pre-set confidence threshold; simultaneously input the original image, output a teacher network heatmap through a pre-trained teacher network; based on the keypoint location probability map and the teacher network heatmap, perform distillation loss function calculation to supervise keypoint location. S2.1 Input a 1 / 8 scale feature map, generate a key point location probability map through the key point detection head, and output a set of key points through non-maximum suppression and preset confidence threshold filtering; This invention uses 1 / 8 scale feature maps The input is fed into a keypoint detection head consisting of a set of 3×3 convolutions. Convolution operations are used to perform deep feature extraction and dimension transformation on the feature map, generating a keypoint location probability map. The corresponding formula for this process is: in, This represents the 1 / 8 scale feature map of the input. This represents the computational process of a keypoint detection head consisting of a set of 3×3 convolutions. This results in a probability map of the generated keypoint locations; Subsequently, non-maximum suppression (NMS) and thresholding are applied to the keypoint location probability map: NMS eliminates redundant candidate points in the local neighborhood, retaining only the keypoints with the highest probability values, as shown in the formula: ,in To suppress the radius, Given the candidate keypoint set after NMS processing, a confidence threshold is then set to filter out those that meet the requirements. The high-confidence keypoint locations are used to obtain the final keypoint set. This embodiment uses modules such as AFOM, SAM, and DSFM to generate descriptors with extremely strong discriminative power. Ideally, the similarity of correct matches should be infinitely close to 1, and the similarity of incorrect matches should be infinitely close to 0. Therefore, this invention sets a high confidence threshold of 0.8. S2.2 Keypoint Location Supervision: Input the original image, pass through the pre-trained teacher network, and output the teacher network heatmap; based on the keypoint location probability map and the teacher network heatmap, perform distillation loss function calculation to supervise keypoint location. Key point location supervision: Simultaneously, this invention introduces ALIKE as a teacher network, inputting the original image to generate a teacher key point heatmap. As a monitoring signal, the key point heatmap generated by the student network is calculated. Learning is guided by the loss function of the teacher network heatmap, and the loss formula is expressed as follows: in, It is the total number of pixels. and These are heatmaps generated by the student network and the teacher network, respectively, at pixel levels. The value at the location; through this monitoring mechanism, the accuracy of the key point locations generated by the student network is ensured. This is the index of the total number of pixels.

[0025] S3. Constructing the descriptor generation module: Input 1 / 4 scale feature map, 1 / 8 scale feature map and 1 / 16 scale feature map, and obtain the optimized feature map through the adaptive feature optimization module. Then input the optimized feature map into the dynamic semantic fusion module, combine it with the global semantic features extracted by the semantic perception module, and use the multi-head attention mechanism to realize the adaptive interaction between visual features and semantic information to generate a descriptor containing global semantic information. Specifically, the following steps are included: S3.1 Input 1 / 4 scale feature map, 1 / 8 scale feature map and 1 / 16 scale feature map, and output the optimized feature map through the adaptive feature optimization module; Adaptive Feature Optimization Module (AFOM): such as Figure 4As shown, the adaptive optimization feature module optimizes the multi-scale feature map generated by the multi-scale feature extraction module. , , Further optimizations were made; this module dynamically adjusts the weights of key information and background noise in the feature map through mechanisms such as microscopic attention, focused attention, and background suppression. Specifically, the following steps are included: S3.1.1 Based on the 1 / 4 scale feature map, an enhanced detail feature map is output through a microscopic attention mechanism; Microscopic attention mechanism: A 3×3 convolution kernel is used to convolve the input 1 / 4 scale feature map to generate a detail attention map. The feature map is then normalized using the Sigmoid activation function, and then the detail attention map is multiplied element-wise with the input feature map to enhance the detail information. The specific expression is as follows: in, To enhance the detailed feature map, This represents a detail attention map. Represents element-wise product. This represents the Sigmoid activation function. A 1 / 4 scale feature map; The microscopic attention mechanism effectively enhances local details in images, especially small features around object boundaries and key points, enabling the model to better capture these precise details; S3.1.2 Input a 1 / 8 scale feature map, and output a key focus attention map through a focus attention mechanism; Focused attention mechanism: Adaptive average pooling is used to scale the input 1 / 8 scale feature map. Compressed to a 1×1 spatial dimension, a focus attention map is generated through two convolutional layers. The input feature map is then normalized using the Sigmoid activation function. Finally, the channel attention map is multiplied element-wise along the channel dimension with the input feature map to highlight key regions. The specific expression is as follows: in, To focus on attention maps, For key focus attention maps, A 1 / 8 scale feature map; Focused attention mechanisms can enhance key information about objects or regions of interest in an image, while avoiding excessive attention to background areas. S3.1.3 Input a 1 / 16 scale feature map, and output a suppressed background feature map through a background suppression mechanism; Background suppression mechanism: Global adaptive average pooling is used to generate a background weight map, which is then used to suppress the background weight map at a 1 / 16 scale from the input feature map. A background weight map is generated through convolutional layers. The background weight map is then normalized using the Sigmoid activation function, and then multiplied element-wise with the input feature map to suppress the background region. The specific expression is as follows: in, For background weighting, To suppress background feature maps, A feature map at a 1 / 16 scale; The background suppression mechanism branch enables the feature map to... The background region becomes smooth, while preserving the high-frequency features of the object and avoiding interference from irrelevant noise on subsequent semantic fusion and feature matching; S3.1.4. Based on the enhanced detail feature map, the key focus attention map, and the suppressed background feature map, perform feature fusion and output the optimized feature map; Feature fusion: , and The features are sampled to a uniform scale using bilinear interpolation, then concatenated along the channel dimension, and finally fused using a 1×1 convolutional layer to obtain the optimized feature map. The specific expression is as follows: in, It is a feature map of a uniform scale obtained through bilinear interpolation sampling. The optimized feature map; S3.2 Based on the 1 / 8 scale feature map, the semantic perception module outputs the aligned semantic features; Semantic Aware Module (SAM): such as Figure 5 As shown, the specific steps include: S3.2.1 Input a 1 / 8 scale feature map, and obtain a low-dimensional semantic feature map through the encoder module; Input 1 / 8 scale feature map After processing by the encoder module, it is mapped into a low-dimensional semantic feature map. The specific formula for obtaining the low-dimensional semantic feature map is as follows: in, Using 3×3 convolutional layers, the spatial resolution is reduced from... Compress to At the same time, the number of channels was adjusted to 128. For a 1×1 convolutional layer, the number of channels is compressed to 64; S3.2.2 Input a low-dimensional semantic feature map. After passing through the semantic alignment module, the decoder restores the low-dimensional semantic feature map to a feature map of the same scale as the input 1 / 8 scale feature map and outputs the aligned semantic feature map. The semantic alignment module includes a self-attention mechanism and a cross-attention mechanism; Semantic alignment module: aligns low-dimensional semantic feature maps Its unfolding yields semantic tokens To further enhance the expressive power of semantic features and their alignment ability across images, self-attention and cross-attention mechanisms are introduced, enabling the module to adaptively adjust the attention paid to different regions according to the specific semantic content of the image. The specific formula is as follows: Where MHSA represents multi-head self-attention mechanism, MHCA represents multi-head cross-attention mechanism, Q is the query feature, K is the key feature, V is the value feature, and d is the dimension of the feature; Finally, the decoder restores the low-dimensional semantic feature map to a 1 / 8 scale feature map. By using feature maps of the same scale, this restoration process ensures the consistency between the spatial structure and semantic content of the image, thereby further enhancing the semantic expressive power of the feature maps and resulting in aligned semantic feature maps: in, For transposed convolutional layers, the resolution is changed from... Restore to , The aligned semantic feature map; S3.2.3. Based on the low-dimensional semantic feature map and the aligned semantic feature map, perform the reconstruction loss function calculation to obtain the reconstruction loss value; The reconstruction loss of this invention calculates the difference between the input feature map and the reconstructed feature map, ensuring that key semantic information is preserved during semantic extraction and alignment, while suppressing irrelevant background information. The specific formula for calculating the reconstruction loss function is as follows: in, Represents a low-dimensional semantic feature map. The aligned semantic feature map Indicates the total number of pixels. An index representing the total number of pixels.

[0026] S3.3 Input the aligned semantic feature map and the optimized feature map, and output a feature map containing semantic information through the dynamic semantic fusion module; Dynamic Semantic Fusion Module (DSFM): such as Figure 6 As shown, the dynamic semantic fusion module integrates semantically aligned semantic features. And the optimized feature map generated by the adaptive feature optimization module As input; Specifically, the following steps are included: S3.3.1 Based on the aligned semantic features, use the K-Means algorithm for joint clustering to initialize key semantic features and output key semantic features; Key semantic feature initialization: using the K-Means algorithm to initialize the semantic features of image pairs. and Joint clustering is performed on the two images. This strategy forces the model to extract shared semantic prototypes (such as cross-scene common semantics like "sky" and "building") from the two images, thereby eliminating semantic ambiguity caused by differences in viewpoint or occlusion (such as local deformation or partial visibility of the same object in different images). The key semantic features generated after clustering It contains 10 representative semantic cluster centers, each cluster center corresponding to a common semantic information across images: in, As the first semantic feature, As a second semantic feature, This represents the number of cluster centers. S3.3.2 Based on key semantic features, the updated key semantic features are output through semantic-visual cross-attention; Dynamic update of key semantic features: Key semantic features extracted through K-Means clustering Essentially, this is a static compression of semantic features, lacking dynamic perception of actual visual content; therefore, semantic-visual cross-attention is used to optimize the image's visual features. As a key value, For queries, dynamically filter semantic information that is strongly related to the visual content. : Here, MHCA stands for Multi-Head Cross-Attention Mechanism, and Concat represents the concatenation operation, used to concatenate and integrate the output features of multiple attention heads. This represents the query weight matrix. Represents the key weight matrix. Represents the value weight matrix. represents the output projection weight matrix, d represents the total dimension of the input features, and h is the number of attention heads; Subsequently, the module further optimizes the relationship between key semantic features through self-attention and cross-attention mechanisms. The self-attention mechanism can enhance the correlation of key semantic features within each image, better capturing the semantic structure within the image: The cross-attention mechanism effectively promotes the fusion of semantic information across images by aligning key semantic features between image pairs. This ensures that shared semantic features between image pairs are transmitted and integrated. This interactive process effectively eliminates semantic differences between image pairs. The calculation expression for the updated key semantic features is as follows: Meanwhile, using key semantic features as intermediate representations for information transmission can significantly reduce computational overhead while ensuring the accuracy of information delivery; specifically, traditional attention mechanisms require calculating the interactions between all features, with a complexity of O(n log n). By using sparse representation of key semantic features, the complexity is reduced to [missing value]. , where k is the number of key semantic features, and is much smaller than n; S3.3.3 Based on the updated key semantic features as keys and values, and the optimized feature map of the image as the query, semantic fusion is performed through a multi-head cross-attention mechanism to generate a feature map containing semantic information. This invention uses updated key semantic features The optimized feature map of the image serves as both key and value. As a query, it is input into a multi-head cross-attention mechanism for fusion, generating a final feature map containing semantic information. : In the formula, Key semantic features The optimized feature map is shown, where K is the key, V is the value, and Q is the query. In this stage, visual and semantic information are fused through an attention mechanism, and the generated feature map retains both the visual details of the image and incorporates semantic information.

[0027] S3.4 Based on the feature map containing semantic information, local features are extracted through the descriptor header to generate descriptors for key points; This invention uses a lightweight descriptor header to extract local features from the feature map after fusing semantic information, generating descriptors for key points. ; in, A feature map containing semantic information. It consists of multiple 3×3 convolutional layers; The lightweight descriptor head of this invention consists of multiple 3×3 convolutional layers, batch normalization, and the ReLU activation function, which can map the fused high-dimensional semantic-visual feature map to a fixed-dimensional descriptor space; firstly, By interpolating and upsampling to the original image size, and then generating discriminative descriptors through descriptor headers, local region features are finally extracted based on key point locations. In this way, the generated descriptors not only preserve the visual details of the image, but also integrate semantic information, thereby effectively improving the accuracy and robustness of feature matching while ensuring computational efficiency.

[0028] S3.5. Based on key point descriptors, descriptor supervision is performed through scalar loss calculation, and scalar loss values ​​are output. This invention supervises the generated descriptors using a dual softmax strategy, thereby ensuring that descriptors of matching points are similar, while descriptors of non-matching points are significantly different. The scalar loss function can be expressed as: in, In feature map A, the first... The first feature and the first feature map B Similarity between feature descriptors Index A for feature map A Index B for feature map.

[0029] S4. Based on the key point descriptor, after similarity calculation and matching confidence weight calculation and weight adjustment, the matching threshold is filtered to output a stable set of matching point pairs. Feature point matching: Based on the descriptors of key points extracted from image pairs, a similarity matrix is ​​constructed by calculating the cosine similarity between the descriptors. The similarity calculation formula is: In the formula, The first feature point in each of the two images Second feature point The descriptor is used to quantify the matching potential of feature point pairs; Introducing the response intensity of fused feature points Neighborhood feature consistency The matching confidence weight is calculated using the following expression: In the formula, The response intensity at the feature point For neighborhood feature consistency, To pre-determine the balance coefficient, the similarity matrix is ​​weighted and adjusted using the following formula: Combine the adjusted matrix with the matching threshold Filter out Candidate matching pairs; then, through two-way verification, require... and Finally, a stable set of matching point pairs is obtained. As a specific embodiment, the present invention sets a matching threshold. =0.9, this threshold effectively filters out most false matches while retaining a sufficient number of correct matches for subsequent geometric verification, where the balance coefficient is... The parameter is a learnable parameter, which is optimized and determined using a grid search method on the training set (Megadepth dataset). Its final value is 0.6, which completes the robust feature matching of image pairs.

[0030] To verify this invention: This embodiment trains on the Megadepth dataset, fully utilizing its large-scale outdoor scene data and multi-view image information to learn rich geometric and semantic feature representations. Pose estimation, image matching, and visual localization experiments were then conducted.

[0031] In the attitude estimation experiment, the evaluation metric used was the AUC of the attitude error at the thresholds {5°, 10°, 20°}. The attitude error was defined as the maximum value of the angular error during translation and rotation. A higher AUC value indicates that, at that error threshold, more matching pairs have smaller rotation errors, thus indicating that the model has higher attitude estimation accuracy and robustness.

[0032] In the outdoor pose estimation experiment, this invention was tested on the Megadepth-1500 dataset, which covers a wide range of outdoor scenes, including urban and natural landscapes. These scenes exhibit significant changes in viewpoint, complex lighting variations, and scale differences, making them ideal for testing the performance of image matching algorithms under conditions of large viewpoint variations and uneven lighting. The test results are shown in Table 1. Table 1: Test Results of the Megadepth-1500 Dataset This invention performs exceptionally well on the outdoor pose estimation task on the MegaDepth-1500 dataset, achieving accuracy rates of 48.7%, 61.7%, and 71.9% for AUC@5°, AUC@10°, and AUC@20°, respectively. Notably, it outperforms other detector-based image matching methods in the crucial AUC@5° metric. This demonstrates that this invention exhibits higher matching accuracy and robustness at small angle error thresholds, effectively addressing complex factors such as illumination variations, viewpoint changes, and scale differences. While this invention lags behind detector-free image matching methods in AUC@10° and AUC@20°, the difference is not significant. This indicates that while maintaining the advantages of detector-based methods, this invention enhances matching accuracy and robustness through innovative designs such as semantic information fusion.

[0033] In the indoor pose estimation experiment, this invention was tested on the ScanNet dataset, which is characterized by dense viewpoint sampling and complex occlusion relationships, effectively verifying the geometric consistency preservation ability of image matching methods in indoor environments. The rich geometric information and diverse indoor environments of the ScanNet dataset make it a standard dataset for evaluating indoor image matching algorithms; the test results are shown in Table 2: Table 2: Test Results of the Scannet Dataset On the AUC@5° and AUC@10° metrics, the test results of this invention reached 14.43% and 28.38% respectively, significantly outperforming other detectorless methods (such as DRC-Net and CSFormer) and most detector-based methods; it also showed good accuracy on the AUC@20° metric. The experimental results verify that even without specific training for indoor scenes, this invention can still effectively cope with changes in viewing angle, lighting differences and occlusion problems in indoor environments.

[0034] In the image matching experiment, the evaluation metric used is the average matching accuracy (MMA). MMA reflects the average ability of the algorithm to correctly match image feature points under different test scenarios. The closer the value is to 1, the better the matching performance.

[0035] Image matching experiments were conducted using the HPatch dataset, a standardized image matching dataset that contains image pairs under various lighting and viewpoint variations. The test results are as follows: Figure 7As shown in the figure, the image matching evaluation results on the HPatches dataset demonstrate that the present invention exhibits significant advantages in overall performance and under different challenging scenarios (illuminance and viewpoint changes). The figure shows the MMA variation curves of different methods at thresholds of 1 to 10 pixels. It can be seen that the present invention consistently ranks among the best methods in both Viewpoint and Illumination tasks, maintaining a stable and rapid upward trend across all threshold ranges, indicating good robustness in general matching tasks and under inconsistent illumination conditions. Particularly noteworthy is that in the more challenging Viewpoint scenario, the present invention significantly outperforms all existing methods, especially maintaining high matching accuracy even at smaller pixel thresholds, demonstrating stronger local invariance and geometric consistency in dealing with geometric deformations and viewpoint shifts. This result fully validates the effectiveness of our introduced semantic fusion strategy in improving feature stability and matching reliability, enabling the present invention to demonstrate leading performance under various complex transformations. In summary, the present invention not only achieves state-of-the-art average accuracy but also achieves excellent performance in balancing robustness and accuracy.

[0036] In the visual positioning experiment, the evaluation index used was the metric provided by HLoc. That is, under different position error thresholds (0.25m, 0.5m, 5m) and rotation error thresholds (2°, 5°, 10°), the positioning was considered successful only when the distance and angle were both within the set thresholds. The success rate (R1) of the corresponding method was then calculated to comprehensively measure the algorithm performance from the aspects of accuracy and angle accuracy.

[0037] Visual localization experiments were conducted using the Aachen Day-Night dataset, which is widely used for visual localization tasks, particularly focusing on localization performance under different lighting conditions (day / night). The dataset provides high-quality 3D reconstruction models and ground truth poses (GT poses) of query images, making it suitable for systematic analysis of the localization robustness of local feature methods. The test results are shown in Table 3. Table 3: Test Results of Aachen Day-Night Dataset In daytime scenarios, the accuracy of this invention gradually improves as the distance and angle thresholds are relaxed. The accuracy is 87.3%, 95.5%, and 99.2% under conditions ranging from 0.25m-2° to 5m-10°, respectively. Overall performance surpasses most baseline algorithms; for example, at 0.5m-5°, it outperforms SuperPoint (93.2%), DISK (95.1%), ZippyPoint (88.6%), and Xfeat (91.5%), demonstrating strong robustness and accuracy. In nighttime scenarios, this invention continues to perform excellently, achieving accuracies of 90.1% and 99.5% at 0.5m-5° and 5m-10°, respectively, exceeding most other baselines. Its advantages are significant under challenging positioning conditions such as large angles and close distances. While there is a slight performance decrease compared to daytime, this reflects the difference in visual positioning difficulty between day and night, demonstrating the algorithm's good adaptability to changing day and night conditions. It effectively addresses the visual positioning challenges posed by complex lighting conditions and exhibits excellent reliability.

[0038] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A SeViMatch feature matching method based on semantic-visual fusion, characterized in that, Includes the following steps: S1. Construct a multi-scale feature extraction module: After preprocessing the input original image pair, it is downsampled three times through three deformable convolutional blocks of the CNN backbone network, and then multi-scale features are extracted through residual structure connection, outputting 1 / 4 scale feature map, 1 / 8 scale feature map and 1 / 16 scale feature map. The preprocessing operations include: converting the original image to grayscale, unifying the scale to 640×640 pixels, enhancing contrast, removing noise, and supplementing training with random cropping, rotation, and flipping data augmentation operations. The backbone network includes three deformable convolutional blocks; S2. Construct a keypoint detection module: Input a 1 / 8 scale feature map, generate a keypoint location probability map through a keypoint detection head, and output a set of keypoints through non-maximum suppression and a pre-set confidence threshold; simultaneously input the original image, output a teacher network heatmap through a pre-trained teacher network; based on the keypoint location probability map and the teacher network heatmap, perform distillation loss function calculation to supervise keypoint location. S3. Constructing the descriptor generation module: Inputting 1 / 4 scale feature maps, 1 / 8 scale feature maps, and 1 / 16 scale feature maps, the adaptive feature optimization module outputs optimized feature maps; based on the 1 / 8 scale feature map, the semantic perception module outputs aligned semantic features; inputting the aligned semantic feature map and the optimized feature map, the dynamic semantic fusion module outputs feature maps containing semantic information, and then extracts local features through the descriptor header to generate descriptors for key points; S4. Based on the key point descriptor, after similarity calculation and matching confidence weighting and weighted calculation, a matching threshold is selected to output a set of stable matching point pairs.

2. The SeViMatch feature matching method based on semantic-visual fusion according to claim 1, characterized in that, S1 specifically includes the following steps: S1.1 Input the original image pair, perform preprocessing operations on the original image pair, and output the preprocessed image pair; S1.2 Input the preprocessed image pair, perform three downsampling operations through three deformable convolutional blocks of the CNN backbone network, extract multi-scale features through residual structure connections, and output 1 / 4 scale feature map, 1 / 8 scale feature map and 1 / 16 scale feature map. The deformable convolutional block includes a deformable convolutional layer, an average pooling layer, and an activation layer; The specific calculation expression for deformable convolutional layers is as follows: Where F is the input feature map, Δp is the generated offset, and K is the convolution kernel. It is the feature map after deformed convolution.

3. The SeViMatch feature matching method based on semantic-visual fusion according to claim 1, characterized in that, S2 specifically includes the following steps: S2.1 Keypoint Detection: Input a 1 / 8 scale feature map, generate a keypoint location probability map through the keypoint detection head, and output a set of keypoints through non-maximum suppression and pre-set confidence threshold filtering; The key point detection head consists of a convolutional layer and an activation layer; The preset confidence threshold screening is based on the key point location probability map. ; The expression for generating the keypoint location probability map is: in, This represents the 1 / 8 scale feature map of the input. This represents the computational process of a keypoint detection head consisting of a set of 3×3 convolutions. This is a probability map of the generated keypoint locations; S2.2 Keypoint Location Supervision: Input the original image, pass through the pre-trained teacher network, and output the teacher network heatmap; based on the keypoint location probability map and the teacher network heatmap, perform distillation loss function calculation to supervise keypoint location. The formula for calculating the distillation loss function is: in, and These are heatmaps generated by the student network and the teacher network, respectively, at pixel levels. The value at that location, It is the total number of pixels. This is the index of the total number of pixels.

4. The SeViMatch feature matching method based on semantic-visual fusion according to claim 1, characterized in that, S3 specifically includes the following steps: S3.1 Input 1 / 4 scale feature map, 1 / 8 scale feature map and 1 / 16 scale feature map, and output the optimized feature map through the adaptive feature optimization module; The adaptive feature optimization module includes: a microscopic attention mechanism, a focused attention mechanism, a background suppression mechanism, and feature fusion; S3.2 Based on the 1 / 8 scale feature map, the semantic perception module outputs the aligned semantic features; The semantic awareness module includes: an encoder module, a semantic alignment module, a decoder, and a reconstruction loss; S3.3 Input the aligned semantic feature map and the optimized feature map, and output a feature map containing semantic information through the dynamic semantic fusion module; S3.4 Based on the feature map containing semantic information, local features are extracted through the descriptor header to generate descriptors for key points; S3.

5. Based on key point descriptors, descriptor supervision is performed through scalar loss calculation, and scalar loss values ​​are output. The scalar loss function can be expressed as: in, In feature map A, the first... The first feature and the first feature map B Similarity between feature descriptors Index A for feature map A Index B for feature map.

5. The SeViMatch feature matching method based on semantic-visual fusion according to claim 4, characterized in that, S3.1 specifically includes the following steps: S3.1.1 Based on the 1 / 4 scale feature map, an enhanced detail feature map is output through a microscopic attention mechanism; The expression for outputting the enhanced detail feature map is: in, To enhance the detailed feature map, This represents a detail attention map. Represents element-wise product. This represents the Sigmoid activation function. A 1 / 4 scale feature map; S3.1.2 Input a 1 / 8 scale feature map, and output a key focus attention map through a focus attention mechanism; The expression for outputting the key focus attention map is: in, To focus on attention maps, For key focus attention maps, A 1 / 8 scale feature map; S3.1.3 Input a 1 / 16 scale feature map, and output a suppressed background feature map through a background suppression mechanism; The expression for outputting the suppressed background feature map is: in, For background weighting, To suppress background feature maps, A feature map at a 1 / 16 scale; S3.1.

4. Based on the enhanced detail feature map, the key focus attention map, and the suppressed background feature map, perform feature fusion and output the optimized feature map; The expression for the optimized feature map is: in, These are, respectively, a uniform-scale enhanced detail feature map, a key focus attention map, and a background-suppressed feature map obtained through bilinear interpolation sampling. This is the optimized feature map.

6. The SeViMatch feature matching method based on semantic-visual fusion according to claim 4, characterized in that, S3.2 specifically includes the following steps: S3.2.1 Input a 1 / 8 scale feature map, and obtain a low-dimensional semantic feature map through the encoder module; The specific formula for obtaining the low-dimensional semantic feature map is as follows: in, Using 3×3 convolutional layers, the spatial resolution is reduced from... Compress to At the same time, the number of channels was adjusted to 128. For a 1×1 convolutional layer, the number of channels is compressed to 64; S3.2.2 Input a low-dimensional semantic feature map. After passing through the semantic alignment module, the decoder restores the low-dimensional semantic feature map to a feature map of the same scale as the input 1 / 8 scale feature map and outputs the aligned semantic feature map. The semantic alignment module includes a self-attention mechanism and a cross-attention mechanism; The expression for obtaining the aligned semantic feature map is: in, For transposed convolutional layers, the resolution is changed from... Restore to , The aligned semantic feature map; S3.2.

3. Based on the low-dimensional semantic feature map and the aligned semantic feature map, perform the reconstruction loss function calculation to obtain the reconstruction loss value; The reconstruction loss function is calculated as follows: in, Represents a low-dimensional semantic feature map. The aligned semantic feature map Indicates the total number of pixels. An index representing the total number of pixels.

7. The SeViMatch feature matching method based on semantic-visual fusion according to claim 4, characterized in that, S3.3 specifically includes the following steps: S3.3.1 Based on the aligned semantic features, use the K-Means algorithm for joint clustering to initialize key semantic features and output key semantic features; The key semantic features It contains 10 representative semantic cluster centers; S3.3.2 Based on key semantic features, the updated key semantic features are output through semantic-visual cross-attention; The updated key semantic features are calculated using the following expression: in, This represents the key semantic features within image A after enhancement. This represents the key semantic features within image B after enhancement. This represents the key semantic features of image A after the update. This represents the key semantic features of image B after the update. S3.3.3 Based on the updated key semantic features as keys and values, and the optimized feature map of the image as the query, semantic fusion is performed through a multi-head cross-attention mechanism to generate a feature map containing semantic information. The computational expression for generating a feature map containing semantic information is: In the formula, Key semantic features The optimized feature map is shown below, where Q represents the query, K represents the key, and V represents the value.

8. The SeViMatch feature matching method based on semantic-visual fusion according to claim 4, characterized in that, In step S3.4, the expression for generating the descriptor of the key points is: in, To generate descriptors for key points, A feature map containing semantic information. It consists of multiple 3×3 convolutional layers.

9. The SeViMatch feature matching method based on semantic-visual fusion according to claim 4, characterized in that, The key-point-based descriptor, after similarity calculation and matching confidence weight calculation and weighted adjustment, undergoes matching threshold filtering to output a stable set of matching point pairs. The similarity calculation formula is as follows: In the formula, The first feature point in each of the two images Second feature point The descriptor; The expression for calculating the matching confidence weight is: In the formula, The response intensity at the feature point For neighborhood feature consistency, The preset balance coefficient; The weighted adjustment calculation expression is: Combining the weighted similarity matrix with the preset matching threshold Filter out The candidate matching pairs are then verified through two-way validation, and finally a set of stable matching point pairs is output.

Images