A multi-branch fusion feature matching method based on wavelet pooling downsampling

Abstract

The present invention discloses a multi-branch fusion feature matching method based on wavelet pooling and downsampling, comprising the following steps: first constructing a data set containing several images; then building a model containing shallow, deep, and bottom-layer extraction modules, a spatial-channel feature fusion module, a double BN convolution activation unit, a key point heat map extraction module, and a feature matching module. Two images to be matched are input, converted into grayscale images, and then passed through each extraction module to obtain shallow, deep, and bottom-layer features; these features are fused to obtain a descriptor feature map, which is then passed through a double BN unit to obtain a reliability heat map; the grayscale image is expanded and combined with the initial features to generate a key point heat map; finally, the three types of heat maps and feature maps are input into a matching module to complete the matching. While maintaining lightweight and inference efficiency, the present invention enhances the spatial detail preservation, multi-scale context perception, and attention focusing capabilities of key point detection, improves detection accuracy and robustness, and has better generalization performance in complex scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of image feature matching, and in particular to a multi-branch fusion feature matching method based on wavelet pooling downsampling. Background Art

[0002] In today's computer vision field, feature matching technology, as a foundational core technology, plays a key role. It supports numerous cutting-edge fields, including 3D reconstruction, autonomous navigation, and augmented reality, and serves as the core channel for achieving cross-modal image alignment and geometric associations between the physical world and digital space. With the advancement of theoretical research, feature matching technology has transcended the limitations of traditional manual feature design. Leveraging a deeply differentiable architecture, it achieves continuous learning from low-level pixels to high-level semantics, laying a solid theoretical foundation for complex systems such as visual SLAM and multi-sensor fusion. Furthermore, research on matching optimization in dynamic environments and weakly textured scenes continues to deepen, revealing the generation mechanism of geometrically invariant features under non-ideal imaging conditions, paving the way for building robust visual perception models. This research has been validated through applications such as centimeter-level positioning for drones and AR virtual-reality fusion, promoting the cross-integration of differential geometry theory and deep learning paradigms. This provides methodological guidance for edge computing devices to overcome the "accuracy-efficiency" trade-off, and has significant fundamental scientific value for the development of strategic fields such as spatial computing and intelligent robotics.

[0003] Among numerous image matching networks, the Accelerated Lightweight Image Matching (Xfeat) network is specifically designed for resource-constrained devices. Its network architecture comprises a lightweight convolutional backbone network, which extracts features from the input image, downsamples it layer by layer, and increases the number of channels to form multi-scale features. Local features are then extracted in two branches: one to describe and generate a feature map; the other is a keypoint detection branch, which rapidly predicts a keypoint heatmap by unfolding the image and using convolution. Furthermore, a reliability branch outputs the matching confidence of each keypoint. During the matching phase, the network supports nearest neighbor searches for points in the feature map and uses a lightweight multi-layer perceptron module to perform pixel-level offset predictions on preliminary matches, enabling more accurate matching.

[0004] However, existing technologies have many shortcomings. When processing images with repetitive textures, large lighting variations, or complex backgrounds, key point detection suffers from poor stability and reduced repeatability. In areas with low texture or blurred edges, key point positioning is inaccurate, and feature matching is prone to failure. Moreover, in scenes with large scale variations or severe occlusion, the descriptor is weak in robustness, affecting the overall matching effect. These problems limit the application and development of image matching technology in complex scenes, and innovative solutions are urgently needed to improve the accuracy and stability of key point positioning. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the present invention provides a multi-branch fusion feature matching method based on wavelet pooling downsampling, which aims to solve the problems in the background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a multi-branch fusion feature matching method based on wavelet pooling downsampling, comprising the following steps:

[0007] Build a dataset; the dataset contains several images;

[0008] Constructing a feature matching model; the feature matching model includes a shallow extraction module, a deep extraction module, a bottom extraction module, a space-channel feature fusion module, a double BN convolution activation unit, a key point heat map extraction module and a feature matching module;

[0009] Input the image to be matched in the data set into the feature matching model. First, convert the image to be matched into a grayscale image. After the grayscale image passes through the shallow extraction module, the first feature and shallow features are obtained. After the shallow features pass through the deep extraction module, the second feature and deep features are obtained. After the second feature passes through the bottom extraction module, the bottom feature is obtained. There are two images to be matched.

[0010] The shallow features, deep features and bottom features are input into the spatial-channel feature fusion module for fusion to obtain the descriptor feature map;

[0011] After the descriptor feature map passes through the double BN convolution activation unit, the reliability heat map is obtained;

[0012] Input the grayscale image and the first feature into the key point heat map extraction module, and output the key point heat map;

[0013] The key point heat map, reliability heat map and descriptor features of the image to be matched Figure 1 The data are input into the feature matching module to complete the feature matching of the image to be matched.

[0014] Furthermore, the key point heat map K, reliability heat map R and descriptor feature map F of the image to be matched are input into the feature matching module together. The specific process of completing the feature matching of the image to be matched is as follows:

[0015] The key point heat map K and the reliability heat map R of the image to be matched are fused to obtain the final response map T;

[0016] Perform non-maximum suppression on the final response graph T and select the first k key points in the final response graph T after non-maximum suppression ; expressed as:

[0017] ;

[0018] Where, Indicates the selected key points; Represents non-maximum suppression operation; Indicates key points coordinates;

[0019] Based on key points , combined with the corresponding descriptor feature map F for bilinear interpolation, to obtain the preliminary feature vector , the initial feature vector Normalize and get the normalized feature vector ;

[0020] Based on key points and the normalized eigenvector Construct a set of key points and corresponding feature vectors of the image to be matched, expressed as:

[0021] ;

[0022] ;

[0023] Where, Represents the set of key points and corresponding feature vectors of the first image in the image to be matched; Indicates the first image in the image to be matched key points; Indicates the first image in the image to be matched The feature vector corresponding to the key points; Represents the set of key points and corresponding feature vectors of the second image in the image to be matched; Indicates the first image of the second image in the image to be matched key points; Indicates the first image of the second image in the image to be matched The feature vector corresponding to the key points;

[0024] Match the key points of the image to be matched, find the key point matching pairs that meet the preset conditions and put them into the final matching set M:

[0025] When each key point of the first image in the image to be matched When fixed, traverse all key points of the second image in the image to be matched , find the The biggest key point ; Represents the key points of the second image in the image to be matched The corresponding feature vector and the key point of the first image in the image to be matched Similarity matrix between corresponding eigenvectors;

[0026] When each key point in the second image in the image to be matched When fixed, traverse all key points of the first image in the image to be matched , find the The biggest key point ; Represents the key points of the first image in the image to be matched The corresponding feature vector and the key points of the second image in the image to be matched Similarity matrix between corresponding eigenvectors;

[0027] Will and Form key point matching pairs and calculate and Similarity matrix between corresponding eigenvectors , the similarity matrix The value of Compare and compare the values ​​that are greater than or equal to the preset similarity threshold The key point matching pairs are put into the final matching set M;

[0028] Finally, the coordinates of the corresponding key points in the key point matching pairs are output from the final matching set M.

[0029] Furthermore, the shallow extraction module includes a first basic block convolution, a first wavelet pooling downsampling module, a second basic block convolution, a second wavelet pooling downsampling module, a third basic block convolution, a fourth basic block convolution, a third wavelet pooling downsampling module, a fifth basic block convolution and a sixth basic block convolution; after the grayscale image passes through the shallow extraction module, the specific process of obtaining the first feature and the shallow feature is:

[0030] The grayscale image is sequentially passed through the first basic block convolution, the first wavelet pooling downsampling module, the second basic block convolution and the second wavelet pooling downsampling module to obtain the first feature ;

[0031] After the grayscale image passes through the average pooling layer and convolution operation, it is skipped and connected to the first feature The output of the third basic block convolution is then processed in sequence through the fourth basic block convolution, the third wavelet pooling downsampling module, the fifth basic block convolution and the sixth basic block convolution to obtain the shallow feature. .

[0032] Furthermore, the deep extraction module includes a fourth wavelet pooling downsampling module, a seventh basic block convolution, an eighth basic block convolution and a first lightweight dynamic upsampling module connected in sequence; after the shallow features pass through the deep extraction module, the specific process of obtaining the second features and the deep features is as follows:

[0033] Shallow features After passing through the fourth wavelet pooling downsampling module, the seventh basic block convolution, and the eighth basic block convolution in sequence, the second feature is obtained. After the second feature passes through the first lightweight dynamic upsampling module, the deep feature is obtained. .

[0034] Furthermore, the bottom layer extraction module is composed of a fifth wavelet pooling downsampling module, a ninth basic block convolution, a tenth basic block convolution, an eleventh basic block convolution and a second lightweight dynamic upsampling module connected in sequence; after the second feature passes through the bottom layer extraction module, the specific process of obtaining the bottom layer feature is as follows:

[0035] The output of the eighth basic block convolution, i.e. the second feature, is sequentially passed through the fifth wavelet pooling downsampling module, the ninth basic block convolution, the tenth basic block convolution, the eleventh basic block convolution and the second lightweight dynamic upsampling module to obtain the underlying feature. .

[0036] Furthermore, the spatial-channel feature fusion module is composed of a first spatial-channel attention module, a residual enhancement module, a second spatial-channel attention module, a first feature splicing module, and a first convolutional fusion module connected in sequence; the shallow features, deep features, and bottom-layer features are input into the spatial-channel feature fusion module for fusion, and the specific process of obtaining the descriptor feature map F is as follows:

[0037] The shallow features, deep features, and bottom features are first enhanced by the first spatial-channel attention module, then residual enhancement is performed by the residual enhancement module, and then secondary enhancement is performed by the second spatial-channel attention module. Finally, they are spliced ​​by the first feature splicing module and then fused in the first convolutional fusion module to obtain the descriptor feature map F; it can be expressed as:

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] Where, Represents features at different levels, including shallow features, deep features, and bottom features; represents channel attention; Represents the spatial attention mechanism; Represents a splicing operation; Represents the output of shallow features, deep features, and bottom-level features after passing through the first spatial-channel attention module; , Represents the output of shallow features after passing through the first spatial-channel attention module; Represents the output of the deep features after passing through the first spatial-channel attention module; Represents the output of the underlying features after passing through the first spatial-channel attention module; represents the output after splicing by the first feature splicing module; express Output after the second spatial-channel attention module; represents the point convolution weight; Represents the depth convolution kernel weight; represents the activation function;

[0043] The first spatial-channel attention module and the second spatial-channel attention module have the same structure.

[0044] Furthermore, the double BN convolution activation unit consists of two batch normalization layers, convolution layers and The specific process of obtaining the reliability heat map R after the descriptor feature map F passes through the double BN convolution activation unit is as follows: the descriptor feature map F passes through two batch normalization layers, a convolution layer and a After the activation function, the reliability heat map R is obtained.

[0045] Furthermore, the key point heat map extraction module is composed of an expansion module, a transposition module, a twelfth basic block convolution, a thirteenth basic block convolution, a two-dimensional convolution, a second feature splicing module, and a second convolution fusion module connected in sequence; the grayscale image and the first feature are input into the key point heat map extraction module together, and the specific process of outputting the key point heat map K is as follows:

[0046] Input the grayscale image into the expansion module for expansion to obtain an expanded image;

[0047] Expand the diagram After the transposed convolution, the twelfth basic block convolution, the thirteenth basic block convolution and a two-dimensional convolution, the preliminary key map is obtained. ;

[0048] The preliminary key diagram and the first feature After being input into the second feature splicing module for splicing, they are processed by the second convolution fusion module to obtain the key point heat map K.

[0049] Furthermore, the first basic block convolution, the second basic block convolution, the third basic block convolution, the fourth basic block convolution, the fifth basic block convolution, the sixth basic block convolution, the seventh basic block convolution, the eighth basic block convolution, the ninth basic block convolution, the tenth basic block convolution, the eleventh basic block convolution, the twelfth basic block convolution and the thirteenth basic block convolution have the same structure, and are all composed of sequentially connected convolution layers, batch normalization layers and Activation function composition.

[0050] Furthermore, the first wavelet pooling downsampling module, the second wavelet pooling downsampling module, the third wavelet pooling downsampling module, the fourth wavelet pooling downsampling module and the fifth wavelet pooling downsampling module have the same structure, and all include a wavelet transform module, an average pooling module, a third feature splicing module, a convolution layer, a third space-channel attention module, a batch normalization layer and a ReLU activation function; the processing flow of the first wavelet pooling downsampling module is: the output of the first basic block convolution is respectively input into the wavelet transform module and the average pooling module, and the output of the wavelet transform module and the average pooling module is spliced ​​through the third feature splicing module and then passed through the convolution layer, the third space-channel attention module, the batch normalization layer and the ReLU activation function in sequence to obtain the output of the first wavelet pooling downsampling module.

[0051] Compared with the existing technology, the present invention has the following beneficial effects:

[0052] (1) This invention introduces a wavelet pooling-based downsampling, a lightweight dynamic upsampling mechanism, and a new spatial and channel feature fusion. Compared with the existing network structure, this design significantly enhances the spatial detail preservation capability, multi-scale context perception capability, and attention focusing capability in the key point detection process while maintaining the model lightweight and inference efficiency, effectively improving the detection accuracy and robustness, especially showing stronger generalization performance in complex backgrounds, posture changes, or occlusion scenes.

[0053] (2) This paper designs a novel spatial-channel feature fusion module. By combining multi-scale feature modeling and regional attention mechanism, it effectively enhances the expression ability of key structural areas while keeping the model lightweight. It captures contextual information of different scales through residual paths and strengthens the expression of significant areas by combining attention mechanism, which significantly improves the accuracy and robustness in key point detection tasks, especially in complex scenarios, with better adaptability and heat map quality.

[0054] (3) The present invention designs a wavelet pooling downsampling module, which downsamples the feature map through two paths: wavelet transformation and average pooling. Wavelet transformation downsampling retains multi-scale frequency detail information, and average pooling downsampling extracts stable overall statistical features. The results of the two paths are fused through convolution and enhanced through attention, so as to enhance the network's dual expression ability of structure and semantics, improve the overall adaptability and information acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a framework diagram of the feature matching model of the present invention.

[0056] Figure 2 Schematic diagram of the spatial-channel feature fusion module structure of the present invention.

[0057] Figure 3 Schematic diagram of the structure of the wavelet pooling downsampling module of the present invention. DETAILED DESCRIPTION

[0058] The present invention provides a technical solution: a multi-branch fusion feature matching method based on wavelet pooling downsampling, comprising the following steps:

[0059] Step S1: Construct a dataset; the dataset contains several images.

[0060] The MegaDepth dataset is used. The dataset collects large-scale, multi-view images from the internet and generates dense depth maps using structure-from-motion and multi-view stereo techniques for training and evaluating monocular depth estimation tasks. The dataset contains 1.07 million internet images of 196 3D landmark models, covering a variety of indoor and outdoor scenes (such as buildings, street scenes, and natural landscapes).

[0061] Step S2: Construct a feature matching model, such as Figure 1 As shown; the feature matching model includes a shallow extraction module, a deep extraction module, a bottom extraction module, a space-channel feature fusion module, a double BN convolution activation unit, a key point heat map extraction module and a feature matching module.

[0062] Step S3: Input the images to be matched (two images) in the data set into the feature matching model. First, convert the images to be matched into grayscale images. After the grayscale images pass through the shallow extraction module, the first features and shallow features are obtained. After the shallow features pass through the deep extraction module, the second features and deep features are obtained. After the second features pass through the bottom extraction module, the bottom features are obtained.

[0063] Step S4: Input the shallow features, deep features and bottom features into the space-channel feature fusion module for fusion to obtain the descriptor feature map F.

[0064] Step S5: After the descriptor feature map F passes through the double BN convolution activation unit, the reliability heat map R is obtained.

[0065] Step S6: Input the grayscale image and the first feature into the key point heat map extraction module, and output the key point heat map K.

[0066] Step S7: The key point heat map K, the reliability heat map R and the descriptor feature map F of the image to be matched are input into the feature matching module to complete the feature matching of the image to be matched.

[0067] Among them, the shallow extraction module includes the first basic block convolution (step size is 1), the first wavelet pooling downsampling module, the second basic block convolution, the second wavelet pooling downsampling module, the third basic block convolution, the fourth basic block convolution, the third wavelet pooling downsampling module, the fifth basic block convolution and the sixth basic block convolution; wherein, after the grayscale image passes through the shallow extraction module, the specific process of obtaining the first feature and the shallow feature is as follows:

[0068] Grayscale image After the first basic block convolution (step size is 1), the first wavelet pooling downsampling module, the second basic block convolution and the second wavelet pooling downsampling module, the first feature is obtained. ; can be expressed as:

[0069] ;

[0070] Where, Represents the first basic block convolution; represents the second basic convolution; represents the first wavelet pooling downsampling module; Represents the second wavelet pooling downsampling module.

[0071] Finally, the grayscale image is skipped after the average pooling layer and convolution operation, and is connected to the first feature The output of the third basic block convolution is then processed in sequence through the fourth basic block convolution, the third wavelet pooling downsampling module, the fifth basic block convolution and the sixth basic block convolution to obtain the shallow feature. ; can be expressed as:

[0072] ;

[0073] Where, represents the average pooling layer; Represents a 1×1 convolution operation; Represents the third basic block convolution; Represents the fourth basic block convolution; represents the third wavelet pooling downsampling module; Represents the fifth basic block convolution; Represents the sixth basic block convolution.

[0074] Among them, the deep extraction module includes the fourth wavelet pooling downsampling module, the seventh basic block convolution, the eighth basic block convolution and the first lightweight dynamic upsampling module connected in sequence; wherein, after the shallow features pass through the deep extraction module, the specific process of obtaining the second features and the deep features is as follows:

[0075] Shallow features After passing through the fourth wavelet pooling downsampling module, the seventh basic block convolution, and the eighth basic block convolution in sequence, the second feature is obtained. After the second feature passes through the first lightweight dynamic upsampling module, the deep feature is obtained. ; can be expressed as:

[0076] ;

[0077] Where, Represents the fourth wavelet pooling downsampling module; Represents the seventh basic block convolution; Represents the eighth basic block convolution; Represents the first lightweight dynamic upsampling module.

[0078] Among them, the bottom extraction module is composed of the fifth wavelet pooling downsampling module, the ninth basic block convolution, the tenth basic block convolution, the eleventh basic block convolution and the second lightweight dynamic upsampling module connected in sequence; after the second feature passes through the bottom extraction module, the specific process of obtaining the bottom feature is as follows:

[0079] The output of the eighth basic block convolution (The second feature) After passing through the fifth wavelet pooling downsampling module, the ninth basic block convolution, the tenth basic block convolution, the eleventh basic block convolution and the second lightweight dynamic upsampling module, the underlying feature is obtained. ; can be expressed as:

[0080] ;

[0081] Where, Represents the fifth wavelet pooling downsampling module; Represents the ninth basic block convolution; Represents the tenth basic block convolution; represents the eleventh basic block convolution; Represents the second lightweight dynamic upsampling module.

[0082] like Figure 2As shown in Figure 1, the spatial-channel feature fusion module consists of a first spatial-channel attention module (Convolutional Block Attention Module, CBAM), a residual enhancement module, a second spatial-channel attention module, a first feature splicing module, and a first convolutional fusion module connected in sequence; wherein, the shallow features, deep features, and bottom-layer features are input into the spatial-channel feature fusion module for fusion, and the specific process of obtaining the descriptor feature map F is as follows:

[0083] The shallow features, deep features, and bottom features are first enhanced by the first spatial-channel attention module, then residual enhancement is performed by the residual enhancement module, and then secondary enhancement is performed by the second spatial-channel attention module. Finally, they are spliced ​​by the first feature splicing module and then fused in the first convolutional fusion module to obtain the descriptor feature map F, which can be expressed as:

[0084] ;

[0085] ;

[0086] ;

[0087] ;

[0088] Where, Represents features at different levels, including shallow features, deep features, and bottom features; represents channel attention; Represents the spatial attention mechanism; Represents a splicing operation; Represents the output of shallow features, deep features, and bottom-level features after passing through the first spatial-channel attention module; , Represents the output of shallow features after passing through the first spatial-channel attention module; Represents the output of the deep features after passing through the first spatial-channel attention module; Represents the output of the underlying features after passing through the first spatial-channel attention module; represents the output after splicing by the first feature splicing module; express Output after the second spatial-channel attention module; represents the point convolution weight; Represents the depth convolution kernel weight; Represents the activation function.

[0089] Among them, the first space-channel attention module and the second space-channel attention module have the same structure.

[0090] Among them, the double BN convolution activation unit consists of two batch normalization layers, convolution layers and The activation function is composed of: After the descriptor feature map F passes through the double BN convolution activation unit, the specific process of obtaining the reliability heat map R is as follows:

[0091] The descriptor feature map F is passed through two batch normalization layers, a convolution layer and a After the activation function, the reliability heat map R is obtained; it can be expressed as:

[0092] ;

[0093] Where, represents the batch normalization layer; represents the convolutional layer; express Activation function.

[0094] The key point heat map extraction module consists of an expansion module, a transposition module, a twelfth basic block convolution, a thirteenth basic block convolution, a two-dimensional convolution, a second feature splicing module, and a second convolution fusion module connected in sequence. The grayscale image and the first feature are input into the key point heat map extraction module together, and the specific process of outputting the key point heat map K is as follows:

[0095] Grayscale image Input the expansion module to expand and get the expansion diagram :

[0096] ;

[0097] Where, Represents a two-dimensional expansion function for grayscale images Perform two-dimensional expansion processing.

[0098] Expand the diagram After the transposed convolution, the twelfth basic block convolution, the thirteenth basic block convolution and a two-dimensional convolution, the preliminary key map is obtained. :

[0099] ;

[0100] Where, represents transposed convolution; represents the twelfth basic block convolution; represents the thirteenth basic block convolution; Represents a two-dimensional convolution.

[0101] The preliminary key diagram and the first feature After being input into the second feature splicing module for splicing, it is processed by the second convolution fusion module to obtain the key point heat map K; it can be expressed as:

[0102] .

[0103] Among them, the key point heat map K, reliability heat map R and descriptor feature map F of the image to be matched are input into the feature matching module together. The specific process of completing the feature matching of the image to be matched is:

[0104] The key point heat map K and the reliability heat map R of the image to be matched are fused to obtain the final response map T, which can be expressed as:

[0105] ;

[0106] Where, Represents point-wise multiplication.

[0107] Perform non-maximum suppression on the final response graph T and select the first k key points in the final response graph T after non-maximum suppression ; can be expressed as:

[0108] ;

[0109] Where, Indicates the selected key points; Represents the non-maximum suppression (NMS) operation; Indicates key points 's coordinates.

[0110] Based on key points , combined with the corresponding descriptor feature map F for bilinear interpolation, to obtain the preliminary feature vector , the initial feature vector Normalize and get the normalized feature vector , which can be expressed as:

[0111] ;

[0112] ;

[0113] Where, Represents bilinear interpolation.

[0114] Based on key points and the normalized eigenvector Construct a set of key points and corresponding feature vectors of the image to be matched; it can be expressed as:

[0115] ;

[0116] ;

[0117] Where, Represents the set of key points and corresponding feature vectors of the first image in the image to be matched; Indicates the first image in the image to be matched key points; Indicates the first image in the image to be matched The feature vector corresponding to the key points; Represents the set of key points and corresponding feature vectors of the second image in the image to be matched; Indicates the first image of the second image in the image to be matched key points; Indicates the first image of the second image in the image to be matched The feature vector corresponding to each key point.

[0118] Calculate the cosine similarity of the feature vectors of the images to be matched one by one, and calculate the similarity matrix between the feature vectors of the images to be matched :

[0119] ;

[0120] Where cos is the cosine similarity calculation.

[0121] Match the key points of the image to be matched, find the key point matching pairs that meet the preset conditions and put them into the final matching set M; specifically:

[0122] When each key point of the first image in the image to be matched When fixed, traverse all key points of the second image in the image to be matched , find the The biggest key point ; Represents the key points of the second image in the image to be matched The corresponding feature vector and the key point of the first image in the image to be matched The similarity matrix between corresponding eigenvectors.

[0123] When each key point in the second image in the image to be matched When fixed, traverse all key points of the first image in the image to be matched , find the The biggest key point ; Represents the key points of the first image in the image to be matched The corresponding feature vector and the key points of the second image in the image to be matched The similarity matrix between corresponding eigenvectors.

[0124] Will and Form key point matching pairs and calculate and Similarity matrix between corresponding eigenvectors , the similarity matrix The value of Compare and compare the values ​​that are greater than or equal to the preset similarity threshold The key point matching pairs are put into the final matching set M.

[0125] Finally, the coordinates of the corresponding key points in the key point matching pairs are output from the final matching set M.

[0126] Among them, the first basic block convolution, the second basic block convolution, the third basic block convolution, the fourth basic block convolution, the fifth basic block convolution, the sixth basic block convolution, the seventh basic block convolution, the eighth basic block convolution, the ninth basic block convolution, the tenth basic block convolution, the eleventh basic block convolution, the twelfth basic block convolution and the thirteenth basic block convolution have the same structure, which is composed of sequentially connected convolution layers, batch normalization layers and Activation function composition.

[0127] like Figure 3 As shown in the figure, the first wavelet pooling downsampling module, the second wavelet pooling downsampling module, the third wavelet pooling downsampling module, the fourth wavelet pooling downsampling module and the fifth wavelet pooling downsampling module have the same structure, and all include a wavelet transform module, an average pooling module, a third feature splicing module, a convolution layer (1×1), a third space-channel attention module, a batch normalization layer and a ReLU activation function; the processing flow of the first wavelet pooling downsampling module is as follows: the output of the convolution of the first basic block is input into the wavelet transform module and the average pooling module respectively, the output of the wavelet transform module and the average pooling module is spliced ​​through the third feature splicing module, and then passes through the convolution layer (1×1), the third space-channel attention module, the batch normalization layer and the ReLU activation function in sequence to obtain the output of the first wavelet pooling downsampling module.

[0128] The first lightweight dynamic upsampling module (DySample) and the second lightweight dynamic upsampling module have the same structure.

[0129] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A multi-branch fusion feature matching method based on wavelet pooling downsampling, characterized in that: The steps include: Build a dataset; the dataset contains several images; Constructing a feature matching model; the feature matching model includes a shallow extraction module, a deep extraction module, a bottom extraction module, a space-channel feature fusion module, a double BN convolution activation unit, a key point heat map extraction module and a feature matching module; Input the image to be matched in the data set into the feature matching model. First, convert the image to be matched into a grayscale image. After the grayscale image passes through the shallow extraction module, the first feature and shallow features are obtained. After the shallow features pass through the deep extraction module, the second feature and deep features are obtained. After the second feature passes through the bottom extraction module, the bottom feature is obtained. There are two images to be matched. The shallow features, deep features and bottom features are input into the spatial-channel feature fusion module for fusion to obtain the descriptor feature map; After the descriptor feature map passes through the double BN convolution activation unit, the reliability heat map is obtained; Input the grayscale image and the first feature into the key point heat map extraction module, and output the key point heat map; The key point heat map, reliability heat map and descriptor feature map of the image to be matched are input into the feature matching module to complete the feature matching of the image to be matched; The key point heat map K, reliability heat map R and descriptor feature map F of the image to be matched are input into the feature matching module together. The specific process of completing the feature matching of the image to be matched is as follows: The key point heat map K and the reliability heat map R of the image to be matched are fused to obtain the final response map T; Perform non-maximum suppression on the final response graph T and select the first k key points in the final response graph T after non-maximum suppression ; expressed as: ； Where, Indicates the selected key points; Represents non-maximum suppression operation; Represents key points coordinates of Based on key points , combined with the corresponding descriptor feature map F for bilinear interpolation, to obtain the preliminary feature vector , the initial feature vector Normalize and get the normalized feature vector ; Based on key points and the normalized eigenvector Construct a set of key points and corresponding feature vectors of the image to be matched, expressed as: ； ； Where, Represents the set of key points and corresponding feature vectors of the first image in the image to be matched; Indicates the first image in the image to be matched key points; Indicates the first image in the image to be matched The feature vector corresponding to the key points; Represents the set of key points and corresponding feature vectors of the second image in the image to be matched; Indicates the first image of the second image in the image to be matched key points; Indicates the first image of the second image in the image to be matched The feature vector corresponding to the key points; Match the key points of the image to be matched, find the key point matching pairs that meet the preset conditions and put them into the final matching set M: When each key point of the first image in the image to be matched When fixed, traverse all key points of the second image in the image to be matched , find the The biggest key point ; Represents the key points of the second image in the image to be matched The corresponding feature vector and the key point of the first image in the image to be matched Similarity matrix between corresponding eigenvectors; When each key point in the second image in the image to be matched When fixed, traverse all key points of the first image in the image to be matched , find the The biggest key point ; Represents the key points of the first image in the image to be matched The corresponding feature vector and the key points of the second image in the image to be matched Similarity matrix between corresponding eigenvectors; Will and Form key point matching pairs and calculate and Similarity matrix between corresponding eigenvectors , the similarity matrix The value of Compare and compare the values ​​that are greater than or equal to the preset similarity threshold The key point matching pairs are put into the final matching set M; Finally, the coordinates of the corresponding key points in the key point matching pairs are output from the final matching set M; The shallow layer extraction module includes a first basic block convolution, a first wavelet pooling downsampling module, a second basic block convolution, a second wavelet pooling downsampling module, a third basic block convolution, a fourth basic block convolution, a third wavelet pooling downsampling module, a fifth basic block convolution and a sixth basic block convolution; after the grayscale image passes through the shallow layer extraction module, the specific process of obtaining the first feature and the shallow layer feature is as follows: The grayscale image is sequentially passed through the first basic block convolution, the first wavelet pooling downsampling module, the second basic block convolution and the second wavelet pooling downsampling module to obtain the first feature ; After the grayscale image passes through the average pooling layer and convolution operation, it is skipped and connected to the first feature Input together into the third basic block convolution for processing; The output of the third basic block convolution is then passed through the fourth basic block convolution, the third wavelet pooling downsampling module, the fifth basic block convolution and the sixth basic block convolution to obtain the shallow features. ; The deep extraction module includes a fourth wavelet pooling downsampling module, a seventh basic block convolution, an eighth basic block convolution and a first lightweight dynamic upsampling module connected in sequence; after the shallow features pass through the deep extraction module, the specific process of obtaining the second features and the deep features is as follows: Shallow features After passing through the fourth wavelet pooling downsampling module, the seventh basic block convolution, and the eighth basic block convolution in sequence, the second feature is obtained. After the second feature passes through the first lightweight dynamic upsampling module, the deep feature is obtained. ; The bottom layer extraction module is composed of a fifth wavelet pooling downsampling module, a ninth basic block convolution, a tenth basic block convolution, an eleventh basic block convolution and a second lightweight dynamic upsampling module connected in sequence; after the second feature passes through the bottom layer extraction module, the specific process of obtaining the bottom layer feature is as follows: The output of the eighth basic block convolution, i.e. the second feature, is sequentially passed through the fifth wavelet pooling downsampling module, the ninth basic block convolution, the tenth basic block convolution, the eleventh basic block convolution and the second lightweight dynamic upsampling module to obtain the underlying feature. .

2. The multi-branch fusion feature matching method based on wavelet pooling downsampling according to claim 1 is characterized in that: The spatial-channel feature fusion module is composed of a first spatial-channel attention module, a residual enhancement module, a second spatial-channel attention module, a first feature splicing module, and a first convolutional fusion module connected in sequence. The shallow features, deep features, and bottom features are input into the spatial-channel feature fusion module for fusion. The specific process of obtaining the descriptor feature map F is as follows: The shallow features, deep features, and bottom features are first enhanced by the first spatial-channel attention module, then residual enhancement is performed by the residual enhancement module, and then secondary enhancement is performed by the second spatial-channel attention module. Finally, they are spliced ​​by the first feature splicing module and then fused in the first convolutional fusion module to obtain the descriptor feature map F; it can be expressed as: ； ； ； ； Where, Represents features at different levels, including shallow features, deep features, and bottom features; represents channel attention; Represents the spatial attention mechanism; Represents a splicing operation; Represents the output of shallow features, deep features, and bottom-level features after passing through the first spatial-channel attention module; , Represents the output of shallow features after passing through the first spatial-channel attention module; Represents the output of the deep features after passing through the first spatial-channel attention module; Represents the output of the underlying features after passing through the first spatial-channel attention module; represents the output after splicing by the first feature splicing module; express Output after the second spatial-channel attention module; represents the point convolution weight; Represents the depth convolution kernel weight; represents the activation function; The first spatial-channel attention module and the second spatial-channel attention module have the same structure.

3. The multi-branch fusion feature matching method based on wavelet pooling downsampling according to claim 2 is characterized in that: The double BN convolution activation unit consists of two batch normalization layers, convolution layers and The specific process of obtaining the reliability heat map R after the descriptor feature map F passes through the double BN convolution activation unit is as follows: the descriptor feature map F passes through two batch normalization layers, a convolution layer and a After the activation function, the reliability heat map R is obtained.

4. The multi-branch fusion feature matching method based on wavelet pooling downsampling according to claim 3 is characterized by: The key point heat map extraction module is composed of an expansion module, a transposition module, a twelfth basic block convolution, a thirteenth basic block convolution, a two-dimensional convolution, a second feature splicing module, and a second convolution fusion module connected in sequence. The grayscale image and the first feature are input into the key point heat map extraction module together, and the specific process of outputting the key point heat map K is as follows: Input the grayscale image into the expansion module for expansion to obtain an expanded image; Expand the diagram After the transposed convolution, the twelfth basic block convolution, the thirteenth basic block convolution and a two-dimensional convolution, the preliminary key map is obtained. ; The preliminary key diagram and the first feature After being input into the second feature splicing module for splicing, they are processed by the second convolution fusion module to obtain the key point heat map K.

5. The multi-branch fusion feature matching method based on wavelet pooling downsampling according to claim 4 is characterized in that: The first basic block convolution, the second basic block convolution, the third basic block convolution, the fourth basic block convolution, the fifth basic block convolution, the sixth basic block convolution, the seventh basic block convolution, the eighth basic block convolution, the ninth basic block convolution, the tenth basic block convolution, the eleventh basic block convolution, the twelfth basic block convolution and the thirteenth basic block convolution have the same structure, which is composed of sequentially connected convolution layers, batch normalization layers and Activation function composition.

6. The multi-branch fusion feature matching method based on wavelet pooling downsampling according to claim 5 is characterized in that: The first wavelet pooling downsampling module, the second wavelet pooling downsampling module, the third wavelet pooling downsampling module, the fourth wavelet pooling downsampling module and the fifth wavelet pooling downsampling module have the same structure, and all include a wavelet transform module, an average pooling module, a third feature splicing module, a convolution layer, a third space-channel attention module, a batch normalization layer and a ReLU activation function; the processing flow of the first wavelet pooling downsampling module is: the output of the first basic block convolution is respectively input into the wavelet transform module and the average pooling module, the output of the wavelet transform module and the average pooling module is spliced ​​through the third feature splicing module and then passed through the convolution layer, the third space-channel attention module, the batch normalization layer and the ReLU activation function in sequence to obtain the output of the first wavelet pooling downsampling module.

Images