A feature matching method, device, equipment and medium

Abstract

The present application discloses a feature matching method, device, equipment and medium, which relates to the field of computer vision, including: obtaining an image pair to be matched; the image pair includes a first image and a second image; extracting key points and feature descriptors from each image in the image pair, and fusing the position code calculated based on the key points with the feature descriptor to obtain a first descriptor; performing a linear transformation on the first descriptor, and using the frequency domain attention weight calculated based on the linear transformation result to weight the frequency domain information of the first descriptor to obtain a second descriptor; performing channel division on the second descriptor to obtain a plurality of branches, and extracting spatial domain features from each branch, and fusing the spatial domain features of each branch to obtain a third descriptor; calculating a matching probability matrix based on the third descriptor corresponding to the first image and the third descriptor corresponding to the second image, so as to determine the best matching point from the matching probability matrix. The accuracy of feature matching can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a feature matching method, device, equipment and medium. Background Art

[0002] Feature matching plays an important role in the field of computer vision. Its core task is to achieve the correspondence between different images or image sequences by comparing local feature points in the image. Feature matching technology is widely used in many scenarios, including image stitching, object recognition, 3D reconstruction, and stereo vision. In the research of feature matching, existing methods can be mainly divided into two categories: traditional algorithms and deep learning-based methods.

[0003] Among them, traditional feature matching methods rely on manually designed feature extraction algorithms, which usually include two types: local feature descriptor-based and global image feature-based. However, due to the complexity of real-world scenes, traditional algorithms are not well suited for this task. Deep learning-based methods can be mainly divided into feature point-based methods, descriptor-based methods, graph matching-based methods, and multi-scale feature matching methods. However, these methods extract image features and perform subsequent matching in the spatial domain, which makes it difficult to fully utilize image features, such as frequency domain features.

[0004] Frequency domain features are the representation of information in an image in the frequency domain space. By performing Fourier transform on an image, the image data originally in the spatial domain can be converted into frequency domain data. In the frequency domain, different frequency components of an image can reflect different image characteristics. High-frequency components usually correspond to drastically changing parts such as edges and details in the image, while low-frequency components reflect the overall structure, smooth areas and other slowly changing parts of the image. Therefore, frequency domain features can not only characterize local details of an image, but also capture global structural information. However, existing methods do not fully utilize the frequency domain features of an image, but directly use the features of the spatial domain for coarse-grained feature matching. Therefore, in some complex environments, the accuracy of feature matching is often insufficient, and there are also problems of poor robustness and high computational overhead.

[0005] In summary, how to make full use of the frequency domain features and spatial domain features of the image to improve the accuracy of feature matching is a problem that needs to be solved. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a feature matching method, device, equipment and medium, which can make full use of the frequency domain features and spatial domain features of the image to improve the accuracy of feature matching. The specific scheme is as follows:

[0007] In a first aspect, the present application discloses a feature matching method, comprising:

[0008] Acquire an image pair to be matched; wherein the image pair includes a first image and a second image;

[0009] Extracting key points and feature descriptors from each image in the image pair, and fusing the position code calculated based on the key points with the feature descriptor to obtain a first descriptor;

[0010] Performing a linear transformation on the first descriptor, and weighting the first descriptor with frequency domain information using a frequency domain attention weight calculated based on the linear transformation result to obtain a second descriptor;

[0011] Dividing the second descriptor into several branches through channels, extracting spatial domain features from each branch, and fusing the spatial domain features of each branch to obtain a third descriptor;

[0012] A matching probability matrix is ​​calculated based on the third descriptor corresponding to the first image and the third descriptor corresponding to the second image, so as to determine the best matching point from the matching probability matrix.

[0013] Optionally, extracting key points and feature descriptors from each image in the image pair, and fusing a position code calculated based on the key points with the feature descriptor to obtain a first descriptor, includes:

[0014] Performing grayscale transformation on each image in the image pair to obtain a corresponding grayscale image;

[0015] Extracting key points and feature descriptors of each grayscale image using a feature detection algorithm, and obtaining the confidence of each key point;

[0016] Concatenating the dimensions corresponding to the key points and the confidence levels, and inputting the concatenated results into a preset multi-layer feedforward network to calculate position encoding;

[0017] The position code is fused to the feature descriptor to obtain a first descriptor.

[0018] Optionally, performing a linear transformation on the first descriptor, and weighting the first descriptor with frequency domain information using a frequency domain attention weight calculated based on a linear transformation result to obtain a second descriptor, includes:

[0019] Performing a linear transformation on the first descriptor using a predefined linear transformation matrix to obtain a query vector, a key vector, and a value vector;

[0020] Decomposing the query vector using a high-pass filter and a low-pass filter respectively to obtain corresponding high-frequency features and low-frequency features, and fusing the high-frequency features and the low-frequency features to obtain a frequency domain total feature;

[0021] Calculating a frequency attention score based on the frequency domain total feature and the key vector, and converting the frequency attention score into a frequency domain attention weight;

[0022] The frequency domain attention weight and the value vector are used to obtain a second descriptor weighted by the frequency domain information.

[0023] Optionally, the decomposing the query vector by using a high-pass filter and a low-pass filter to obtain corresponding high-frequency features and low-frequency features respectively includes:

[0024] Using high-pass filter convolution kernels of different sizes to perform high-pass filtering on the query vector to obtain corresponding high-frequency components, and fusing the high-frequency components to obtain high-frequency features;

[0025] The query vector is low-pass filtered using average pooling functions of different sizes to obtain corresponding low-frequency components, and bilinear interpolation is performed on each of the low-frequency components, and the interpolated low-frequency components are fused to obtain low-frequency features.

[0026] Optionally, dividing the second descriptor into channels to obtain a plurality of branches, and extracting spatial domain features from each branch, includes:

[0027] Dividing the second descriptor into a plurality of branches evenly based on the channel dimension, and numbering each branch according to a preset numbering sequence;

[0028] Based on the numbering results, target convolution kernels of different sizes are assigned to each branch, and based on the preset feature extraction rules, the corresponding target convolution kernels are used to extract spatial domain features on each branch;

[0029] Among them, the preset feature extraction rule includes a rule for extracting features of the branch corresponding to the current number and the spatial domain features corresponding to the previous number using the target convolution kernel corresponding to the current number to obtain the spatial domain features of the branch corresponding to the current number.

[0030] Optionally, the calculating a matching probability matrix based on a third descriptor corresponding to the first image and a third descriptor corresponding to the second image includes:

[0031] Normalizing a third descriptor corresponding to the first image to obtain a first processing result;

[0032] Normalizing the third descriptor corresponding to the second image to obtain a second processing result;

[0033] A target correlation matrix is ​​obtained based on the first processing result and the second processing result, and a matching probability matrix is ​​obtained by performing a preset number of operations on the target correlation matrix using a preset activation function.

[0034] Optionally, determining the best matching point from the matching probability matrix includes:

[0035] Determine the maximum value of each row and each column from the matching probability matrix;

[0036] The best matching point is determined based on the maximum value in each row and column.

[0037] In a second aspect, the present application discloses a feature matching device, comprising:

[0038] An image acquisition module, used to acquire an image pair to be matched; wherein the image pair includes a first image and a second image;

[0039] A feature extraction module, used for extracting key points and feature descriptors from each image in the image pair, and fusing a position code calculated based on the key points with the feature descriptors to obtain a first descriptor;

[0040] A frequency domain weighting module, used to perform a linear transformation on the first descriptor, and use the frequency domain attention weight calculated based on the linear transformation result to weight the first descriptor with frequency domain information to obtain a second descriptor;

[0041] A spatial domain fusion module, used for performing channel division on the second descriptor to obtain a plurality of branches, extracting spatial domain features from each branch, and fusing the spatial domain features of each branch to obtain a third descriptor;

[0042] The best matching point determination module is used to calculate a matching probability matrix based on the third descriptor corresponding to the first image and the third descriptor corresponding to the second image, so as to determine the best matching point from the matching probability matrix.

[0043] In a third aspect, the present application discloses an electronic device, comprising:

[0044] Memory, used to store computer programs;

[0045] The processor is used to execute the computer program to implement the steps of the feature matching method disclosed above.

[0046] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the feature matching method disclosed above are implemented.

[0047] It can be seen that the present application obtains an image pair to be matched; wherein the image pair includes a first image and a second image; extracts key points and feature descriptors for each image in the image pair, and fuses the position code calculated based on the key points with the feature descriptor to obtain a first descriptor; performs a linear transformation on the first descriptor, and uses the frequency domain attention weight calculated based on the linear transformation result to weight the frequency domain information of the first descriptor to obtain a second descriptor; divides the second descriptor into channels to obtain a plurality of branches, extracts spatial domain features for each branch, and fuses the spatial domain features of each branch to obtain a third descriptor; calculates a matching probability matrix based on the third descriptor corresponding to the first image and the third descriptor corresponding to the second image, so as to determine the best matching point from the matching probability matrix.

[0048] Beneficial effect: After obtaining the image pair to be matched, the present application first needs to extract key points and feature descriptors for each image in the image pair, and fuse the position code calculated based on the key points with the feature descriptor to obtain the first descriptor; wherein the image pair includes the first image and the second image to be matched. Further, the present application needs to perform a linear transformation on the first descriptor, and weight the frequency domain information of the first descriptor based on the frequency domain attention weight calculated from the linear transformation result to obtain the second descriptor, that is, the present application needs to use the linear transformation result to weight the frequency domain information of the first descriptor, so as to make full use of the frequency domain features of the image. Then, the second descriptor after frequency domain weighting is divided into several branches, and the spatial domain features are extracted from each branch, and the spatial domain features of each branch are fused to obtain the third descriptor. In this way, the third descriptor includes both the frequency domain features and the spatial domain features of the image, and finally the best matching point is determined by calculating the matching probability moment obtained by the third descriptor corresponding to the first image and the third descriptor corresponding to the second image. Therefore, the present application can realize the adaptive fusion of spatial domain features and frequency domain features, thereby utilizing these features for robust feature matching and improving the accuracy of feature matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0050] Figure 1 A flow chart of a feature matching method disclosed in this application;

[0051] Figure 2A flowchart of an adaptive feature matching network disclosed in this application;

[0052] Figure 3 A specific feature matching method flow chart disclosed in this application;

[0053] Figure 4 A flowchart of a frequency attention module disclosed in this application;

[0054] Figure 5 This is a flow chart of a spatial domain feature enhancement module disclosed in this application;

[0055] Figure 6 This is a schematic diagram of the structure of a feature matching device disclosed in this application;

[0056] Figure 7 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0058] Currently, existing methods do not fully utilize the frequency domain features of images, but directly use the features of the spatial domain for coarse-grained feature matching. Therefore, in some complex environments, the accuracy of feature matching is often insufficient, and there are also problems of poor robustness and high computational overhead. To this end, the embodiments of the present application disclose a feature matching method, device, equipment and medium that can fully utilize the frequency domain features and spatial domain features of images to improve the accuracy of feature matching.

[0059] See also Figure 1 and Figure 2 As shown, the embodiment of the present application discloses a feature matching method, which includes:

[0060] Step S11: Acquire an image pair to be matched; wherein the image pair includes a first image and a second image.

[0061] In this embodiment, firstly, an image pair to be matched is obtained, that is, a first image I1 and a second image I2 to be matched are obtained.

[0062] Step S12: extracting key points and feature descriptors from each image in the image pair, and fusing the position code calculated based on the key points with the feature descriptor to obtain a first descriptor.

[0063] In this embodiment, after obtaining the image pair to be matched, it is first necessary to extract key points and feature descriptors from each image in the image pair, and fuse the position code calculated based on the key points with the feature descriptor to obtain a first descriptor.

[0064] In a specific implementation, the above-mentioned extraction of key points and feature descriptors for each image in the image pair, and fusing the position coding calculated based on the key points with the feature descriptor to obtain a first descriptor, includes: performing grayscale transformation on each image in the image pair to obtain a corresponding grayscale image; using a feature detection algorithm to extract the key points and feature descriptors of each of the grayscale images, and obtaining the confidence of each of the key points; splicing the dimensions corresponding to the key points and the confidence, and inputting the splicing results into a preset multi-layer feedforward network to calculate the position coding; fusing the position coding to the feature descriptor to obtain the first descriptor.

[0065] That is, firstly, the two input RGB three-channel images I1 and I2 are converted to grayscale space, that is, grayscale transformation is performed on them first to obtain grayscale image I 1g and I 2g Then, the ORB (Oriented Fast and Rotated Brief, a feature detection algorithm; Fast stands for Features from Accelerated Segment Test, a fast corner detection algorithm; Brief stands for Binary Robust Independent Elementary Features) algorithm is used to extract key points and feature descriptors from the two grayscale images, and the confidence of each key point is obtained, which are recorded as and ; where k, D and S are two-dimensional vectors, and their dimensions are and , k represents the key point, D represents the feature descriptor, S represents the key point confidence, N represents the number of key points, and C represents the dimension of the descriptor.

[0066] The specific process is as follows:

[0067] ;

[0068] ;

[0069] in, It is the extraction function of key points and feature descriptors.

[0070] Furthermore, the dimensions corresponding to the key point vector and the confidence vector are concatenated to form a joint representation that contains both spatial and appearance information. This can be expressed as:

[0071] ;

[0072] in, Represents a vector concatenation operation, i represents the index of the i-th image, i=1 or 2.

[0073] The concatenation result is input into a preset multi-layer feedforward network to calculate the position encoding. This encoding function converts the original key points and their confidence into a high-dimensional representation that contains both spatial information (the location of the key points) and appearance information (the confidence of the detection):

[0074] ;

[0075] in, Represents a 1×1 convolution, where two 1×1 convolutions are used to implement position encoding. Represents the Relu activation function, which is used to add nonlinear constraints.

[0076] Then the position code is fused into the feature descriptor to obtain the first descriptor, which is the calculated position code. Fusion with the feature descriptor D to obtain the first descriptor after position embedding and .

[0077] ; where i=1,2.

[0078] Step S13: Perform a linear transformation on the first descriptor, and use the frequency domain attention weight calculated based on the linear transformation result to weight the frequency domain information of the first descriptor to obtain a second descriptor.

[0079] In this embodiment, a frequency attention module is set to weight the frequency domain information of the position-fused descriptor, so as to make full use of the frequency domain characteristics of the image. Specifically, the first descriptor is first linearly transformed, and the frequency domain attention weight calculated based on the linear transformation result is used to weight the frequency domain information of the first descriptor to obtain the second descriptor. That is, this application needs to use the linear transformation result to weight the frequency domain information of the first descriptor, so as to make full use of the frequency domain characteristics of the image.

[0080] Step S14: performing channel division on the second descriptor to obtain a plurality of branches, extracting spatial domain features from each branch, and fusing the spatial domain features of each branch to obtain a third descriptor.

[0081] In this embodiment, a spatial domain feature enhancement module is also provided to extract the spatial domain features of the descriptor, and to process the spatial domain information more finely by extracting and aggregating multi-scale features. Specifically, the second descriptor after frequency domain weighting is firstly divided into channels to obtain several branches, and the spatial domain features are extracted from each branch, and the spatial domain features of each branch are fused to obtain the third descriptor. In this way, the third descriptor includes both the frequency domain features and the spatial domain features of the image.

[0082] Step S15: Calculate a matching probability matrix based on the third descriptor corresponding to the first image and the third descriptor corresponding to the second image, so as to determine the best matching point from the matching probability matrix.

[0083] In this embodiment, the best matching point is determined by calculating the matching probability moment obtained by the third descriptor corresponding to the first image and the third descriptor corresponding to the second image. Through the above scheme, the present application can realize adaptive fusion of spatial domain features and frequency domain features, and use these features to perform robust feature matching, thereby improving the accuracy of feature matching.

[0084] In a specific implementation, the above-mentioned calculation of the matching probability matrix based on the third descriptor corresponding to the first image and the third descriptor corresponding to the second image includes: normalizing the third descriptor corresponding to the first image to obtain a first processing result; normalizing the third descriptor corresponding to the second image to obtain a second processing result; obtaining a target correlation matrix based on the first processing result and the second processing result, and using a preset activation function to perform a preset number of operations on the target correlation matrix to obtain a matching probability matrix.

[0085] Among them, the third descriptor corresponding to the first image is recorded as ,right Perform normalization to obtain the first processing result ; The third descriptor corresponding to the second image is recorded as ,right Perform normalization to obtain the first processing result , its specific expression is:

[0086] ;

[0087] ;

[0088] in, Represents the bi-norm of a vector.

[0089] Furthermore, the first processing result And the second processing result Perform dot product operation to get the target correlation matrix ;

[0090] ;

[0091] According to the obtained correlation matrix, the preset activation function is used to calculate the target correlation matrix for a preset number of times to obtain the matching probability matrix. Specifically, the double Softmax function is used to calculate the target correlation matrix to obtain the matching probability matrix P 12 ;

[0092] .

[0093] In a specific implementation, the above-mentioned determining the best matching point from the matching probability matrix includes: determining the maximum value of each row and each column from the matching probability matrix; and determining the best matching point based on the maximum value of each row and each column. That is, by calculating the probability matrix P 12 The best matching point can be extracted by finding the maximum value of each row and column in the image. It should be noted that the index of the position of each element in the matching probability matrix corresponds to the key point in the image. Therefore, the best matching point can be found by first finding the maximum value of each row and column, and then according to the position index corresponding to the maximum value.

[0094] In addition, a maximum number of optimal matching points can be set. Assuming that the maximum number of optimal matching points is 3, and the maximum values ​​of each row and column determined from the matching probability matrix are 5, only the key points corresponding to the first three values ​​are taken as the best matching points.

[0095] It can be seen that after obtaining the image pair to be matched, the present application first needs to extract key points and feature descriptors for each image in the image pair, and fuse the position code calculated based on the key points with the feature descriptor to obtain the first descriptor; wherein the image pair includes the first image and the second image to be matched. Further, the present application needs to perform a linear transformation on the first descriptor, and weight the frequency domain information of the first descriptor based on the frequency domain attention weight calculated from the linear transformation result to obtain the second descriptor, that is, the present application needs to use the linear transformation result to weight the frequency domain information of the first descriptor, so as to make full use of the frequency domain features of the image. Then, the second descriptor after frequency domain weighting is divided into several branches, and the spatial domain features are extracted from each branch, and the spatial domain features of each branch are fused to obtain the third descriptor. In this way, the third descriptor includes both the frequency domain features and the spatial domain features of the image, and finally the matching probability moment calculated from the third descriptor corresponding to the first image and the third descriptor corresponding to the second image is used to determine the best matching point. Therefore, the present application can realize the adaptive fusion of spatial domain features and frequency domain features, thereby utilizing these features for robust feature matching and improving the accuracy of feature matching.

[0096] See also Figure 3 As shown, the embodiment of the present application discloses a specific feature matching method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically, it includes:

[0097] Step S21: Acquire an image pair to be matched; wherein the image pair includes a first image and a second image.

[0098] Step S22: extracting key points and feature descriptors from each image in the image pair, and fusing the position code calculated based on the key points with the feature descriptor to obtain a first descriptor.

[0099] Step S23: Use a predefined linear transformation matrix to perform linear transformation on the first descriptor to obtain a query vector, a key vector and a value vector, use a high-pass filter and a low-pass filter to decompose the query vector to obtain corresponding high-frequency features and low-frequency features, and fuse the high-frequency features and the low-frequency features to obtain the total frequency domain features.

[0100] In this embodiment, see Figure 4 As shown in the frequency attention module, the first descriptor is first transformed using a predefined linear transformation matrix and Linear transformation is performed to obtain the query vector Q, the key vector K and the value vector V. That is, the embodiment of the present application first transforms and Mappings are respectively obtained to obtain three vectors in different high-dimensional spaces, recorded as and ,in:

[0101] ;

[0102] ;

[0103] ;

[0104] Among them, W Q , W K and W V are three linear transformation matrices.

[0105] Further, the query vector is decomposed by a high-pass filter and a low-pass filter to obtain the corresponding high-frequency features and low-frequency features, and then the high-frequency features and the low-frequency features are fused to obtain the total frequency domain features. It can be understood that the frequency domain features are the representation of the information in the image in the frequency domain space. By performing frequency analysis such as Fourier transform on the image, the image data originally in the spatial domain can be converted into frequency domain data. In the frequency domain, different frequency components of the image can reflect different image characteristics, among which the high-frequency components usually correspond to the edges, details and other parts of the image that change dramatically, while the low-frequency components reflect the overall structure, smooth areas and other parts of the image that change more slowly. Therefore, the frequency domain features can not only characterize the local details of the image, but also capture the global structural information. However, the existing methods do not make full use of the frequency domain features of the image, but directly use the spatial domain features for coarse-grained feature matching. Therefore, in some complex environments, the accuracy of feature matching is often insufficient. The scheme of the present application can make full use of the frequency domain features of the image.

[0106] Specifically, the above-mentioned decomposition of the query vector by a high-pass filter and a low-pass filter respectively obtains corresponding high-frequency features and low-frequency features, including: using high-pass filter convolution kernels of different sizes to high-pass filter the query vector to obtain corresponding high-frequency components, and fusing the high-frequency components to obtain high-frequency features; using average pooling functions of different sizes to low-pass filter the query vector to obtain corresponding low-frequency components, and bilinearly interpolating each of the low-frequency components and fusing the interpolated low-frequency components to obtain low-frequency features.

[0107] That is, first, the present application uses a high-pass filter to decompose the query vector Q to obtain high-frequency features ;

[0108] Specifically, high-pass filter convolution kernels of sizes 3×3, 5×5, and 7×7 are used to perform high-pass filtering on Q to obtain high-frequency components High1, High2, and High3 at different levels;

[0109] ;

[0110] ;

[0111] ;

[0112] in, , and They represent 3×3, 5×5, and 7×7 convolution kernels respectively.

[0113] Then, the high-frequency components decomposed are fused in the channel dimension to obtain the final high-frequency features. :

[0114] .

[0115] Secondly, this application uses a low-pass filter to decompose the query vector Q to obtain low-frequency features ;

[0116] Specifically, 3×3, 7×7 and 11×11 adaptive average pooling functions are used to low-pass filter Q and output different low-frequency components Low1, Low2 and Low3:

[0117] ;

[0118] ;

[0119] ;

[0120] in, , and They represent 3×3, 5×5, and 7×7 adaptive average pooling operations respectively.

[0121] It should be noted that the average pooling operation will downsample the feature size, so the pooled features need to be restored to the original size through bilinear interpolation to keep the size consistent with other channels. The formula is expressed as:

[0122] ;

[0123] ;

[0124] ;

[0125] Among them, Interpolate() represents the bilinear interpolation operation, and finally the interpolated low-frequency components are spliced ​​and fused in the channel dimension to obtain the final low-frequency features :

[0126] .

[0127] Finally, high-frequency features are integrated and low frequency characteristics Get the total frequency domain features ;

[0128] Specifically, 1×1 convolution is used to fuse high-frequency and low-frequency features:

[0129] .

[0130] In the above scheme, the multi-scale feature extractor can perceive the information of descriptors at different scales, which greatly enhances the robustness of feature matching at different clarity levels.

[0131] Step S24: Calculate the frequency attention score based on the total frequency domain features and the key vector, convert the frequency attention score into a frequency domain attention weight, and use the frequency domain attention weight and the value vector to obtain a second descriptor weighted by the frequency domain information.

[0132] In this embodiment, firstly based on the total frequency domain characteristics And the key vector K is used to calculate the frequency attention score. The process can be expressed as:

[0133] ;

[0134] Among them, K T represents the transpose of K, and C represents the channel dimension size of the vector.

[0135] The frequency attention score is then converted into frequency domain attention weights using the preset activation function Softmax. These weights represent the degree of attention the network pays to different frequencies of the descriptor features:

[0136] ;

[0137] Among them, Softmax() represents the activation function.

[0138] Finally, by multiplying the frequency domain attention weight with the value vector V, the second descriptor weighted by the frequency domain information is obtained to focus on the features related to specific frequency domain information:

[0139] ;

[0140] Among them, i=1,2.

[0141] Step S25: Divide the second descriptor into several branches based on the channel dimension, and number each branch according to a preset numbering sequence, assign a target convolution kernel of a different size to each branch based on the numbering result, and extract spatial domain features on each branch using the corresponding target convolution kernel based on the preset feature extraction rules, and fuse the spatial domain features of each branch to obtain a third descriptor.

[0142] In this embodiment, see Figure 5 As shown, in the spatial domain feature enhancement module, the second descriptor after the frequency domain information is weighted is first and Divide the channel dimension into four equal parts, and number each branch according to the preset numbering sequence. The process can be expressed as:

[0143] ;

[0144] Among them, Split() represents the channel division operation. Then, target convolution kernels of different sizes are assigned to each branch based on the numbering result, so as to use convolutions of different scales to extract spatial domain features on each branch based on the preset feature extraction rules. Among them, the preset feature extraction rule is to use the target convolution kernel corresponding to the current number to extract the spatial domain features of the branch corresponding to the current number and the spatial domain features corresponding to the previous number to obtain the spatial domain features of the branch corresponding to the current number. This process can be expressed as:

[0145] ;

[0146] ;

[0147] ;

[0148] ;

[0149] Among them, σ() represents the Relu activation function. , , and They are the extracted spatial domain features respectively. , , and They represent convolution kernels of 1×1, 3×3, 5×5, and 7×7 respectively. By using only convolution operations to realize feature extraction and post-processing, it is convenient to deploy embedded devices.

[0150] Finally, the spatial domain features of different branches are fused to obtain the third descriptor with enhanced spatial domain features. and :

[0151] .

[0152] Step S26: Calculate a matching probability matrix based on the third descriptor corresponding to the first image and the third descriptor corresponding to the second image, so as to determine the best matching point from the matching probability matrix.

[0153] For more specific processing procedures of the above steps S21, S22 and S26, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be described in detail here.

[0154] It can be seen that the present application discloses a frequency attention module, which explicitly extracts and fuses the frequency domain features of the image. The frequency domain features can capture global and local information that is difficult to extract in the spatial domain. After the high-frequency and low-frequency features are combined, the image can be described more comprehensively, which helps to improve the accuracy of feature matching. In addition, after the frequency domain features are fused, the algorithm has strong robustness to problems such as image illumination changes, noise, and blur. The present application also discloses a spatial domain feature enhancement module, which can extract and enhance the spatial domain features of the descriptor. By making full use of the different features of the image, it is beneficial to improve the stability and accuracy of feature matching. By using the above two modules, a frequency-guided adaptive feature matching network is built. The network can realize end-to-end robust feature matching function, that is, by inputting two images to be matched, the network can realize adaptive fusion of spatial domain and frequency domain features, and use these features for robust feature matching, and accurate matching results can be obtained.

[0155] See also Figure 6 As shown, the embodiment of the present application discloses a feature matching device, which includes:

[0156] An image acquisition module 11 is used to acquire an image pair to be matched; wherein the image pair includes a first image and a second image;

[0157] A feature extraction module 12 is used to extract key points and feature descriptors from each image in the image pair, and fuse the position code calculated based on the key points with the feature descriptor to obtain a first descriptor;

[0158] A frequency domain weighting module 13, configured to perform a linear transformation on the first descriptor, and weight the first descriptor with frequency domain information using a frequency domain attention weight calculated based on the linear transformation result to obtain a second descriptor;

[0159] A spatial domain fusion module 14 is used to divide the second descriptor into channels to obtain a plurality of branches, extract spatial domain features from each branch, and fuse the spatial domain features of each branch to obtain a third descriptor;

[0160] The best matching point determination module 15 is used to calculate a matching probability matrix based on the third descriptor corresponding to the first image and the third descriptor corresponding to the second image, so as to determine the best matching point from the matching probability matrix.

[0161] It can be seen that after obtaining the image pair to be matched, the present application first needs to extract key points and feature descriptors for each image in the image pair, and fuse the position code calculated based on the key points with the feature descriptor to obtain the first descriptor; wherein the image pair includes the first image and the second image to be matched. Further, the present application needs to perform a linear transformation on the first descriptor, and weight the frequency domain information of the first descriptor based on the frequency domain attention weight calculated from the linear transformation result to obtain the second descriptor, that is, the present application needs to use the linear transformation result to weight the frequency domain information of the first descriptor, so as to make full use of the frequency domain features of the image. Then, the second descriptor after frequency domain weighting is divided into several branches, and the spatial domain features are extracted from each branch, and the spatial domain features of each branch are fused to obtain the third descriptor. In this way, the third descriptor includes both the frequency domain features and the spatial domain features of the image, and finally the matching probability moment calculated from the third descriptor corresponding to the first image and the third descriptor corresponding to the second image is used to determine the best matching point. Therefore, the present application can realize the adaptive fusion of spatial domain features and frequency domain features, thereby utilizing these features for robust feature matching and improving the accuracy of feature matching.

[0162] Since the embodiments of the device part correspond to the embodiments of the method part, the embodiments of the device part refer to the description of the embodiments of the method part, which will not be described here. In addition, the invention has the same beneficial effects as the feature matching method mentioned above.

[0163] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the feature matching method performed by the electronic device disclosed in any of the aforementioned embodiments.

[0164] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0165] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0166] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0167] Among them, the operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, which can be Windows, Unix, Linux, etc. In addition to including a computer program that can be used to complete the feature matching method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. In addition to data transmitted from an external device received by the electronic device, the data 223 can also include data collected by its own input and output interface 25, etc.

[0168] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the feature matching method steps disclosed in any of the aforementioned embodiments are implemented.

[0169] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0170] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0171] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of storage medium known in the art.

[0172] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0173] The above is a detailed introduction to a feature matching method, device, equipment and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A feature matching method, characterized in that: include: Acquire an image pair to be matched; wherein the image pair includes a first image and a second image; Extracting key points and feature descriptors from each image in the image pair, and fusing the position code calculated based on the key points with the feature descriptor to obtain a first descriptor; Performing a linear transformation on the first descriptor, and weighting the first descriptor with frequency domain information using a frequency domain attention weight calculated based on the linear transformation result to obtain a second descriptor; Performing channel division on the second descriptor to obtain a plurality of branches, extracting spatial domain features from each branch, and fusing the spatial domain features of each branch to obtain a third descriptor; Calculating a matching probability matrix based on a third descriptor corresponding to the first image and a third descriptor corresponding to the second image, so as to determine a best matching point from the matching probability matrix; The step of extracting key points and feature descriptors from each image in the image pair, and fusing the position code calculated based on the key points with the feature descriptors to obtain a first descriptor includes: Performing grayscale transformation on each image in the image pair to obtain a corresponding grayscale image; Extracting key points and feature descriptors of each grayscale image using a feature detection algorithm, and obtaining the confidence of each key point; Concatenating the dimensions corresponding to the key points and the confidence levels, and inputting the concatenated results into a preset multi-layer feedforward network to calculate position encoding; Fusion the position code to the feature descriptor to obtain a first descriptor; The performing a linear transformation on the first descriptor, and weighting the first descriptor with frequency domain information using a frequency domain attention weight calculated based on the linear transformation result to obtain a second descriptor, comprises: Using a predefined linear transformation matrix to linearly transform the first descriptor to obtain a query vector, a key vector, and a value vector; Decomposing the query vector using a high-pass filter and a low-pass filter respectively to obtain corresponding high-frequency features and low-frequency features, and fusing the high-frequency features and the low-frequency features to obtain a frequency domain total feature; Calculating a frequency attention score based on the frequency domain total feature and the key vector, and converting the frequency attention score into a frequency domain attention weight; The frequency domain attention weight and the value vector are used to obtain a second descriptor weighted by the frequency domain information.

2. The feature matching method according to claim 1, characterized in that: The step of respectively using a high-pass filter and a low-pass filter to decompose the query vector to obtain corresponding high-frequency features and low-frequency features includes: Using high-pass filter convolution kernels of different sizes to perform high-pass filtering on the query vector to obtain corresponding high-frequency components, and fusing the high-frequency components to obtain high-frequency features; The query vector is low-pass filtered using average pooling functions of different sizes to obtain corresponding low-frequency components, and bilinear interpolation is performed on each of the low-frequency components, and the interpolated low-frequency components are fused to obtain low-frequency features.

3. The feature matching method according to claim 1, characterized in that: The channel division of the second descriptor to obtain a plurality of branches, and extracting spatial domain features from each branch, comprises: Dividing the second descriptor into a plurality of branches evenly based on the channel dimension, and numbering each branch according to a preset numbering sequence; Based on the numbering results, target convolution kernels of different sizes are assigned to each branch, and based on the preset feature extraction rules, the corresponding target convolution kernels are used to extract spatial domain features on each branch; Among them, the preset feature extraction rule includes a rule for extracting features of the branch corresponding to the current number and the spatial domain features corresponding to the previous number using the target convolution kernel corresponding to the current number to obtain the spatial domain features of the branch corresponding to the current number.

4. The feature matching method according to claim 1, characterized in that: The calculating a matching probability matrix based on the third descriptor corresponding to the first image and the third descriptor corresponding to the second image includes: Normalizing a third descriptor corresponding to the first image to obtain a first processing result; Normalizing the third descriptor corresponding to the second image to obtain a second processing result; A target correlation matrix is ​​obtained based on the first processing result and the second processing result, and a matching probability matrix is ​​obtained by performing a preset number of operations on the target correlation matrix using a preset activation function.

5. The feature matching method according to any one of claims 1 to 4, characterized in that: Determining the best matching point from the matching probability matrix includes: Determine the maximum value of each row and each column from the matching probability matrix; The best matching point is determined based on the maximum value in each row and column.

6. A feature matching device, characterized in that: include: An image acquisition module, used to acquire an image pair to be matched; wherein the image pair includes a first image and a second image; A feature extraction module, used for extracting key points and feature descriptors from each image in the image pair, and fusing a position code calculated based on the key points with the feature descriptors to obtain a first descriptor; A frequency domain weighting module, used to perform a linear transformation on the first descriptor, and use the frequency domain attention weight calculated based on the linear transformation result to weight the first descriptor with frequency domain information to obtain a second descriptor; A spatial domain fusion module, used for performing channel division on the second descriptor to obtain a plurality of branches, extracting spatial domain features from each branch, and fusing the spatial domain features of each branch to obtain a third descriptor; a best matching point determination module, configured to calculate a matching probability matrix based on a third descriptor corresponding to the first image and a third descriptor corresponding to the second image, so as to determine a best matching point from the matching probability matrix; The feature extraction module is used to perform grayscale transformation on each image in the image pair to obtain a corresponding grayscale image; extract key points and feature descriptors of each grayscale image using a feature detection algorithm, and obtain the confidence of each key point; splice the dimensions corresponding to the key points and the confidence, and input the splicing result into a preset multi-layer feedforward network to calculate the position code; fuse the position code into the feature descriptor to obtain a first descriptor; The frequency domain weighting module is used to use a predefined linear transformation matrix to linearly transform the first descriptor to obtain a query vector, a key vector and a value vector; respectively decompose the query vector using a high-pass filter and a low-pass filter to obtain corresponding high-frequency features and low-frequency features, and fuse the high-frequency features and the low-frequency features to obtain a total frequency domain feature; calculate a frequency attention score based on the total frequency domain feature and the key vector, and convert the frequency attention score into a frequency domain attention weight; use the frequency domain attention weight and the value vector to obtain a second descriptor weighted by the frequency domain information.

7. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the feature matching method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein, when the computer program is executed by a processor, the steps of the feature matching method as described in any one of claims 1 to 5 are implemented.

Images