Image feature matching method, system and device and computer storage medium

Abstract

The invention discloses an image feature matching method, system and device and a computer storage medium, and relates to the technical field of image processing, a first feature descriptor of a first image and a second feature descriptor of a second image are acquired, and the feature descriptors are descriptors embedded with position information; performing similarity analysis on features between the first feature descriptor and the second feature descriptor to obtain a feature similarity analysis result; performing feature weight adjustment on the first feature descriptor and the second feature descriptor according to a feature similarity analysis result to obtain a corresponding first target descriptor and a corresponding second target descriptor; and performing feature matching on the first image and the second image according to the first target descriptor and the second target descriptor to obtain a feature matching result. The weight of the similar features is increased, the weight of the dissimilar features is reduced, the relevance between the features in the two descriptors is improved, the similar features can be accurately recognized, and a more accurate feature matching result is obtained.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to an image feature matching method, system, device and computer storage medium. Background Art

[0002] Feature matching is a core task in computer vision, which aims to find corresponding points between different images or image sequences by comparing local feature points in the image. For example, a feature matching method based on deep learning can automatically extract feature points and generate descriptors from images through convolutional neural networks, and then directly match feature points in the same network.

[0003] Although the feature matching method based on deep learning can automatically learn features, has stronger robustness, and has better adaptability to changes in illumination, rotation, and scale, it directly uses coarse-grained descriptors for feature matching, and the accuracy of feature matching is low.

[0004] In summary, how to accurately perform feature matching on images is a problem that currently needs to be solved urgently by those skilled in the art. Summary of the invention

[0005] The purpose of this application is to provide an image feature matching method, which can solve the technical problem of how to accurately match image features to a certain extent. This application also provides an image feature matching system, an electronic device and a computer-readable storage medium.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] An image feature matching method, comprising:

[0008] Obtaining a first feature descriptor of the first image and a second feature descriptor of the second image, wherein the feature descriptors are descriptors embedded with position information;

[0009] Performing similarity analysis on features between the first feature descriptor and the second feature descriptor to obtain a feature similarity analysis result;

[0010] According to the feature similarity analysis result, feature weights of the first feature descriptor and the second feature descriptor are adjusted respectively to obtain corresponding first target descriptors and second target descriptors;

[0011] According to the first object descriptor and the second object descriptor, feature matching is performed on the first image and the second image to obtain a feature matching result.

[0012] Preferably, performing similarity analysis on features between the first feature descriptor and the second feature descriptor to obtain a feature similarity analysis result includes:

[0013] Performing feature fusion on the first feature descriptor and the second feature descriptor through a convolution operation to obtain a joint feature and generate a feature vector of the joint feature;

[0014] Mapping the feature vector based on a fully connected layer to generate a first similarity weighted vector and a second similarity weighted vector;

[0015] Processing the first similarity weighted vector and the second similarity weighted vector respectively to generate a first similarity weight coefficient and a second similarity weight coefficient;

[0016] Using the first similarity weight coefficient and the second similarity weight coefficient as feature similarity analysis results;

[0017] The step of adjusting the feature weights of the first feature descriptor and the second feature descriptor according to the feature similarity analysis result to obtain the corresponding first target descriptor and second target descriptor comprises:

[0018] Adjusting the feature weight of the first feature descriptor by using the first similarity weight coefficient to obtain a first target descriptor;

[0019] The feature weight of the second feature descriptor is adjusted using the second similarity weight coefficient to obtain a second target descriptor.

[0020] Preferably, the step of fusing the first feature descriptor and the second feature descriptor to obtain a joint feature includes:

[0021] Performing feature extraction on the first feature descriptor through a convolution operation to obtain a first local feature;

[0022] Performing feature extraction on the second feature descriptor through a convolution operation to obtain a second local feature;

[0023] Splicing the first local feature and the second local feature to obtain a spliced ​​feature;

[0024] Performing dimension-upgrading on the splicing features to obtain dimension-upgrading features;

[0025] Summing the dimension-increased features on the newly added dimension to obtain a joint feature;

[0026] The step of generating a feature vector of the joint feature comprises:

[0027] Performing a global average pooling operation on the second dimension of the joint feature to obtain a feature vector;

[0028] The mapping of the feature vector based on the fully connected layer to generate a first similarity weighted vector and a second similarity weighted vector includes:

[0029] Mapping the feature vector through a first fully connected layer to generate a mapping vector, wherein the dimension of the mapping vector is lower than that of the feature vector;

[0030] Processing the mapping vector through a second fully connected layer to generate a first similarity weighted vector;

[0031] The mapping vector is processed by a third fully connected layer to generate a second similarity weighted vector.

[0032] Preferably, the obtaining of the first feature descriptor of the first image and the second feature descriptor of the second image comprises:

[0033] Extracting first feature points and a first initial descriptor of the first image, and extracting second feature points and a second initial descriptor of the second image;

[0034] Extracting position information of the first feature point and the second feature point respectively through a learnable linear transformation to obtain corresponding first position information and second position information;

[0035] According to the first position information and the second position information, position information is respectively embedded in the first initial descriptor and the second initial descriptor to obtain a corresponding first feature descriptor and a corresponding second feature descriptor.

[0036] Preferably, the position information includes a position vector, and the extracting the position information of the first feature point and the second feature point respectively through a learnable linear transformation to obtain the corresponding first position information and second position information includes:

[0037] Projecting the first feature point and the second feature point respectively through a learnable linear transformation to obtain corresponding first projection results and second projection results;

[0038] generating a first cosine value and a first sine value of the first projection result;

[0039] generating a second cosine value and a second sine value of the second projection result;

[0040] Combining the first cosine value and the second cosine value to obtain a first position vector;

[0041] The first sine value and the second sine value are combined to obtain a second position vector.

[0042] Preferably, performing feature matching on the first image and the second image according to the first object descriptor and the second object descriptor to obtain a feature matching result includes:

[0043] Performing multi-scale feature perception on the first target descriptor to obtain a first perception descriptor;

[0044] Performing multi-scale feature perception on the second target descriptor to obtain a second perception descriptor;

[0045] According to the first perceptual descriptor and the second perceptual descriptor, feature matching is performed on the first image and the second image to obtain a feature matching result.

[0046] Preferably, performing feature matching on the first image and the second image according to the first perceptual descriptor and the second perceptual descriptor to obtain a feature matching result includes:

[0047] Performing linear projection and normalization processing on the first perceptual descriptor and the second perceptual descriptor to obtain corresponding first processed descriptors and second processed descriptors;

[0048] Performing an inner product on the first processing descriptor and the second processing descriptor to obtain a matching similarity matrix;

[0049] generating a first logarithmic score of the matching similarity matrix;

[0050] transposing the matching similarity matrix and generating a second logarithmic score of the transposed matrix;

[0051] Integrating the first logarithmic score, the second logarithmic score, the first processing descriptor, and the second processing descriptor to obtain a matching score matrix;

[0052] Perform feature matching on the first image and the second image according to the matching score matrix to obtain a feature matching result.

[0053] An image feature matching system, comprising:

[0054] A first acquisition module, used to acquire a first feature descriptor of the first image and a second feature descriptor of the second image, wherein the feature descriptor is a descriptor embedded with position information;

[0055] A first analysis module, configured to perform similarity analysis on features between the first feature descriptor and the second feature descriptor to obtain a feature similarity analysis result;

[0056] A first adjustment module, configured to adjust the feature weights of the first feature descriptor and the second feature descriptor respectively according to the feature similarity analysis result, to obtain a corresponding first target descriptor and a second target descriptor;

[0057] The first matching module is used to perform feature matching on the first image and the second image according to the first target descriptor and the second target descriptor to obtain a feature matching result.

[0058] An electronic device, comprising:

[0059] Memory for storing computer programs;

[0060] A processor is used to implement the steps of any of the above-mentioned image feature matching methods when executing the computer program.

[0061] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned image feature matching methods are implemented.

[0062] The present application provides an image feature matching method, which obtains a first feature descriptor of a first image; and a second feature descriptor of a second image, wherein the feature descriptor is a descriptor that embeds position information; performs similarity analysis on features between the first feature descriptor and the second feature descriptor to obtain a feature similarity analysis result; according to the feature similarity analysis result, performs feature weight adjustment on the first feature descriptor and the second feature descriptor respectively to obtain corresponding first target descriptors and second target descriptors; and performs feature matching on the first image and the second image according to the first target descriptors and the second target descriptors to obtain a feature matching result. After obtaining the first feature descriptor and the second feature descriptor, the present application needs to perform a similarity analysis on the features between the first feature descriptor and the second feature descriptor, so as to obtain a feature similarity analysis result that reflects the feature similarity information between the two descriptors. Since the feature descriptor is embedded with position information, the similarity between the features at the corresponding positions can be more accurately analyzed based on the position information, thereby improving the accuracy of the feature similarity analysis result; then, according to the feature similarity analysis result, the feature weights of the first feature descriptor and the second feature descriptor are adjusted respectively, so that the weights of similar features can be increased and the weights of dissimilar features can be reduced, thereby improving the correlation between the features in the two descriptors and facilitating accurate identification of similar features. In this way, if the first image and the second image are feature matched according to the first target descriptor and the second target descriptor, a more accurate feature matching result can be obtained. The image feature matching system, electronic device and computer-readable storage medium provided by the present application also solve the corresponding technical problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0064] Figure 1 A flowchart of an image feature matching method provided in an embodiment of the present application;

[0065] Figure 2 This is the processing flow chart of the similarity perception module;

[0066] Figure 3 Processing flow chart of multi-scale perception module;

[0067] Figure 4 This is a flow chart of the multi-scale feature matching method based on similarity perception;

[0068] Figure 5 A schematic diagram of the structure of an image feature matching system provided in an embodiment of the present application;

[0069] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0070] Figure 7 Another structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0072] See also Figure 1 , Figure 1 A flowchart of an image feature matching method provided in an embodiment of the present application.

[0073] An image feature matching method provided in an embodiment of the present application may include the following steps:

[0074] Step S101: Obtain a first feature descriptor of a first image.

[0075] Step S102: Obtain a second feature descriptor of the second image.

[0076] In practical applications, after determining the first image and the second image that need to be feature matched, the first feature descriptor of the first image and the second feature descriptor of the second image can be obtained, so that the first feature descriptor and the second feature descriptor can be subsequently processed to determine the feature matching results of the first image and the second image.

[0077] It should be noted that the types of the first image and the second image can be flexibly determined according to the application scenario. For example, in the process of vehicle tracking, the first image and the second image can be driving images taken at different locations, etc. For another example, in vehicle detection, the first image and the second image can be vehicle images taken at different angles, etc. It should also be noted that the feature descriptor refers to the descriptor used for subsequent feature matching. The descriptor refers to the information reflecting the image features obtained after feature extraction of the image, which may include feature points extracted from the image and their surrounding points, etc.

[0078] In practical applications, although the descriptor can reflect the image feature information, the descriptor directly obtained from the image can only reflect the image's own feature information, and cannot provide more effective information for subsequent feature matching. In order to avoid this situation, in the process of obtaining the first feature descriptor of the first image and obtaining the second feature descriptor of the second image, the first feature point and the first initial descriptor of the first image can be extracted, and the second feature point and the second initial descriptor of the second image can be extracted; then, the position information of the feature point is extracted using a Fourier transform-based position embedding method, and the position information of the feature point is embedded into the descriptor, so that the method can perceive the relative position relationship of the feature points, thereby more accurately performing feature matching, that is, the feature descriptor in this application is a descriptor embedded with position information. In a specific application scenario, the feature descriptor may be a descriptor that embeds the position information of feature points in an image. In the process of obtaining the first feature descriptor and the second feature descriptor, the first feature point and the first initial descriptor of the first image may be extracted, and the second feature point and the second initial descriptor of the second image may be extracted; the position information of the first feature point and the second feature point are extracted respectively through a learnable linear transformation to obtain the corresponding first position information and the second position information; based on the first position information and the second position information, the position information of the first initial descriptor and the second initial descriptor are embedded respectively to obtain the corresponding first feature descriptor and the second feature descriptor.

[0079] In a specific application scenario, the position information can be specifically a position vector. Accordingly, in the process of extracting the position information of the first feature point and the second feature point respectively through a learnable linear transformation to obtain the corresponding first position information and the second position information, the first feature point and the second feature point can be projected respectively through a learnable linear transformation to obtain the corresponding first projection result and the second projection result, and the first cosine value and the first sine value of the first projection result are generated, and the second cosine value and the second sine value of the second projection result are generated. The cosine and sine operations produce two position-related vectors, which correspond to the high-frequency and low-frequency components of the input feature point vector respectively; the first cosine value and the second cosine value are combined to obtain the first position vector, and the first sine value and the second sine value are combined to obtain the second position vector, so as to effectively capture the relative position information of the feature point by projecting the position vector to the frequency domain and encoding it using sine and cosine functions; at this time, the first position vector is the first position information, and the second position vector is the second position information. Of course, after obtaining the first initial descriptor and the second initial descriptor, the first initial descriptor can be directly used as the first feature descriptor, and the second initial descriptor can be used as the second feature descriptor for subsequent processing, etc. This application does not make specific limitations here.

[0080] In specific application scenarios, the SIFT algorithm can be used to extract feature points and descriptors of an image. The extraction formula can be: , , i=1,2; where i represents the image index, X1 represents the first image of the RGB three-channel, X2 represents the second image of the RGB three-channel, gray represents the grayscale processing of the image, I1 represents the first grayscale image, I2 represents the second grayscale image; K represents the feature point, , D represents the descriptor, , N represents the number of feature points, C represents the number of channels of the descriptor, It describes the spatial dimensions of feature points and descriptors, that is, the spatial dimension of K is N×2, and the spatial dimension of D is N×C; K1 represents the first feature point, K2 represents the second feature point, D1 represents the first initial descriptor, and D2 represents the second initial descriptor. Correspondingly, the projection result generation formula can be , i=1, 2, where Z represents the projection result, Z1 represents the first projection result, Z2 represents the second projection result, and W represents the linear transformation matrix, which can be implemented with a 1×1 convolution. Unlike traditional fixed-position embedding, the projection matrix W of Fourier position embedding coding is learnable, which means that the present application can automatically adjust the position encoding method during training, thereby improving the model's ability to express complex structures. The position vector generation formula can be: emb[0]=stack([cosines1,cosines2]), emb[1]=stack([sines1,sines2]), where emb[0] represents the first position vector, cosines1 represents the first cosine value, cosines2 represents the second cosine value, emb[1] represents the second position vector, sines1 represents the first sine value, sines2 represents the second sine value, and cosinesi=cos(Z i ), sinesi = sin(Z i ), i=1, 2. The feature descriptor generation formula can be: , i=1,2, where, represents the first feature descriptor, Represents the second feature descriptor, and Rot represents a rotation operation. In this way, by multiplying the position vector emb by the descriptor vector element by element, the position information can be effectively embedded into the descriptor, so that the present application can perceive the relative position relationship of the feature points.

[0081] Step S103: performing similarity analysis on the features between the first feature descriptor and the second feature descriptor to obtain a feature similarity analysis result.

[0082] Step S104: According to the feature similarity analysis result, feature weights of the first feature descriptor and the second feature descriptor are adjusted respectively to obtain corresponding first target descriptors and second target descriptors.

[0083] In practical applications, if there are matching features between two images, there is a similar relationship between the features of the two images, and similar features are conducive to feature matching, while dissimilar features are not conducive to feature matching. Therefore, after obtaining the first feature descriptor and the second feature descriptor, the features between the first feature descriptor and the second feature descriptor can be analyzed for similarity to obtain the feature similarity analysis result, so as to characterize the similarity between the features of the first feature descriptor and the second feature descriptor with the help of this feature similarity analysis result. Correspondingly, after obtaining the feature similarity analysis result, the feature weights of the first feature descriptor and the second feature descriptor can be adjusted respectively according to the feature similarity analysis result, so as to increase the weights of similar features in the first feature descriptor and the second feature descriptor and / or reduce the weights of dissimilar features in the first feature descriptor and the second feature descriptor, and obtain the corresponding first target descriptor and the second target descriptor.

[0084] In practical applications, in the process of performing similarity analysis on features between a first feature descriptor and a second feature descriptor to obtain a feature similarity analysis result, the first feature descriptor and the second feature descriptor can be fused through a convolution operation to obtain a joint feature; a feature vector of the joint feature is generated; the feature vector is mapped based on a fully connected layer to generate a first similarity weighted vector and a second similarity weighted vector; the first similarity weighted vector and the second similarity weighted vector are processed respectively, for example, the first similarity weighted vector and the second similarity weighted vector can be processed based on an activation function to generate a first similarity weight coefficient and a second similarity weight coefficient; the first similarity weight coefficient and the second similarity weight coefficient are used as the feature similarity analysis result.

[0085] In a specific application scenario, in the process of fusing the first feature descriptor and the second feature descriptor to obtain the joint feature, the first feature descriptor can be subjected to feature extraction through a convolution operation to obtain the first local feature, and the second feature descriptor can be subjected to feature extraction through a convolution operation to obtain the second local feature. For example, the first feature descriptor and the second feature descriptor can be subjected to feature extraction through a 3×3 convolution to obtain the corresponding first local feature and second local feature, etc. The local feature generation formula can be: , , where F1 represents the first local feature, Represents a 3×3 convolution operation; represents the first feature descriptor; F2 represents the second local feature, represents the second feature descriptor; the first local feature and the second local feature are spliced ​​to obtain the spliced ​​feature, that is, the spliced ​​feature generation formula can be , F represents the concatenation feature, and stack represents the concatenation operation; the concatenation feature is upgraded in dimension, for example, a new dimension can be added before or after the concatenation feature to obtain the upgraded feature, and the upgraded feature is summed on the newly added dimension to obtain the joint feature, that is, the joint feature generation formula can be F u =sum(F, dim=1), F u represents the joint feature, sum represents the summation operation, and dim=1 represents the added dimension. Of course, after obtaining the concatenated features, the joint feature can be obtained by directly summing the first dimension of the concatenated features according to actual needs.

[0086] In specific application scenarios, in the process of generating the feature vector of the joint feature, the feature vector can be obtained by taking the average of the joint features, or a global average pooling operation can be performed on the second dimension of the joint feature to obtain the feature vector. That is, the feature vector generation formula can be F s =avgpool(F u , dim=2), F s represents the feature vector, avgpool represents the global average pooling operation; in the process of mapping the feature vector based on the fully connected layer to generate the first similarity weighted vector and the second similarity weighted vector, the feature vector can be mapped through the first fully connected layer to generate a mapping vector, the dimension of the mapping vector is lower than the feature vector, the mapping vector is processed through the second fully connected layer to generate the first similarity weighted vector, and the mapping vector is processed through the third fully connected layer to generate the second similarity weighted vector, that is, the similarity weighted vector generation formula can be , , i=1, 2, where Fz represents the mapping vector, FC1 represents the fully connected layer, FC1 represents the second fully connected layer, FC2 represents the third fully connected layer, vector1 represents the first similarity weighted vector, and vector2 represents the second similarity weighted vector. Accordingly, based on the activation function, the first similarity weighted vector and the second similarity weighted vector are processed respectively to generate the first similarity weight coefficient and the second similarity weight coefficient. The similarity weight coefficient generation formula can be used. , i=1, 2 is generated, where weight1 represents the first similarity weight coefficient, represents the second similarity weight coefficient, and sigmoid represents the activation function.

[0087] In specific application scenarios, convolutional layers, global average pooling layers, fully connected layers, activation layers, etc. can be combined into a similarity perception module to dynamically perceive the similarity between different descriptors with the help of this similarity perception module, and dynamically adjust the respective feature weights to enhance the feature representation of each descriptor. Of course, there may be other ways to perceive the similarity between descriptor features and adjust the weights, which are not specifically limited in this application.

[0088] Correspondingly, in the process of adjusting the feature weights of the first feature descriptor and the second feature descriptor respectively according to the feature similarity analysis result to obtain the corresponding first target descriptor and the second target descriptor, the feature weight of the first feature descriptor can be adjusted by the first similarity weight coefficient to obtain the first target descriptor; the feature weight of the second feature descriptor can be adjusted by the second similarity weight coefficient to obtain the second target descriptor, that is, the target descriptor generation formula can be: , i=1,2, where, represents the first target descriptor, represents the second target descriptor. The whole process can be as follows Figure 2 shown.

[0089] Step S105: performing feature matching on the first image and the second image according to the first object descriptor and the second object descriptor to obtain a feature matching result.

[0090] In practical applications, because the circle of similar features in the first target descriptor and the second target descriptor is higher than the weight of dissimilar features, similar features and dissimilar features can be analyzed more accurately according to the feature weights. Therefore, if feature matching is performed on the first image and the second image according to the first target descriptor and the second target descriptor, a more accurate feature matching result can be obtained.

[0091] In practical applications, in the process of performing feature matching on the first image and the second image according to the first target descriptor and the second target descriptor to obtain the feature matching result, multi-scale feature perception can be performed on the first target descriptor to obtain the first perception descriptor; multi-scale feature perception can be performed on the second target descriptor to obtain the second perception descriptor; and feature matching can be performed on the first image and the second image according to the first perception descriptor and the second perception descriptor to obtain the feature matching result.

[0092] In specific application scenarios, in the process of multi-scale feature perception of target descriptors, it can be performed through a multi-scale feature perception module, such as Figure 3As shown in the figure, for example, a 3×3 convolution kernel can be used to capture smaller-scale local features, a 5×5 convolution kernel can be used to capture medium-scale features, and a 7×7 convolution kernel can be used to capture larger-scale global features. Finally, two consecutive 1×1 convolutions are used to fuse multi-scale features to obtain a perceptual descriptor with multi-scale feature perception. That is, the perceptual descriptor generation formula can be: , , , , i=1, 2; where d1 represents the first perceptual descriptor, d2 represents the second perceptual descriptor, Represents the process of fusing multi-scale features in the corresponding channel dimension, Conv 3×3 、Conv 5×5 、Conv 7×7 、Conv 1×1 They represent convolution operations of 3×3, 5×5, 7×7, and 1×1, respectively. Of course, convolution kernels of other sizes can be used for multi-scale feature fusion, and this application does not make any specific limitation here.

[0093] In practical applications, in the process of performing feature matching on the first image and the second image according to the first perceptual descriptor and the second perceptual descriptor to obtain the feature matching result, the first perceptual descriptor and the second perceptual descriptor can be linearly projected and normalized respectively to obtain the corresponding first processing descriptor and the second processing descriptor; the inner product of the first processing descriptor and the second processing descriptor can be obtained to obtain the matching similarity matrix; the first logarithmic score of the matching similarity matrix is ​​generated; the matching similarity matrix is ​​transposed, and the second logarithmic score of the transposed matrix is ​​generated; the first logarithmic score, the second logarithmic score, the first processing descriptor and the second processing descriptor are integrated to obtain the matching score matrix; the first image and the second image are feature matched according to the matching score matrix to obtain the feature matching result. For example, the best matching pair matched1 and matched2 can be selected as the feature matching result by using a two-way consistency check according to the obtained matching score matrix. At this time, the entire solution processing process is as follows: Figure 4 shown.

[0094] In a specific application scenario, the processing descriptor generation formula can be: , , i=1, 2; where proj represents the linear projection function, which can be implemented by 1×1 convolution; represents the fourth root of the channel dimension of the descriptor; desc1 represents the projection descriptor obtained by linearly projecting the first perceptual descriptor, desc2 represents the projection descriptor obtained by linearly projecting the second perceptual descriptor, mdesc1 represents the first processing descriptor, and mdesc2 represents the second processing descriptor. Correspondingly, the matching similarity matrix generation formula can be , where sim represents the matching similarity matrix and T represents the transpose of the matrix. The logarithmic score generation formula can be: scores1=log_softmax(sim), scores2=log_softmax(sim T ), scores1 represents the first logarithmic score, scores2 represents the second logarithmic score, and log_softmax represents the logarithmic exponential normalization function. The matching score matrix generation formula can be: , where scores 12 Represents the matching score matrix.

[0095] It should be noted that the image feature matching method of the present application can be applied to scenes such as image stitching, object recognition, three-dimensional reconstruction, and stereoscopic vision, and the present application does not make specific limitations here. Among them, image stitching refers to seamlessly stitching multiple overlapping images into a panoramic image; object recognition refers to identifying objects in an image by matching feature points in the image; three-dimensional reconstruction refers to building a three-dimensional model of a scene by matching feature points from multiple perspectives; and stereoscopic vision refers to estimating the depth information of an object by matching feature points in two images taken by a binocular camera.

[0096] The present application provides an image feature matching method, which obtains a first feature descriptor of a first image and a second feature descriptor of a second image, wherein the feature descriptor is a descriptor that embeds position information; performs similarity analysis on features between the first feature descriptor and the second feature descriptor to obtain a feature similarity analysis result; according to the feature similarity analysis result, performs feature weight adjustment on the first feature descriptor and the second feature descriptor respectively to obtain corresponding first target descriptors and second target descriptors; performs feature matching on the first image and the second image according to the first target descriptors and the second target descriptors to obtain a feature matching result. After obtaining the first feature descriptor and the second feature descriptor, the present application needs to perform a similarity analysis on the features between the first feature descriptor and the second feature descriptor, so as to obtain a feature similarity analysis result that reflects the feature similarity information between the two descriptors. Since the position information is embedded in the feature descriptor, the similarity between the features at the corresponding positions can be more accurately analyzed based on the position information, thereby improving the accuracy of the feature similarity analysis result; then, according to the feature similarity analysis result, the feature weights of the first feature descriptor and the second feature descriptor are adjusted respectively, which can increase the weights of similar features and reduce the weights of dissimilar features, thereby improving the correlation between the features in the two descriptors and facilitating accurate identification of similar features. In this way, if feature matching is performed on the first image and the second image according to the first target descriptor and the second target descriptor, a more accurate feature matching result can be obtained.

[0097] See also Figure 5 , Figure 5 A schematic diagram of the structure of an image feature matching system provided in an embodiment of the present application.

[0098] An image feature matching system provided in an embodiment of the present application may include:

[0099] A first acquisition module 101 is used to acquire a first feature descriptor of a first image and a second feature descriptor of a second image, wherein the feature descriptor is a descriptor embedded with position information;

[0100] The first analysis module 102 is used to perform similarity analysis on features between the first feature descriptor and the second feature descriptor to obtain a feature similarity analysis result;

[0101] A first adjustment module 103 is used to adjust the feature weights of the first feature descriptor and the second feature descriptor respectively according to the feature similarity analysis result to obtain the corresponding first target descriptor and the second target descriptor;

[0102] The first matching module 104 is used to perform feature matching on the first image and the second image according to the first object descriptor and the second object descriptor to obtain a feature matching result.

[0103] An image feature matching system provided in an embodiment of the present application, wherein the first analysis module may include:

[0104] A first fusion unit is used to perform feature fusion on the first feature descriptor and the second feature descriptor through a convolution operation to obtain a joint feature;

[0105] A first generating unit, used to generate a feature vector of a joint feature;

[0106] A first mapping unit, used for mapping the feature vector based on the fully connected layer to generate a first similarity weighted vector and a second similarity weighted vector;

[0107] A first processing unit, configured to process the first similarity weighted vector and the second similarity weighted vector respectively based on an activation function to generate a first similarity weight coefficient and a second similarity weight coefficient;

[0108] A first setting unit, configured to use the first similarity weight coefficient and the second similarity weight coefficient as a feature similarity analysis result;

[0109] The first adjustment module may include:

[0110] A first adjustment unit, configured to adjust the feature weight of the first feature descriptor by using a first similarity weight coefficient to obtain a first target descriptor;

[0111] The second adjustment unit is used to adjust the feature weight of the second feature descriptor by using the second similarity weight coefficient to obtain a second target descriptor.

[0112] An image feature matching system provided by an embodiment of the present application, the first fusion unit can be used to: extract features from a first feature descriptor through a convolution operation to obtain a first local feature; extract features from a second feature descriptor through a convolution operation to obtain a second local feature; concatenate the first local feature and the second local feature to obtain a concatenated feature; perform dimension upgrade on the concatenated feature to obtain an upgraded dimension feature; and sum the upgraded dimension features in the newly added dimension to obtain a joint feature.

[0113] In an image feature matching system provided by an embodiment of the present application, the first generating unit may be used to: perform a global average pooling operation on a second dimension of the joint feature to obtain a feature vector;

[0114] The first mapping unit can be used to: map the feature vector through the first fully connected layer to generate a mapping vector, the dimension of the mapping vector is lower than the feature vector; process the mapping vector through the second fully connected layer to generate a first similarity weighted vector; process the mapping vector through the third fully connected layer to generate a second similarity weighted vector.

[0115] An image feature matching system provided in an embodiment of the present application, wherein the first acquisition module may include:

[0116] A first extraction unit, used to extract first feature points and a first initial descriptor of the first image, and to extract second feature points and a second initial descriptor of the second image;

[0117] A second extraction unit is used to extract position information of the first feature point and the second feature point respectively through a learnable linear transformation to obtain corresponding first position information and second position information;

[0118] The first embedding unit is used to embed the position information of the first initial descriptor and the second initial descriptor respectively according to the first position information and the second position information to obtain the corresponding first feature descriptor and the second feature descriptor.

[0119] An image feature matching system provided by an embodiment of the present application, wherein position information includes a position vector, and a second extraction unit can be used to: project a first feature point and a second feature point respectively through a learnable linear transformation to obtain corresponding first projection results and second projection results; generate a first cosine value and a first sine value of the first projection result; generate a second cosine value and a second sine value of the second projection result; combine the first cosine value and the second cosine value to obtain a first position vector; combine the first sine value and the second sine value to obtain a second position vector.

[0120] An image feature matching system provided in an embodiment of the present application, wherein the first matching module may include:

[0121] A first perception unit, used for performing multi-scale feature perception on the first target descriptor to obtain a first perception descriptor;

[0122] A second perception unit is used to perform multi-scale feature perception on the second target descriptor to obtain a second perception descriptor;

[0123] The first matching unit is used to perform feature matching on the first image and the second image according to the first perceptual descriptor and the second perceptual descriptor to obtain a feature matching result.

[0124] An image feature matching system provided by an embodiment of the present application, wherein the first matching unit can be used to: perform linear projection and normalization processing on the first perceptual descriptor and the second perceptual descriptor respectively to obtain the corresponding first processing descriptor and the second processing descriptor; perform inner product on the first processing descriptor and the second processing descriptor to obtain a matching similarity matrix; generate a first logarithmic score of the matching similarity matrix; transpose the matching similarity matrix and generate a second logarithmic score of the transposed matrix; integrate the first logarithmic score, the second logarithmic score, the first processing descriptor and the second processing descriptor to obtain a matching score matrix; perform feature matching on the first image and the second image according to the matching score matrix to obtain a feature matching result.

[0125] The present application also provides an electronic device and a computer-readable storage medium, both of which have the corresponding effects of the image feature matching method provided in the embodiment of the present application. Figure 6 , Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0126] An electronic device provided in an embodiment of the present application includes a memory 201 and a processor 202. The memory 201 stores a computer program. When the processor 202 executes the computer program, the steps of the image feature matching method described in any of the above embodiments are implemented.

[0127] See also Figure 7 , another electronic device provided in the embodiment of the present application may also include: an input port 203 connected to the processor 202, for transmitting commands input from the outside to the processor 202; a display unit 204 connected to the processor 202, for displaying the processing results of the processor 202 to the outside; a communication module 205 connected to the processor 202, for realizing communication between the electronic device and the outside. The display unit 204 can be a display panel, a laser scanning display, etc.; the communication method adopted by the communication module 205 includes but is not limited to mobile high-definition link technology (Mobile High-Definition Link, MHL), Universal Serial Bus (Universal Serial Bus, USB), High-Definition Multimedia Interface (High-DefinitionMultimedia Interface, HDMI), wireless connection: wireless fidelity technology (WIreless Fidelity, WiFi), Bluetooth communication technology, low-power Bluetooth communication technology, and communication technology based on IEEE802.11s.

[0128] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the image feature matching method described in any of the above embodiments are implemented.

[0129] The computer-readable storage medium involved in this application includes random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs (Compact Disc Read-Only Memory), or any other form of storage medium known in the technical field.

[0130] For the description of the relevant parts of the image feature matching system, electronic device, and computer-readable storage medium provided in the embodiments of the present application, please refer to the detailed description of the corresponding parts in the image feature matching method provided in the embodiments of the present application, which will not be repeated here. In addition, the parts of the above technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive elaboration.

[0131] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0132] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image feature matching method, characterized in that: include: Obtaining a first feature descriptor of the first image and a second feature descriptor of the second image, wherein the feature descriptors are descriptors embedded with position information; Performing similarity analysis on features between the first feature descriptor and the second feature descriptor to obtain a feature similarity analysis result; According to the feature similarity analysis result, feature weights of the first feature descriptor and the second feature descriptor are adjusted respectively to obtain corresponding first target descriptors and second target descriptors; According to the first object descriptor and the second object descriptor, feature matching is performed on the first image and the second image to obtain a feature matching result.

2. The method according to claim 1, characterized in that The performing similarity analysis on the features between the first feature descriptor and the second feature descriptor to obtain a feature similarity analysis result includes: Performing feature fusion on the first feature descriptor and the second feature descriptor through a convolution operation to obtain a joint feature and generate a feature vector of the joint feature; Mapping the feature vector based on a fully connected layer to generate a first similarity weighted vector and a second similarity weighted vector; Processing the first similarity weighted vector and the second similarity weighted vector respectively to generate a first similarity weight coefficient and a second similarity weight coefficient; Using the first similarity weight coefficient and the second similarity weight coefficient as feature similarity analysis results; The step of adjusting the feature weights of the first feature descriptor and the second feature descriptor according to the feature similarity analysis result to obtain the corresponding first target descriptor and second target descriptor comprises: Adjusting the feature weight of the first feature descriptor by using the first similarity weight coefficient to obtain a first target descriptor; The feature weight of the second feature descriptor is adjusted using the second similarity weight coefficient to obtain a second target descriptor.

3. The method according to claim 2, characterized in that The step of fusing the first feature descriptor and the second feature descriptor to obtain a joint feature includes: Performing feature extraction on the first feature descriptor through a convolution operation to obtain a first local feature; Performing feature extraction on the second feature descriptor through a convolution operation to obtain a second local feature; Splicing the first local feature and the second local feature to obtain a spliced ​​feature; Performing dimension-upgrading on the splicing features to obtain dimension-upgrading features; Summing the dimension-increased features on the newly added dimension to obtain a joint feature; The step of generating a feature vector of the joint feature comprises: Performing a global average pooling operation on the second dimension of the joint feature to obtain a feature vector; The mapping of the feature vector based on the fully connected layer to generate a first similarity weighted vector and a second similarity weighted vector includes: Mapping the feature vector through a first fully connected layer to generate a mapping vector, wherein the dimension of the mapping vector is lower than that of the feature vector; Processing the mapping vector through a second fully connected layer to generate a first similarity weighted vector; The mapping vector is processed by a third fully connected layer to generate a second similarity weighted vector.

4. The method according to any one of claims 1 to 3, characterized in that: The obtaining of a first feature descriptor of the first image and a second feature descriptor of the second image includes: Extracting first feature points and a first initial descriptor of the first image, and extracting second feature points and a second initial descriptor of the second image; Extracting position information of the first feature point and the second feature point respectively through a learnable linear transformation to obtain corresponding first position information and second position information; According to the first position information and the second position information, position information is respectively embedded in the first initial descriptor and the second initial descriptor to obtain a corresponding first feature descriptor and a corresponding second feature descriptor.

5. The method according to claim 4, characterized in that The position information includes a position vector, and the extracting the position information of the first feature point and the second feature point respectively through a learnable linear transformation to obtain the corresponding first position information and second position information includes: Projecting the first feature point and the second feature point respectively through a learnable linear transformation to obtain corresponding first projection results and second projection results; generating a first cosine value and a first sine value of the first projection result; generating a second cosine value and a second sine value of the second projection result; Combining the first cosine value and the second cosine value to obtain a first position vector; The first sine value and the second sine value are combined to obtain a second position vector.

6. The method according to claim 5, characterized in that The performing feature matching on the first image and the second image according to the first object descriptor and the second object descriptor to obtain a feature matching result includes: Performing multi-scale feature perception on the first target descriptor to obtain a first perception descriptor; Performing multi-scale feature perception on the second target descriptor to obtain a second perception descriptor; According to the first perceptual descriptor and the second perceptual descriptor, feature matching is performed on the first image and the second image to obtain a feature matching result.

7. The method according to claim 6, characterized in that The performing feature matching on the first image and the second image according to the first perceptual descriptor and the second perceptual descriptor to obtain a feature matching result includes: Performing linear projection and normalization processing on the first perceptual descriptor and the second perceptual descriptor to obtain corresponding first processed descriptors and second processed descriptors; Performing an inner product on the first processing descriptor and the second processing descriptor to obtain a matching similarity matrix; generating a first logarithmic score of the matching similarity matrix; transposing the matching similarity matrix and generating a second logarithmic score of the transposed matrix; Integrating the first logarithmic score, the second logarithmic score, the first processing descriptor, and the second processing descriptor to obtain a matching score matrix; Perform feature matching on the first image and the second image according to the matching score matrix to obtain a feature matching result.

8. An image feature matching system, characterized in that: include: A first acquisition module, used to acquire a first feature descriptor of the first image and a second feature descriptor of the second image, wherein the feature descriptor is a descriptor embedded with position information; A first analysis module, configured to perform similarity analysis on features between the first feature descriptor and the second feature descriptor to obtain a feature similarity analysis result; A first adjustment module, configured to adjust the feature weights of the first feature descriptor and the second feature descriptor respectively according to the feature similarity analysis result, to obtain a corresponding first target descriptor and a second target descriptor; The first matching module is used to perform feature matching on the first image and the second image according to the first target descriptor and the second target descriptor to obtain a feature matching result.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the image feature matching method as claimed in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image feature matching method according to any one of claims 1 to 7 are implemented.

Images