Feature matching method, system and device based on multi-geometry collaborative learning and storage medium

Abstract

The application provides a feature matching method, system and device based on multi-geometry collaborative learning and a storage medium, and relates to the technical field of robot positioning and navigation. The steps include: acquiring an image pair, extracting two groups of sparse local features in the image pair and inputting the local features into a multi-geometry collaborative network model; enhancing the local features through an attention mechanism and iteratively updating the sparse network through a multi-layer perception and / or linear projection; generating and screening reliable matching points through affine transformation estimation, position coding and key point neighborhood expansion; constraining the matching process through pose information and adaptively adjusting the matching strategy for matching prediction; combining real matching information and homography geometry information for loss calculation to train the multi-geometry network model; and repeating the steps until a preset iteration number is reached. Through collaborative optimization of affine geometry, epipolar geometry and homography geometry, the application realizes mutual promotion of local feature discriminability and geometric consistency, and can achieve a better balance between speed and accuracy in feature matching.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot positioning and navigation, and particularly relates to a feature matching method and system based on multi-geometry collaborative learning, a device and a storage medium. BACKGROUND

[0002] Feature matching is a basic task in intelligent robot positioning and navigation, aiming to find corresponding points from image pairs of overlapping scenes and eliminate unmatchable points.

[0003] However, feature matching faces challenges such as scene differences such as illumination, viewing angle, texture and appearance changes, and real-time requirements; traditional methods extract sparse features based on key point detectors, use Transformers to achieve matching, and perform well in strong texture scenes, but have limited effect in weak texture scenes; while dense matching methods solve the problem of weak texture matching, but have redundant calculations in strong texture scenes and are low in efficiency; therefore, it is difficult to achieve a better balance between speed and accuracy in feature matching. SUMMARY

[0004] The present application aims to provide a feature matching method and system based on multi-geometry collaborative learning, a device and a storage medium, to solve the problem that the prior art mentioned in the background art cannot achieve a better balance between speed and accuracy in feature matching.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solution: a feature matching method based on multi-geometry collaborative learning, applied to a robot positioning and navigation system, the method steps comprising: S1: obtaining an image pair, extracting two groups of sparse local features in the image pair and inputting them into a multi-geometry collaborative network model to construct a sparse network; S2: enhancing local features through an attention mechanism and iteratively updating the sparse network through a multi-layer perception and / or linear projection; S3: embedding affine geometric information into local feature representation and generating and screening reliable matching points in the key point neighborhood in the sparse network; S4: calculating the pose information of the corresponding reliable matching points, constraining the matching process using epipolar geometric information, and adaptively adjusting the matching strategy for matching prediction; S5: calculating the homography geometric information of the corresponding reliable matching points, combining the real matching information obtained by manual labeling with the homography geometric information to calculate the loss, and training the multi-geometry network model; S6: repeating steps S2-S5 until a preset number of iterations is reached, to optimize the multi-geometry collaborative network model.

[0006] Optionally, the S1 step specifically comprises: acquiring two images of the same scene, and extracting sparse local features in the two images respectively, wherein each local feature in the sparse local features comprises a key point and a descriptor; predicting a partial assignment matrix between the two groups of sparse local features in advance through a multi-geometry collaborative network model, and outputting a matching matrix, and constructing a sparse network through the assignment matrix and the matching matrix.

[0007] Optionally, the S2 step specifically comprises: updating the intermediate state vectors of the two images, aggregating information to enhance local features through a self-attention and cross-attention mechanism, and iteratively updating the intermediate state vectors of different layers through a multi-layer perception and / or nonlinear transformation of linear projection.

[0008] Optionally, the S3 step specifically comprises: estimating affine transformation through the key point position correspondence relationship in the two groups of sparse local features to obtain affine geometric information; wherein the affine transformation estimation equation is: In the formula, the is affine geometric information, the is a rotation component to be solved, the is a translation component to be solved; dense matching points are generated by expanding the key point neighborhood, and multiple rounds of iteration are performed through the embedding of affine geometric information in the attention score to screen the dense matching points to obtain reliable matching points; wherein the cross-attention score uses position encoding to embed the affine geometric information between the image pair I 1 and I 2 , and the calculation formula is: The self-attention score uses the self-relative position in the image I 1 to encode, and the calculation formula is: In the formula, the is a cross-attention score, the is a self-attention score, the is an attention query transpose, the R(p) is a rotation encoding, the kp j is a key point position of the image I 2 , the kp i is a key point position of the image I 1 , the is an attention key of the image I 1 , and the is an attention key of the image I 2 .

[0009] Optionally, the S4 step specifically comprises: converting the state into an enhanced descriptor by learning linear projection, predicting a partial assignment matrix in combination with a matching likelihood score and a similarity score, outputting a matching matrix according to a set threshold, and removing untrustworthy key points according to a confidence classifier and a decay threshold.

[0010] Optionally, the S4 step specifically further comprises: estimating a fundamental matrix through the corresponding reliable matching points, calculating a epipolar distance, respectively pruning sparse matching and dense matching, and determining whether to exit the network layer iteration according to the confidence of the matching points.

[0011] Optionally, the S5 step specifically comprises: calculating homography geometric information of the corresponding reliable matching points, obtaining real matching information through artificial labeling, determining unmatching points according to the relationship between predicted matching information obtained through matching prediction and the real matching information, calculating matching loss, estimating homography using the predicted matching information and calculating homography loss, and obtaining total loss by comprehensively considering the matching loss and the homography loss of each layer.

[0012] In another aspect, the present application also provides a feature matching system based on multi-geometric collaborative learning, comprising: a feature extraction module, configured to obtain an image pair, extract two groups of sparse local features in the image pair, and input the two groups of sparse local features into a multi-geometric collaborative network model to construct a sparse network; a feature enhancement module, configured to enhance local features through an attention mechanism, and iteratively update the sparse network through a multi-layer perception and / or a linear projection; a feature screening module, configured to generate and screen reliable matching points through affine transformation estimation, position coding, and key point neighborhood expansion; a matching prediction module, configured to calculate pose information of the corresponding reliable matching points, constrain a matching process through the pose information, and adaptively adjust a matching strategy for matching prediction; a loss calculation module, configured to calculate homography geometric information of the corresponding reliable matching points, obtain real matching information through artificial labeling, and calculate loss in combination with the real matching information and the homography geometric information; and a model optimization module, configured to repeat the steps until a preset iteration number is reached, so as to optimize the multi-geometric collaborative network model.

[0013] In another aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the feature matching method based on multi-geometric collaborative learning when executing the computer program.

[0014] In another aspect, the present application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the feature matching method based on multi-geometric collaborative learning.

[0015] Compared with the prior art, the application has the beneficial effects that:

[0016] 1. In order to realize the better balance of speed and accuracy of feature matching and improve the matching performance in different texture scenes, the application inputs two groups of sparse local features in an image pair into a multi-geometry collaborative network model to construct a sparse network, and proposes a scheme of enhancing the sparse matching capability of the multi-geometry collaborative network; through the collaborative optimization of affine geometry, epipolar geometry and homography geometry, the mutual promotion of discriminative local features and geometric consistency is realized, the feature robustness is improved through geometric constraints, and high-quality features reduce the ambiguity of geometric solving, forming a positive cycle of end-to-end joint optimization; in the inference stage, multi-geometry constraint fusion can be quickly completed, and the better balance of speed and accuracy of feature matching can be realized.

[0017] 2. In the front end of the multi-geometry collaborative network model, affine geometry information is embedded into local feature representation, and reliable matching points are generated and screened through neighborhood expansion of key points in the sparse network; through affine enhancement auxiliary extraction of enhanced local features, the matching effect in challenging scenes such as weak texture can be improved.

[0018] 3. In the middle end of the multi-geometry collaborative network model, the pose information of the corresponding reliable matching points is calculated, the matching process is constrained through epipolar geometry information, and the matching strategy is adaptively adjusted for matching prediction.

[0019] 4. In the back end of the multi-geometry collaborative network model, the homography geometry information of the corresponding reliable matching points is calculated, the loss calculation is performed combining the real matching information obtained by manual labeling and the homography geometry information, the multi-geometry network model is trained, the homography supervision mechanism is introduced, the pixel-level correspondence relationship is established through homography geometry constraint, and the network learns more accurate matching capability. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 It is a schematic diagram of the overall architecture of the method of the application.

[0021] Figure 2 It is a schematic diagram of the step flow of the method of the application.

[0022] Figure 3 It is a schematic diagram of the system function module of the application.

[0023] Figure 4 It is a schematic diagram of the equipment structure of the application.

[0024] In the figure, 10 is a feature extraction module, 20 is a feature enhancement module, 30 is a feature screening module, 40 is a matching prediction module, 50 is a loss calculation module, 60 is a model optimization module, 70 is a processor, and 80 is a memory. DETAILED DESCRIPTION

[0025] The technical scheme of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments.

[0026] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0027] Those skilled in the art can understand that, unless specifically stated, the singular forms "a", "an" and "the" used herein also include the plural forms. It should be further understood that the use of the phrase "comprise" in the specification of the present application means that a feature, integer, step, operation, element and / or component exists, but does not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intermediate elements. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any of the associated listed items and all combinations thereof.

[0028] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0029] It should be understood that the sequence numbers and sizes of the steps in the embodiments do not mean the order of execution, and the execution order of the processes is determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0030] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0031] Please refer to Figures 1-2 The feature matching method based on multi-geometry collaborative learning is applied to a robot positioning and navigation system, and the method steps include:

[0032] S1, an image pair is obtained, and two groups of sparse local features in the image pair are extracted and input to a multi-geometry collaborative network model to construct a sparse network.

[0033] Specifically, the image pair is two images with a certain correlation, which can be understood as two pictures in the same scene in the present application, for example, images of different angles in the same scene, images of the same object at different time points, images with similar content but large style difference, or original images and images after data enhancement, and there is a certain degree of correlation between them. By extracting the sparse local features of the two images and inputting them to the multi-geometry collaborative network model, a sparse network is constructed, which can avoid a large amount of calculation.

[0034] S2, the local features are enhanced through an attention mechanism, and the sparse network is iteratively updated through a multi-layer perception and / or linear projection.

[0035] Specifically, the attention mechanism includes a self-attention mechanism and a cross-attention mechanism, which dynamically focuses on the key areas in the image to enhance the representation ability of the local features, and can improve the representation accuracy and robustness of the local features. By using a multi-layer perception to mine the corresponding relationship of deep local features, and by mapping high-dimensional sparse features to a low-dimensional space, the amount of calculation is reduced, and the computational complexity is controlled.

[0036] S3, affine geometric information is embedded in the local feature representation, and reliable matching points are generated and selected in the key point neighborhood expansion in the sparse network.

[0037] Specifically, affine geometric information is obtained by estimating affine transformation through the corresponding relationship of key point positions in the two groups of sparse local features, dense matching points are generated in the key point neighborhood expansion, and the affine geometric information is embedded in the attention score for multiple iterations to select the reliable matching points from the dense matching points, which can improve the quantity and quality of image feature matching, and the rotation code in the affine geometric information helps the model to capture global and relative position information.

[0038] S4, the pose information of the corresponding reliable matching points is calculated, the matching process is constrained by epipolar geometric information, and the matching strategy is adaptively adjusted for matching prediction.

[0039] Specifically, based on adaptive prediction embedding pose guidance, guided matching prediction is performed to complete the matching prediction task. By learning linear projection, the state is converted into an enhanced descriptor. The matching possibility score and the similarity score are combined to predict the partial assignment matrix. Then, the matching matrix is output according to the set threshold. The corresponding reliable matching points are used to estimate the fundamental matrix. The epipolar distance is calculated. Based on the epipolar geometric constraint, the abnormal points in the initial corresponding relationship are detected and removed. The epipolar geometry is used to optimize the matching process. The sparse matching and dense matching are pruned respectively. Whether to exit the network layer iteration is determined according to the confidence of the matching points. The matching points are screened and optimized through the epipolar constraint. The guided matching prediction is fast, and the redundancy is reduced. The network layer iteration is from coarse to fine.

[0040] S5, homographic geometric information of the corresponding reliable matching points is calculated. The real matching information obtained by manual annotation and the homographic geometric information are combined to calculate the loss, so as to train the multi-geometric network model.

[0041] Specifically, the homographic geometric information of the corresponding reliable matching points is calculated, and the real matching information is obtained by manual annotation. The unmatching points are determined according to the relationship between the predicted matching information obtained by matching prediction and the real matching information. The matching loss is calculated. The homography is estimated by using the predicted matching information, and the homographic loss is calculated. The total loss is obtained by comprehensively considering the matching loss and the homographic loss of each layer. In the process of training the multi-geometric collaborative network model under multi-supervision, the loss is calculated by combining the real matching information and the homographic geometric information, which greatly improves the performance of the multi-geometric collaborative network model.

[0042] S6, steps S2-S5 are repeated until a preset iteration number is reached, so as to optimize the multi-geometric collaborative network model.

[0043] Specifically, first, the network is pre-trained on a large synthetic homography dataset sampled from the Oxford-Paris retrieval-1M image; then, fine-tuning is continued on the MegaDepth dataset; the model is realized by using PyTorch, and is matched with the Adam optimizer. The batch size is set to 128 during pre-training, and is set to 32 during fine-tuning. The initial learning rate is set to 0.0001 on a 2x3090 GPU with 24GB of video memory. The iteration layer L=9 is adaptive depth and width, and is also affected by the method of affine enhancement and pose guidance. It is worth noting that the multi-geometric matrix estimated at the Lth layer is approximately equal to the estimated value of some estimators. After training, a good MGC-Net model with good performance can be obtained, which can be used for subsequent image matching and related tasks.

[0044] It can be understood that, in order to achieve a better balance between speed and accuracy of feature matching, and to improve the matching performance in different texture scenes, the application obtains an image pair, extracts two groups of sparse local features in the image pair, inputs the two groups of sparse local features into a multi-geometry collaborative network model, constructs a sparse network, and proposes a scheme for enhancing the sparse matching capability of the multi-geometry collaborative network; the local features are enhanced through the attention mechanism, and the sparse network is iteratively updated through the multi-layer perception and / or linear projection, which greatly improves the computing efficiency; in the front end of the multi-geometry collaborative network model, affine geometric information is embedded into the local feature representation, and reliable matching points are generated and screened in the key point neighborhood in the sparse network; the enhanced local features are extracted through affine enhancement auxiliary extraction, which can improve the matching effect in challenging scenes such as weak texture; in the middle end of the multi-geometry collaborative network model, the pose information of the corresponding reliable matching points is calculated, the matching process is constrained through epipolar geometry information, and the matching strategy is adaptively adjusted for matching prediction; in the back end of the multi-geometry collaborative network model, the homography geometric information of the corresponding reliable matching points is calculated, and the true matching information is obtained through artificial labeling; the true matching information and the homography geometric information are combined for loss calculation to train the multi-geometry network model, introduce a homography supervision mechanism, establish a pixel-level correspondence relationship through homography geometric constraint, and drive the network to learn more accurate matching capability; through the collaborative optimization of affine geometry, epipolar geometry and homography geometry, the multi-geometry collaborative network model realizes the mutual promotion of local feature discriminability and geometric consistency, improves feature robustness through geometric constraint, and reduces geometric solution ambiguity through high-quality features, forming a positive cycle of end-to-end joint optimization; in the inference stage, the multi-geometry constraint fusion can be quickly completed, and a better balance between speed and accuracy of feature matching can be achieved.

[0045] In some embodiments, the S1 step specifically includes: obtaining two images of the same scene, and extracting sparse local features in the two images respectively, wherein each local feature in the sparse local features includes a key point and a descriptor; predicting a partial assignment matrix between the two groups of sparse local features in advance through a multi-geometry collaborative network model, and outputting a group of matching matrices, and constructing a sparse network through the assignment matrix and the matching matrix.

[0046] Specifically, for two images I 1 and I 2 of the same scene, a feature detector and a descriptor are used to process the two images, and two groups of sparse local features F 1 and F 2 are obtained; each local feature is composed of a normalized key point position and a descriptor; the key point position calculation formula is: kp i =(x, y) i ∈[0, 1] 2In the formula, Kp i For key point locations; the descriptor calculation formula is: In the formula, d i d is the descriptor, and d is the vector dimension.

[0047] Furthermore, the multi-geometric collaborative network model pre-predicts the partial assignment matrix P between two sets of sparse local features, and then outputs a matching matrix M. The l-th layer of this sparse network is stacked and iteratively updated by each other through an attention mechanism, while a classifier is set to adaptively determine the depth and width to stop the update.

[0048] Furthermore, at each layer, the similarity and matching scores are combined to predict the layer l assignment matrix (l)P. Based on this prediction, the top N with higher scores are further selected. t A single match is used to estimate multiple geometric relations, because a good local match can approximate a global correspondence.

[0049] Furthermore, the subsequent multi-geometric cooperative network model estimates the multi-geometric relationships at each layer, including affine transformations. Fundamental matrix and homography matrix The fundamental matrix It can be decomposed into poses, and these geometric relationships are co-embedded into the network to learn matching.

[0050] By synergistically optimizing affine geometry, epipolar geometry, and homography geometry, the discriminative power of local features and geometric consistency are mutually promoted. Geometric constraints enhance feature robustness, while high-quality features reduce geometric ambiguity, forming a positive loop of end-to-end joint optimization. In the inference stage, the fusion of multiple geometric constraints can be completed quickly, achieving a better balance between speed and accuracy in feature matching. The multi-geometric collaborative learning strategy proposed in this application can be effectively applied to the localization and navigation of intelligent robots.

[0051] In some embodiments, step S2 specifically includes: updating the intermediate state vectors of the two images, combining self-attention and cross-attention mechanisms to aggregate information and enhance local features, and iteratively updating the intermediate state vectors of different layers through nonlinear transformations of multilayer perceptrons and / or linear projections.

[0052] Specifically, from the two images I 1 and I 2 All corresponding intermediate states and In the middle, arbitrarily select an intermediate state where the number of layers l < 10. (l) x i When the layer number l = 0, it is initialized to the descriptor d. iUsing a multilayer perceptron combined with messages obtained from the aggregation of all local features, the intermediate state of the l-th layer is updated according to a calculation formula, where the calculation formula for the intermediate state of the l-th layer is: (l) x i (l-1) x i +mlp([ (l-1) x i |m i ]), where, the (l) x i This is an intermediate state, where [·|·] represents the concatenation of two vectors, and m... i For message.

[0053] Furthermore, the message m i Message aggregation by self-attention and cross-attention aggregation messages It consists of two parts, assuming state x i In image I 1 In the middle, self-attention aggregation messages Through image I 1 Each local feature in the dataset is combined with a self-attention score. Aggregation is performed, and the calculation formula is as follows: In the formula, the For self-attention aggregation messages, the Let be the self-state attention value; where is the self-state attention value. It is through a learnable linear projection onto image I 1 self-state in The calculation formula is as follows: In the formula, the For the self-state attention value, the As a weighting term, the As a self-state, the For bias terms; while cross-attention aggregates messages Then through image I 2 Each local feature in the dataset is combined with a cross-attention score. The aggregation is performed, and the calculation formula is as follows: In the formula, the To aggregate messages for cross-attention, Here, the cross-state attention value is... It is through a learnable linear projection onto image I 2 Crossover state The calculation formula is as follows: In the formula, the For the cross-state attention value, the As a weighting term, the For cross state, the is a bias term; at the same time, the learning parameters are projected using linear projection Get the corresponding attention key And calculate the attention query Including weight term W q And bias term b q ; Attention score α ij Is the similarity between attention query q i And attention key k j , and the position encoder is embedded inside for different position encoders for self-attention and cross-attention units; By dynamically focusing on the key areas in the image, the representation ability of local features is enhanced, which can improve the representation accuracy and robustness of local features; By multi-layer perception to mine the corresponding relationship of deep local features, and by mapping high-dimensional sparse features to low-dimensional space, the calculation amount is reduced, and the calculation complexity is controlled.

[0054] In some embodiments, the S3 step specifically includes: estimating affine geometric information by key point position correspondence relationship in two groups of sparse local features, generating dense matching points by expanding in the neighborhood of key points, and performing multiple rounds of iteration by embedding affine geometric information in attention score to screen the dense matching points to obtain reliable matching points.

[0055] Specifically, select the top N t > 3 matching key point position correspondences {(x, y) i , (x, y) j} from the image, estimate the 6-DOF affine transformation The affine transformation estimation equation is established as In the formula, is the rotation component to be solved, is the translation component to be solved, which together constitute the affine transformation between images

[0056] Further, the affine geometric information is embedded into the attention score. For cross attention score Use position encoding Embed the affine geometric information between image I 1 And image I 2 , and the calculation formula is: For self-attention score Use the self-relative position in image I 1 To encode, and the calculation formula is: Wherein, the is the transpose of the attention query, the is the attention key of image I 1 , the For image I 2 The attention key, the kp i For image I 1 The key point location, the kp j For image I 2 The key point location, the The affine transformation is estimated by the affine transformation estimation equation; the rotation code calculation formula is: In the formula, R(p) is a rotation encoding that follows Fourier characteristics, and the k-th rotation angle θ k From the learnable angular frequency ω k The parameter p determines whether the model can capture global and relative positions.

[0057] Furthermore, at each sparse keypoint kp i The neighborhood is expanded to a set of dense points to generate a dense matching M. dense Using key point positions kp i Centered on a central point, a uniform expansion of N units is formed at equal Euclidean distances r. e Dense points np i，k =(x,y) i，k ∈[0,1] 2 k = 1, 2, ... N e The calculation formula is as follows: Image I 1 As the source image, through affine transformation np these dense points i，k Projected onto target image I 2 Above, another set of dense points np is obtained. j，k This generates a dense matching M. dense ={(np i，k ,np j，k )},Right now:

[0058] Furthermore, reliable matching points are filtered; at each layer, the projection onto image I is removed. 2 External expansion points, and using a threshold λ 0→L To preserve reliable matching from layer 0 to layer L; by calculating the extension point of the l-th iteration layer. (l) np j，k With the extension point of the Lth iteration layer (L) np j，k The error between them, and the threshold λ. 0→L The comparisons are used to filter out the final reliable matching points, where the error calculation formula is:

[0059] In some embodiments, the S4 step specifically comprises: converting the state into an enhanced descriptor by learning linear projection, predicting the partial assignment matrix combining the matching likelihood score and the similarity score, outputting the matching matrix according to the set threshold, and removing the untrusted key points according to the confidence classifier and the decay threshold.

[0060] Specifically, at each layer l, the learned linear projection is used to convert each state x i into an enhanced descriptor y i , where the weight term W and the bias term b are learning parameters; the matching likelihood score is calculated from these enhanced descriptors , which represents the probability that a key point matches its corresponding point, and the formula is

[0061] Further, the similarity score is recalculated , which reflects the similarity between the key points in the image I 1 and the image I 2 , and the formula is , where is the enhanced descriptor on the image I 1 , and is the enhanced descriptor traversing the image I 2 .

[0062] Further, the assignment matrix P ij is calculated jointly from the matching likelihood score and the similarity score, and the formula is When the assignment matrix P ij is greater than the threshold τ and is the maximum value in the row and column, a set of matches is output, denoted as the sparse match M sparse .

[0063] Further, a confidence classifier c f = sigmoid(mlp(x f )) is set, where f is the index of the local feature, x f is the intermediate state corresponding to the local feature, and the confidence of the predicted assignment of each key point is inferred; a threshold λ l is set at the l-th layer, and when the confidence classifier c f is less than the threshold λ l , the untrusted key points are pruned and removed, and the threshold is decayed and adjusted according to the validation accuracy of each classifier in each layer.

[0064] In some embodiments, the S4 step specifically further comprises: estimating the fundamental matrix from the corresponding reliable matching points, calculating the epipolar distance, pruning the sparse match and the dense match respectively, and determining whether to exit the network layer iteration according to the confidence of the matching points.

[0065] Specifically, the basis matrix is estimated which indirectly reflects the pose information between images; from the correspondence of the top N t >7 matching points {(x i , y i ), (x j , y j )}, the eight-point algorithm is used for estimation, and the equation is established through iterative optimization: Through the equation, the epipolar line e is calculated from the source image I 1 to the target image I 2 , and the function dist(·) is used to calculate the distance d of the corresponding point p 2 =(x j , y j ) in the target image I j to the epipolar line e.

[0066] Further, on the basis of key point pruning, further use Prune the matching set M containing sparse matching M sparse and dense matching M dense to quickly guide matching prediction; at each layer, use sparse threshold λ sparse and dense threshold λ dense to prune the matching, further reducing redundancy; for sparse matching pruning, since key point pruning has been performed, the sparse threshold λ sparse is set relatively loose; after prediction and key point pruning, the pose guiding module further eliminates outliers in the sparse matching M sparse , calculates the confidence c f of each inlier, and compares it with the threshold λ l of the lth layer to quickly and adaptively determine the network depth; n1+n2 calculates the total number of key points of the two images, and the local feature f belongs to the local feature set {F 1 , F 2}; the condition for exiting the network layer iteration is: In the formula, is an indicator function, which can exit the network layer iteration when the exit parameter exit satisfies the threshold parameter p.

[0067] Further, for dense matching pruning, since rough matching expansion is performed in the affine enhancement module, the dense threshold λ dense is set relatively strict; after removing the expanded points projected outside the target image, the rough dense matching M dense is generated; from the expanded rough dense matching M denseIn the process, matches inconsistent with the global epipolar geometry are removed at each layer, and then a threshold λ is applied from layer 0 to layer L. 0→L Screening is performed; this is because the estimated fundamental matrix... Higher degrees of freedom than affine transformation It can guide the matching from coarse to fine, reducing the burden of network layer iteration.

[0068] In some embodiments, step S5 specifically includes: calculating the homography geometric information of the corresponding reliable matching points, obtaining the true matching information through manual annotation, determining the unmatched points based on the relationship between the predicted matching information and the true matching information obtained from the matching prediction, calculating the matching loss, estimating the homography using the predicted matching information and calculating the homography loss, and obtaining the total loss by combining the matching loss and homography loss of each layer.

[0069] Specifically, in sparse matching prediction, if certain key points ( or Matching the true M gt If they are not close, mark them as unmatched points; calculate their unmatch score respectively. Included in image I 1 Unmatched score and in image I 2 Unmatched score After prediction and fine pruning, P is assigned by minimizing the negative log-likelihood. ij Calculate the matching loss of the l-th layer. (l) L m The calculation formula is as follows:

[0070] Furthermore, with affine transformation and fundamental matrix Similarly, from the top N t Estimating homography of 8 degrees of freedom in matching with a value >4 Used to increase geometric supervision; final prediction M pred Obtained from the affine enhancement module and the pose guidance module, it contains information from the negative log-likelihood minimization assignment P. ij Sparse matching M sparse Dense matching M dense ; Calculate the homography loss of the l-th layer. (l) L h The calculation formula is as follows: In the formula, PE(·) represents the projection error between homography and matching.

[0071] Furthermore, by applying different weights α to the multi-supervision approach, the total loss of layer L is calculated using the following formula:

[0072] In some embodiments, steps S2-S5 are repeated until a preset number of iterations is reached to optimize the multi-geometry collaborative network model.

[0073] Specifically, the network is first pre-trained on a synthetic homography large dataset sampled from the Oxford-Paris retrieval-1M image; then, fine-tuning is continued on the MegaDepth dataset; the model is implemented using PyTorch, with an Adam optimizer, a batch size of 128 for pre-training and 32 for fine-tuning, and an initial learning rate of 0.0001 on a 2x3090 GPU with 24GB of video memory; the iteration number L = 9 is adaptive in depth and width, and is also affected by the methods of affine enhancement and pose guidance; it is worth noting that the multi-geometry matrix estimated at the Lth layer approximates the estimated value of some estimators; after training, a well-performing MGC-Net model can be obtained for subsequent image matching and related tasks.

[0074] Please refer to Figure 3 On the other hand, the present application also provides a feature matching system based on multi-geometry collaborative learning, comprising: a feature extraction module 10 for obtaining an image pair, extracting two groups of sparse local features in the image pair, and inputting the sparse local features into a multi-geometry collaborative network model to construct a sparse network; a feature enhancement module 20 for enhancing local features through an attention mechanism and iteratively updating the sparse network through a multi-layer perception and / or linear projection; a feature screening module 30 for generating and screening reliable matching points through affine transformation estimation, position coding, and key point neighborhood expansion; a matching prediction module 40 for calculating the pose information of the corresponding reliable matching points, constraining the matching process through the pose information, and adaptively adjusting the matching strategy for matching prediction; a loss calculation module 50 for calculating the homographic geometric information of the corresponding reliable matching points, obtaining true matching information through artificial labeling, and combining the true matching information and the homographic geometric information for loss calculation; and a model optimization module 60 for repeating the steps until a preset number of iterations is reached to optimize the multi-geometry collaborative network model.

[0075] Please refer to Figure 4 On the other hand, the present application also provides a computer device comprising a memory 80 and a processor 70, the memory storing a computer program, and the processor implementing the steps of the above-mentioned feature matching method based on multi-geometry collaborative learning when executing the computer program.

[0076] In another aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the feature matching method based on multi-geometry collaborative learning.

[0077] Specifically, Figure 4 is a structural schematic diagram of a feature matching device based on multi-geometry collaborative learning provided by an embodiment of the present application. The feature matching device based on multi-geometry collaborative learning can have great differences due to different configurations or performances, and can include one or more processors 70 (central processing units, CPU) (for example, one or more processors 70) and a memory 80, one or more storage media (for example, one or more mass storage devices) for storing application programs or data; wherein the memory 80 and the storage medium can be temporary storage or persistent storage; further, the processor 70 can be configured to communicate with the storage medium, execute a series of instruction operations in the storage medium on the feature matching device based on multi-geometry collaborative learning, so as to implement the steps of the feature matching method based on multi-geometry collaborative learning provided by each method embodiment.

[0078] If the integrated unit is implemented in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0079] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the computer program can include the processes of the above-mentioned embodiments of each method. Any reference to the memory 80, storage, database or other medium used in each embodiment of the present application can include non-volatile and / or volatile memory 80. The non-volatile memory 80 can include read-only memory 80 (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. The volatile memory 80 can include random access memory 80 (RAM) or external cache memory 80. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synch link) DRAM (SLDRAM), memory 80 bus (Rambus) direct RAM (RDRAM), direct memory 80 bus dynamic RAM (DRDRAM), and memory 80 bus dynamic RAM (RDRAM), etc.

[0080] The above is only an embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent transformation or direct or indirect application in the related technical field using the content of the present application specification and drawings is also included in the patent protection scope of the present application.

Claims

1. A method of feature matching based on multi-geometry co-learning, applied to a robot positioning and navigation system, characterized in that, The method steps include: S1: acquiring an image pair, extracting two groups of sparse local features in the image pair, and inputting the two groups of sparse local features into a multi-geometry collaborative network model to construct a sparse network; S2: enhancing the local features through an attention mechanism, and iteratively updating the sparse network through a multilayer perceptron and / or linear projection; S3: embedding affine geometric information into the local feature representation, and generating and screening reliable matching points in the key point neighborhood expansion in the sparse network; S4: calculating the pose information of the corresponding reliable matching points, constraining the matching process using epipolar geometric information, and adaptively adjusting the matching strategy for matching prediction; S5: calculating the homography geometric information of the corresponding reliable matching points, combining the real matching information obtained by manual labeling with the homography geometric information to calculate the loss, and training the multi-geometry network model; S6: repeating steps S2-S5 until a preset number of iterations is reached to optimize the multi-geometry collaborative network model.

2. The multi-geometry co-learning based feature matching method according to claim 1, wherein, The S1 step specifically includes: Acquiring two images of the same scene, and extracting sparse local features in the two images, wherein each local feature in the sparse local features includes a key point and a descriptor; Pre-predicting a partial assignment matrix between the two groups of sparse local features through a multi-geometry collaborative network model, and outputting a matching matrix, and constructing a sparse network through the assignment matrix and the matching matrix. 3.The multi-geometry co-learning based feature matching method of claim 1, wherein, The S2 step specifically includes: Updating the intermediate state vectors of the two images, aggregating information to enhance the local features through self-attention and cross-attention mechanisms, and iteratively updating the intermediate state vectors of different layers through nonlinear transformation of a multilayer perceptron and / or linear projection. 4.The multi-geometry co-learning based feature matching method of claim 1, wherein, The S3 step specifically includes: Estimating affine geometric information through the correspondence of key point positions in the two groups of sparse local features; wherein the affine transformation estimation equation is: In the formula, the is affine geometric information, the is a rotation component to be solved, the is a translation component to be solved; Generating dense matching points in the key point neighborhood expansion, and iteratively filtering the dense matching points through the embedding of affine geometric information in the attention score to obtain reliable matching points. where the cross-attention scores encode affine geometric information between the image pairs I 1 and I 2 using positional encoding, computed as: The self-attention scores encode self-relative positions in the image I 1 using positional encoding, computed as: In the formula, the For cross-attention scores, the For self-attention scores, the stated For attention query transpose, R(p) is rotation encoding, and kp is... j For image I 2 The key point location, the kp i For image I 1 The key point location, the For image I 1 The attention key, the For image I 2 Attention key.

5. The multi-geometry co-learning based feature matching method of claim 1, wherein, The S4 step specifically includes: Converting the state into enhanced descriptors through learning linear projection, predicting a partial assignment matrix by combining matching possibility scores and similarity scores, then outputting a matching matrix according to a set threshold, and removing untrustworthy key points according to a confidence classifier and a decay threshold.

6. The multi-geometry co-learning based feature matching method of claim 4, wherein, The S4 step specifically further includes: Estimating a fundamental matrix through the corresponding reliable matching points, calculating epipolar distances, respectively pruning sparse matching and dense matching, and determining whether to exit network layer iteration according to the confidence of the matching points.

7. The multi-geometry co-learning based feature matching method of claim 1, wherein, The S5 step specifically includes: Calculating the homography geometric information of the corresponding reliable matching points, obtaining real matching information through manual labeling, determining unmatched points according to the relationship between the predicted matching information obtained by matching prediction and the real matching information, calculating matching loss, estimating homography using predicted matching information and calculating homography loss, and obtaining total loss by integrating the matching loss and the homography loss of each layer.

8. A feature matching system based on multi-geometry co-learning, characterized in that, It includes: The feature extraction module is configured to obtain an image pair, extract two groups of sparse local features in the image pair, and input the two groups of sparse local features into a multi-geometric collaborative network model to construct a sparse network. The feature enhancement module is configured to enhance the local features by using an attention mechanism, and iteratively update the sparse network by using a multi-layer perception and / or linear projection. The feature screening module is configured to generate and screen reliable matching points by using affine transformation estimation, position coding, and key point neighborhood expansion. The matching prediction module is configured to calculate pose information of the reliable matching points corresponding to each other, constrain a matching process by using the pose information, and adaptively adjust a matching strategy to perform matching prediction. The loss calculation module is configured to calculate homography geometric information of the reliable matching points corresponding to each other, obtain real matching information by using artificial labeling, and perform loss calculation by combining the real matching information and the homography geometric information. The model optimization module is configured to repeat the steps until a preset iteration number is reached, and optimize the multi-geometric collaborative network model. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the feature matching method based on multi-geometric collaborative learning in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the feature matching method based on multi-geometric collaborative learning in any one of claims 1 to 7.

Images