System for three-dimensional geometry-guided student-teacher feature matching (3dg-stfm)

Abstract

A system for teacher-student feature matching for three-dimensional geometric guidance includes a multi-modal teacher model, a single-modal student model, a coarse-level knowledge transfer loss function, and a fine-level knowledge transfer loss function. The multi-modal teacher model is configured to determine feature matching between a pair of RGB-D images, each RGB-D image being a combination of an RGB image and its corresponding depth image. The single-modal student model is configured to determine feature matching from the pair of RGB images and the teacher model, the teacher model guiding the student model to learn RGB-induced depth information for coarse-level and fine-level feature matching. The coarse-level knowledge transfer loss function is configured to determine a loss for transferring coarse-level matching knowledge from the teacher model to the student model. The fine-level knowledge transfer loss function is configured to determine a loss for transferring fine-level matching knowledge from the teacher model to the student model, and to guide the student model to preferentially learn a fine-level prediction distribution.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a system for three-dimensional geometry-guided teacher-student feature matching (3DG-STFM) to establish correspondences between overlapping images. Background Technology

[0002] Establishing correspondences between overlapping images is crucial for many computer vision tasks, such as Structure for Motion Inference (SfM), Simultaneous Localization and Mapping (SLAM), and visual localization.

[0003] Most existing techniques for solving this problem follow a classic three-stage process: feature detection, feature description, and feature matching. To generate robust descriptors unaffected by changes in lighting or viewpoint, traditional manual feature descriptors, such as SIFT, SURF, and BRISK, and deep network representations of features have been extensively studied. Point-to-point correspondences can be generated through matching algorithms based on nearest neighbor search or learnable matching strategies.

[0004] SuperGlue, a learning-based matching system utilizing graph neural networks (GNNs), constructs densely connected graphs between two sets of keypoints. Self-attention and cross-attention mechanisms are used to integrate and exchange the geometric correlations and visual features of keypoints within the GNN. However, detector-based local feature matching algorithms only produce sparse keypoints, especially in low-texture regions.

[0005] Therefore, the aforementioned defects and deficiencies still need to be addressed in this field. Summary of the Invention

[0006] Based on the aforementioned deficiencies and shortcomings of the existing technology, one objective of this invention is to provide a framework / architecture that utilizes RGB-induced depth information to improve local feature matching performance. Another objective of this invention is to provide a model compression system to train efficient, lightweight models, thereby consuming fewer computational resources and achieving higher matching quality and accuracy. Specifically, this invention provides a 3D geometry-guided teacher-student feature matching (3DG-STFM) system, a teacher-student learning framework that can transfer deep knowledge learned by a multimodal teacher model to a unimodal student model, thereby improving local feature matching.

[0007] In one aspect of the invention, the system includes a multimodal teacher model and a unimodal student model. The teacher model is used to determine feature matching between a pair of RGB-D images, wherein each RGB-D image is a combination of an RGB image and its corresponding depth image. The depth image is an image channel, where each pixel is related to the distance between an image plane in the RGB image and a corresponding object. The unimodal student model is used to determine feature matching from the RGB image pair and the teacher model, wherein the teacher model guides the student model to learn RGB-induced depth information for coarse and fine-grained feature matching. The system also includes a coarse-level knowledge transfer loss function and a fine-grained knowledge transfer loss function. The coarse-level knowledge transfer loss function determines the loss for transferring coarse-level matching knowledge from the teacher model to the student model, and the fine-grained knowledge transfer loss function determines the loss for transferring fine-grained matching knowledge from the teacher model to the student model, and guides the student model to preferentially learn a fine-grained prediction distribution.

[0008] In one embodiment, each teacher model and student model includes a Feature Pyramid Network (FPN) for extracting coarse local features with 1 / 8 of the original image resolution and fine features with 1 / 2 of the original image resolution from the image pair.

[0009] In one embodiment, the FPN of the teacher model is a four-channel input of an RGB-D image, and the FPN of the student model is a three-channel input of an RGB image, where D represents depth information.

[0010] In one embodiment, each teacher model and student model further includes a coarse-level local feature transformation module, a coarse-level matching module, and a fine-level matching module.

[0011] In one embodiment, the coarse-level local feature transformation module is used for:

[0012] The extracted coarse feature map is flattened into a one-dimensional vector;

[0013] Positional encoding is performed on the flattened one-dimensional vector to assign a unique embedding vector in sinusoidal format to each feature entry, thereby encoding spatial information; and

[0014] The encoded local feature vectors are processed by a coarse-level matching transformer that includes self-attention and cross-attention layers to generate a logical layer containing a correlation matrix.

[0015] In one embodiment, the self-attention layer and the cross-attention layer are interleaved in the coarse-level matching converter. c Second-rate.

[0016] In one embodiment, the coarse-level matching module is used for:

[0017] By adjusting the correlation matrix S in the row and column directions cApplying a double softmax operation to generate the confidence score matrix P c ,in:

[0018]

[0019] Let be the matching probability at position (i, j); and

[0020] Selecting a probability score higher than the threshold The matching pairs are used to generate coarse-level matching predictions.

[0021] In one embodiment, given a truth matrix derived from the corresponding labels, the cross-entropy loss, which measures the performance of the coarse-level local feature transformation module and the coarse-level matching module, is calculated using the following formula:

[0022]

[0023]

[0024] in The corresponding set generated for the true labels, with FL having a predicted probability p, is the focus loss term used to address the imbalance between matching and non-matching pairs.

[0025] In one embodiment, the fine-level matching module is used for:

[0026] Project the coarse matching position (i,j) onto the position in the fine-level feature map. At point ), features are extracted from adjacent windows of size w×w as matching candidates;

[0027] Load the selected coarse-level features and associate the loaded coarse-level features with the fine-level features; and

[0028] The associated fine-grained features are processed by a fine-grained matching converter to predict sub-pixel-level correspondences.

[0029] In one embodiment, the fine-grained matching converter includes a self-attention layer L. f And cross-attention layers to aggregate fine-grained local information and generate heatmap distributions for location refinement.

[0030] In one embodiment, the final loss based on direct supervision is calculated according to the following formula:

[0031]

[0032] in( () represents the ground truth location for converting an image solution into a fine-grained heatmap legend. For predictions associated with a rough location, The total variance of the heatmap distribution. This is the fine-grained matching set predicted by the module.

[0033] In one embodiment, the total variance of the similarity distribution is randomly assigned weights to each fine-grained match, where the larger total variance is the unknown prediction associated with the lower weights.

[0034] In one embodiment, a coarse-level knowledge transfer loss function is used to divide the logic layer into multiple independent query distributions and guide the student model to learn these distributions.

[0035] In one embodiment, the mutual query deviation (MQD) loss is determined by a coarse-level knowledge transfer loss function, which utilizes knowledge transfer across all mutual query distributions using the following formula:

[0036]

[0037]

[0038]

[0039]

[0040] in and The query distributions for students and teachers extracted at temperature T are used respectively, and the additional focus loss weight FL is added to balance the true / false pairs of matching / mismatching. Standard confidence scores predicted by the student model. Coarse-level total MQD loss. Let be the average of the knowledge distillation (KD) loss for all n distributions, where n equals 2 × hw, h is the height of the feature map, and w is the width of the feature map.

[0041] In one embodiment, a fine-grained knowledge transfer loss function is used to assign weights to the predictions of the teacher model, with predictions having higher certainty being assigned greater weights.

[0042] In one embodiment, the heatmaps of the teacher model and the student model are respectively ( )and ( The Gaussian distribution of ).

[0043] In one embodiment, the attention loss used to help the student model learn a Gaussian distribution is determined by a fine-grained knowledge transfer loss function. ):

[0044]

[0045] in, and Output the expected values ​​of the distributions for students and teachers, respectively, and their corresponding sets at the finer levels. (in) (Corresponding match)

[0046] In one embodiment, the attention loss is an L2 distance loss, which pays more attention to high attention weights. Related predictions.

[0047] In one embodiment, teacher predictions with smaller total variance indicate that the teacher model is very specific about the corresponding location. In this case, the loss can be given a large weight to guide the student model to preferentially learn specific predictions from the teacher model.

[0048] In one embodiment, the final losses for the teacher model and the student model are respectively:

[0049] ;and

[0050] .

[0051] These and other aspects of the invention will become apparent from the following figures and description of preferred embodiments, although variations and modifications therein may affect the invention without departing from the spirit and scope of the novel concept of this disclosure. Attached Figure Description

[0052] The accompanying drawings are used to illustrate one or more embodiments of the invention and, together with the specification, to explain the principles of the invention. The same reference numerals may be used throughout the drawings to illustrate the same or similar elements in the embodiments.

[0053] Figure 1 This is a schematic diagram of an embodiment of the system for three-dimensional geometry-guided teacher-student feature matching (3DG-STFM) of the present invention.

[0054] Figure 2 This is a schematic diagram of an embodiment of the system of the present invention used in 3DG-STFM.

[0055] Figure 3 This is a schematic diagram of an embodiment of the coarse-order differentiable matching mechanism of the present invention.

[0056] Figure 4 This is a schematic diagram of an embodiment of the coarse-level knowledge converter of the present invention.

[0057] Figure 5 This is a schematic diagram of an embodiment of the fine-level attention knowledge converter of the present invention.

[0058] Figure 6 This is a block diagram of an embodiment of the system of the present invention used in 3DG-STFM. Detailed Implementation

[0059] The invention will now be described more fully below with reference to the accompanying drawings, in which exemplary embodiments of the invention are shown. However, the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. The same reference numerals refer to the same elements throughout.

[0060] The terms used in this specification generally have their ordinary meanings in the art, in the context of this invention, and in the specific context in which each term is used. Certain terms used to describe the invention are discussed below or elsewhere in the specification to provide practitioners with additional guidance regarding the description of the invention. For convenience, certain terms may be highlighted, for example, using italics and / or quotation marks. The use of highlighting does not affect the scope and meaning of the terms; in the same context, the scope and meaning of the terms are the same whether they are highlighted or not. It is understood that the same thing can be described in more than one way. Therefore, alternative languages ​​and synonyms may be used for any one or more terms discussed herein, without having any particular meaning as to whether the terms are elaborated or discussed herein. Synonyms for certain terms are provided. The recitation of one or more synonyms does not preclude the use of other synonyms. Examples used anywhere in this specification, including examples of any terms discussed herein, are merely illustrative and in no way limit the scope and meaning of the invention or any of the example terms. Similarly, the invention is not limited to the various embodiments given in this specification.

[0061] It should be understood that, as used herein and throughout the claims, the terms “a,” “an,” and “the” include plural references unless the context clearly specifies otherwise. Furthermore, it should be understood that when an element is referred to as being “on” another element, it may be directly on the other element or there may be intermediate elements between them. Conversely, when an element is referred to as being “directly on” another element, no intermediate elements are present. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0062] It should be understood that although the terms first, second, third, etc., may be used herein to describe various elements, components, regions, layers, and / or portions, these elements, components, regions, layers, and / or portions should not be limited by these terms. These terms are used only to distinguish one element, component, region, layer, or portion from another element, component, region, layer, or portion. Therefore, without departing from the teachings of the invention, the first element, component, region, layer, or portion discussed below may be referred to as the second element, component, region, layer, or portion.

[0063] It will be further understood that the terms “comprises” and / or “comprising” or “includes” and / or “including” or “has” and / or “having”, or “carry” and / or “carrying”, or “contain” and / or “containing” or “involve” and / or “involving”, etc., are open-ended, meaning including but not limited to. When used in this invention, they specify the presence of the stated features, regions, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, regions, integers, steps, operations, elements, components, and / or groups thereof.

[0064] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in common dictionaries, "term" should be interpreted as having a meaning consistent with its meaning in the context of the relevant technology and this invention, and should not be interpreted in an idealized or overly formal manner unless explicitly defined herein.

[0065] The following description is illustrative in nature and is in no way intended to limit the invention, its application, or its uses. The broad teachings of the invention can be implemented in many forms. Therefore, although the invention includes specific examples, its true scope should not be limited thereto, as other modifications will become apparent upon examination of the drawings, description, and appended claims. For clarity, the same reference numerals will be used in the drawings to identify the same elements. As used herein, at least one of the phrases A, B, and C should be interpreted as representing the logic (A or B or C) using non-exclusive logical OR. It should be understood that one or more steps in the method can be performed in different orders (or simultaneously) without altering the principles of the invention.

[0066] As used herein, the term "module" can refer to, be part of, or include application-specific integrated circuits (ASICs); electronic circuits; combinational logic circuits; field-programmable gate arrays (FPGAs); processors (shared, dedicated, or grouped) that execute code; other suitable hardware components that provide the aforementioned functionality; or combinations thereof, such as in a system-on-a-chip. The term "module" can include memory (shared, dedicated, or grouped) that stores code executed by the processor.

[0067] These apparatuses and methods will be described in detail below and illustrated in the accompanying drawings by various blocks, components, circuits, processes, algorithms, etc. (collectively, “elements”). These elements can be implemented using electronic hardware, computer software, or any combination thereof. Whether these elements are implemented as hardware or software depends on the specific application and design constraints imposed on the system as a whole. For example, an element, or any part of an element, or any combination of elements, can be implemented as a “processing system” including one or more processors. Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, system-on-a-chip (SoCs), baseband processors, field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functions described throughout this disclosure. One or more processors in a processing system can execute software. Software should be interpreted broadly as instructions, instruction sets, code, code segments, program code, programs, subroutines, software components, application programs, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether it is called software, firmware, middleware, microcode, hardware description language, or other names.

[0068] Therefore, in one or more example embodiments, the described functionality can be implemented in hardware, software, or any combination thereof. If implemented in software, the functionality can be stored on or encoded as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media. Storage media can be any available medium that is accessible to a computer. By way of example and not limitation, such computer-readable media can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, combinations of computer-readable media of the types described above, or any other medium that can be used to store computer-executable code in the form of computer-accessible instructions or data structures.

[0069] The following description is illustrative in nature and is in no way intended to limit the invention, its application, or its uses. The broad teachings of the invention can be implemented in many forms. Therefore, although the invention includes specific examples, its true scope should not be limited thereto, as other modifications will become apparent upon examination of the drawings, description, and appended claims. For clarity, the same reference numerals will be used in the drawings to identify the same elements. It should be understood that one or more steps of the method can be performed in a different order (or simultaneously) without altering the principles of the invention.

[0070] Establishing correspondences between overlapping images is crucial for many computer vision tasks, such as Structure for Motion Inference (SfM), Simultaneous Localization and Mapping (SLAM), and visual localization.

[0071] Some existing techniques attempt to avoid the detection step and establish fully supervised dense matching by considering all points of a regular grid. NCNet proposes an end-to-end approach by directly learning dense correspondences, enumerating all possible matches between two images and constructing a 4D correlation tensor graph. The 4D consensus community network has learned to identify reliable matching pairs and filter out unreliable ones accordingly. Based on this concept, Sparse NCNet processes the 4D correlation graph through submanifold sparse convolutions to improve NCNet's efficiency and performance. Furthermore, DRC-Net proposes a coarse-to-fine approach to form dense correspondences with higher accuracy.

[0072] A recent proposal proposes a Local Feature Transformer (LoFTR) method to learn global consistency between image correspondences. To improve memory efficiency, coarse-level matching features are first predicted by a large LoFTR and then fed into a small LoFTR to produce the final fine-level matches. Thanks to the global receptive field of the transformer, LoFTR significantly improves matching performance.

[0073] All of the dense local feature matching methods described above require dense ground truth correspondences as supervision. To find pixel-level ground truth correspondences, point reprojections from one image to another are computed using depth maps, camera intrinsic and extrinsic matrices. On the other hand, given a set of images with dense correspondences, 3D scenes and depth maps can be easily reconstructed using triangulation. Therefore, depth information can be implicitly provided by dense correspondence supervision.

[0074] However, to our knowledge, existing technologies have not explored depth modalities to further improve their matching performance. Unlike RGB images, depth maps provide 3D information that can explicitly describe geometric distributions, and the depth modalities introduced in this paper can offer dual benefits.

[0075] First, even with low-quality or sparse depth information, it can still eliminate much of the blurring in 2D image space and enhance the geometric consistency of feature matching, which is very difficult when using only RGB input, and especially when there are multiple similar objects in an image pair. In this case, most existing techniques tend to look for unreliable matching candidates because they purely distinguish 2D descriptors without depth or size knowledge. Second, low-texture regions of individual objects plague 2D descriptors in terms of implementing dense and consistent matching. This deficiency can also be well addressed by utilizing the recognition of depth modalities.

[0076] Despite the advantage of depth information, high-quality RGB-D inputs can only be collected in well-controlled laboratory environments, and are rare, especially for low-cost consumer devices, capable of capturing similar well-aligned RGB-D pairs in real-world scenarios. Most imaging systems are equipped with only RGB sensors as inputs and cannot afford the high computational costs associated with multimodal inference.

[0077] This makes naive multimodal fusion of RGB and depth inputs during inference and training a limiting solution. Therefore, in real-world scenarios, considering hardware and computational constraints, a better method is needed to convert expensive RGB-D knowledge into RGB modal inference.

[0078] Teacher-student learning has been actively researched in knowledge transfer environments, including model compression, acceleration, and cross-modal knowledge transferors. Given a well-trained, high-weighted teacher model, the goal of teacher-student learning is to extract and compress knowledge from the teacher and guide a low-weighted student model to achieve better performance. On the other hand, multimodal data often provides more valuable supervision and is beneficial to model performance compared to single-modal data. However, due to the lack of data or labels for some modalities during training or testing, knowledge transfer between different modalities is crucial.

[0079] In view of the aforementioned deficiencies and shortcomings of the prior art, this invention provides a system / framework that, compared to the prior art's use of dense correspondence ground truth as direct supervision for local feature matching training, trains 3DG-STFM: a system for 3D geometry-guided teacher-student feature matching, a multimodal matching model (teacher) for implementing depth consistency under 3D dense correspondence supervision, and transferring knowledge to a 2D unimodal matching model (student). Each teacher and student model contains two transformer-based matching modules that can acquire dense correspondences from coarse to fine. The teacher model guides the student model to learn RGB-induced depth information for matching on coarse and fine branches. In one embodiment, 3DG-STFM is evaluated based on a model compression task. To the best of our knowledge, this 3DG-STFM is the first teacher-student learning method for local feature matching tasks. Experiments show that this method outperforms state-of-the-art methods on indoor and outdoor camera pose evaluation and homography evaluation problems.

[0080] Please refer to Figure 1This teacher-student learning system typically includes a multimodal teacher model and a unimodal student model. The multimodal teacher model determines feature matching between pairs of RGB-D images, each RGB-D image being a combination of an RGB image and its corresponding depth image, where the depth image is an image channel, and each pixel in that channel is related to the distance between the image plane in the RGB image and the corresponding object. The unimodal student model determines feature matching from the RGB image pairs and the teacher model, where the teacher model guides the student model to learn RGB-induced depth information for coarse and fine-grained feature matching.

[0081] In some embodiments, each teacher model and student model includes a feature pyramid network (FPN) for extracting coarse local features with 1 / 8 of the original image resolution and fine features with 1 / 2 of the original image resolution from the image pair.

[0082] In some embodiments, the FPN of the teacher model is a four-channel input of an RGB-D image, and the FPN of the student model is a three-channel input of an RGB image, where D represents depth information.

[0083] In some embodiments, each teacher model and student model further includes a coarse-level local feature transformation module, a coarse-level matching module, and a fine-level matching module. The coarse-level local feature transformation module includes a large converter for coarse-level matching, and the fine-level matching module includes a small converter for fine-level matching. The large converter includes a self-attention layer and a cross-attention layer to aggregate coarse-level local information and generate a confidence score matrix for coarse-level matching prediction. High-scoring coarse-level matches are fed into the fine-level matching module for refinement. The small converter includes a self-attention layer and a cross-attention layer to aggregate fine-level local information and generate a heatmap distribution for location refinement.

[0084] The system also includes a coarse-level knowledge transfer loss function and a fine-level knowledge transfer loss function. The coarse-level knowledge transfer loss function determines the loss for transferring coarse-level matching knowledge from the teacher model to the student model. The fine-level knowledge transfer loss function determines the loss for transferring fine-level matching knowledge from the teacher model to the student model, and guides the student model to preferentially learn the fine-level prediction distribution.

[0085] In some embodiments, the coarse selection position of the student model is sent to the teacher branch to extract fine-grained features of the teacher model at the same position.

[0086] In some embodiments, the coarse-level knowledge transfer loss function includes mutual query deviation (MQD) loss to transfer coarse-level knowledge matching from the teacher model to the student model. The MQD loss divides the logic layer into multiple independent query distributions and guides the student model to learn these distributions.

[0087] In some embodiments, the fine-grained knowledge transfer loss function includes an attention loss to transfer fine-grained knowledge matching from the teacher model to the student model. The attention loss assigns weights to the teacher model's predictions, with predictions of higher certainty being assigned larger weights. This attention loss guides the student model to preferentially learn the fine-grained prediction distribution.

[0088] Figure 2 This is a schematic diagram of an embodiment of the teacher-student learning system of the present invention.

[0089] For each teacher-student model / branch, FPN is used to extract coarse local features with 1 / 8 of the original image resolution from the image pair. and fine-grained features with 1 / 2 the resolution of the original image .like Figure 2 As shown, ( For the teaching branch, ( The student branch is provided. The teacher model's FPN input channel has four channels for exploring additional depth maps.

[0090] For matching purposes, the coarse local feature transformation module is a converter-based coarse matching module. In some embodiments, the coarse feature dimension is 1 / 8 of the input image. For example, if the input RGB image is 640×640×3, the coarse feature is 80×80×c, where c is the number of channels in the feature map, depending on the structure of the deep network. In one embodiment, the flattening (reshaping) operation is only performed on the first two dimensions. Due to its high performance, the converter-based architecture is applied in teacher-student learning systems. The coarse feature map of dimension h×w×c is flattened to hw×c, and positional encoding assigns a unique embedding vector in sinusoidal format to each feature entry to encode their spatial information. The encoded local feature vectors are fed into the coarse matching converter. Unlike classic visual converters that focus on self-attention, the matching converter considers the relationships between different image pixels by adding a cross-attention layer. This self-attention layer and the cross-attention layer are interleaved in the coarse matching converter. c Second-rate.

[0091] like Figure 2 As shown, a coarse-to-fine module approach is employed to reduce computational cost. Coarse predictions with high confidence scores are selected and mapped to fine-level feature maps. Location data is collected through a w×w window. The surrounding features are fed into a self-attention layer L. f A lightweight fine-grained converter with a cross-attention layer. The fine-grained matching module is used to predict sub-pixel-level correspondences.

[0092] For the coarse-level matching module Figure 3 This shows the output of the logic layer, with a correlation matrix S of size hw×hw. cThe double softmax operation is used to generate the confidence score matrix Pc:

[0093]

[0094] The softmax function is applied to the correlation matrix S. c Horizontal direction S c (i, ) j and vertical direction S c ( ,j) i The product of these two values ​​is the final matching probability. Given the truth matrix derived from the corresponding labels, the cross-entropy loss is calculated as follows:

[0095]

[0096]

[0097] Where P c The confidence score matrix is... The corresponding set generated for the truth labels. The FL with predicted probability p is the focus loss term, used to resolve the imbalance between matching and non-matching pairs.

[0098] In some embodiments, the coarsely selected locations of the student model are sent to the teacher branch to extract fine-grained features at the same locations in the teacher model.

[0099] For the fine-grained matching module, based on the confidence score matrix Pc, the fine-grained matching module selects and refines probability scores higher than the threshold. The matching pairs. Project the coarse matching position (i,j) onto the fine-level feature map. At the selected coarse-level position, features are extracted from adjacent w×w windows as matching candidates. Before being passed to the fine-grained matching transformer, the selected coarse-level features are sampled and associated with the fine-grained features. The fine-grained matching transformer is a type of transformer containing L... f The attention layer's lightweight transformer aggregates selected contextual information and passes it to the discriminative matching module. The fine-grained matching module does not generate a confidence score matrix; instead, it selects... The central features and their relationship with All features are correlated. Thus, a similarity distribution is generated, and the expected value µ is considered as the prediction. The final loss based on direct supervision is calculated as follows:

[0100]

[0101] in( ) represents the true location for converting an image solution to a finer heatmap scale. To correspond with coarse position Related predictions This represents the total variance of the heatmap distribution. This is the set of fine-grained matches predicted by the module. The total variance of the similarity distribution is randomly weighted for each fine-grained match. A larger total variance indicates that it is an unknown prediction associated with a lower weight.

[0102] To address the coarse-level knowledge transfer loss, a response-based knowledge distillation strategy is employed to help students learn from the teacher at the coarse level. This method can extract the distribution of the logical layer and guide student learning. Figure 3 The output of the logic layer is shown, along with a correlation matrix Sc of size hw×hw. Each row or column describes the relationship between a pixel and every pixel in another image.

[0103] The double softmax operation can be viewed as a query process in two directions. For example... Figure 4 As shown, the relevance matrix is ​​divided into two matching query matrices. The mutual query deviation (MQD) loss uses knowledge transfer across all mutual query distributions of size 2×hw:

[0104]

[0105]

[0106]

[0107]

[0108] in and These are the query distributions for students and teachers extracted at temperature T, respectively. An additional focus loss weight (FL) is added to balance the true / false matching pairs. Standard confidence scores predicted by the student model. Coarse-level total MQD loss. Let be the average of the knowledge distillation (KD) loss for all n distributions, where n equals 2 × hw, h is the height of the feature map, and w is the width of the feature map.

[0109] Regarding the loss of fine-level knowledge transfer Figure 5 This demonstrates granular knowledge transfer. During the teacher-student learning process, both the teacher and student branches can generate heatmaps. In some embodiments, the heatmaps for the teacher model and the student model are respectively... ( )and ( The loss is a Gaussian distribution. Note the loss ( Used to help students learn about distribution:

[0110]

[0111] in and Output the expected values ​​of the distributions for students and teachers, corresponding to the finer-level sets. (in) (matches)

[0112] Therefore, the total loss is the matching set The average of the weighted sums of all levels of attention with respect to the L2 loss. In some embodiments, the attention loss is the L2 distance loss, which pays more attention to high attention weights. Associated predictions. Total variance is often considered a metric for determinism. Teacher predictions with smaller total variance indicate that the teacher is very certain about the corresponding location. In this case, a larger weight can be assigned to the loss to guide the student model to preferentially learn specific predictions from the teacher.

[0113] Both the teacher and student training processes are conducted under direct supervision based on the ground truth. During the student model training, the teacher model receives additional supervision. The final losses for the teacher and student models are as follows:

[0114] ,and

[0115]

[0116] It is worth noting that the teacher-student learning system may also include an electronic device, at least one processor operatively coupled to the electronic device, and at least one memory. The memory is operatively coupled to the at least one processor and is used to store executable instructions. When the teacher-student learning system is executed by the at least one processor, the system performs 3D geometry-guided teacher-student feature matching. For example, a multimodal teacher model determines feature matching between a pair of RGB-D images, each RGB-D image being a combination of an RGB image and its corresponding depth image. The depth image is an image channel, where each pixel is related to the distance between the image plane in the RGB image and the corresponding object. A unimodal student model determines feature matching between the RGB image pair and the teacher model, wherein the teacher model guides the student model to learn RGB-induced depth information for coarse and fine-grained feature matching. A coarse-grained knowledge transfer loss function determines the loss for transferring coarse-grained matching knowledge from the teacher model to the student model. A fine-grained knowledge transfer loss function determines the loss for transferring fine-grained matching knowledge from the teacher model to the student model, and guides the student model to preferentially learn fine-grained prediction distributions.

[0117] Furthermore, each teacher model and student model may also include an electronic device, one or more processors operatively coupled to the electronic device, and one or more memories. The memory is operatively coupled to the one or more processors and is used to store executable instructions. When the teacher-student learning system is executed by the one or more processors, the one or more processors perform feature extraction, local feature transformation, and matching, etc. The teacher model may include a teacher network, and the student model may include a student network.

[0118] Furthermore, the system can communicate wirelessly or wired with other devices such as mobile devices, servers, databases or data centers, and the cloud. The system may also include a display and / or a graphical user interface (GUI) for showing results and for interaction between the user and the system.

[0119] Figure 6 An exemplary block diagram of the physical components (i.e., hardware) of system 600 can be used to illustrate embodiments of practicing the invention. However, Figure 6 The devices and systems described and discussed herein are for illustrative purposes only and do not limit other device configurations for practicing the invention. In basic configuration 608, system 600 includes at least one processing unit 602 and system memory 604. In some embodiments, system memory 604 includes, but is not limited to, volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory), flash memory, or any combination of such memory. System memory 604 may include operating system 605 and one or more program modules 606 adapted to run software application 650. System memory 604 also includes model trainer 620 for training teacher and student models. Furthermore, system 600 may have additional features or functions and may include additional data storage devices 610 (removable and / or non-removable), such as disks, optical discs, or magnetic tapes.

[0120] In some embodiments, program modules and data files may be stored in system memory 604. When program module 606 is executed on processing unit 602, program module 606 (e.g., model trainer 620) performs actions such as Figures 2 to 5 The process is shown.

[0121] In some embodiments, system 600 may further include one or more input devices 612 and output devices 614, such as a display. Furthermore, system 600 may include a communication protocol 616 for wireless and / or wired communication with other devices.

[0122] On the other hand, the present invention provides a non-transitory tangible computer-readable medium for storing instructions. When this non-transitory tangible computer-readable medium is executed by one or more processors, the teacher-student system is capable of performing three-dimensional geometry-guided teacher-student feature matching (3DG-STFM). Computer-executable instructions or program code enable the disclosed apparatus or similar system to perform various operations according to the described method. The storage medium / memory may include, but is not limited to: high-speed random access media / memory, such as DRAM, SRAM, DDRRAM or other random access solid-state storage devices, and non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices.

[0123] These and other aspects of the invention will be further described below. Without limiting the scope of the invention, exemplary instruments, apparatus, methods, and related results are listed below according to embodiments of the invention. It is worth noting that headings or subheadings may be used in the examples for ease of reading, but this does not limit the scope of the invention. Furthermore, certain theories presented and disclosed herein, whether correct or incorrect, should not limit the scope of the invention, provided that the invention can be practiced without considering any particular theory or mode of operation.

[0124] Example

[0125] The following examples / experiments demonstrate that this method outperforms state-of-the-art methods in both indoor and outdoor camera pose estimation and homography estimation. Specifically, given two distinct cameras A and B, the relative camera pose matrix E describes the 3D relative orientation and position of camera B in world coordinates, defined by the reference camera A. Once we assess that there is sufficient correspondence between two images captured by these two cameras, we can regress the camera pose matrix. The more reliable the correspondence in the images, the more accurate the established pose estimation. Therefore, camera pose estimation is a good evaluation method for local feature matching tasks. These experimental data show that the RGBD model outperforms the standard RGB model, and the student RGB model can learn from the teacher model and achieve better performance.

[0126] In an embodiment, a novel method of the present invention is as follows: a camera pose evaluation task is performed on indoor and outdoor datasets, as shown in Table 1. LoFTR[1] is the current state-of-the-art method and is considered as the comparison baseline. The results show the pose error AUC (area under the curve) at thresholds (5°, 10°, 20°). This pose error is defined as the maximum angular error in rotation and translation.

[0127] Table 1: Performing the camera pose evaluation task on indoor and outdoor datasets

[0128]

[0129] Table 2 shows the comparison results between the teacher (RGBD) model, student model, and ordinary RGB model on the indoor / outdoor dataset.

[0130] Table 2: Comparison results of teacher (RGBD) model, student model, and ordinary RGB model on indoor / outdoor datasets.

[0131]

[0132] The results listed in Table 2 demonstrate that both the 3DG-STFM teacher and student models outperform their ordinary unimodal competitors on both indoor and outdoor datasets. Deep modality can eliminate many blurs in the 2D image space and enhance the geometric consistency of feature matching. However, deep modality is not always available during inference because most imaging systems are only equipped with RGB sensors. The results in the table also demonstrate that the student RGB model learns from the teacher model and outperforms the ordinary RGB model.

[0133] The foregoing description of exemplary embodiments of the present invention is for illustrative purposes only and is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations can be made based on the above teachings.

[0134] The embodiments described are chosen to explain the principles of the invention and its practical application, thereby enabling others skilled in the art to make various modifications to the invention and its various embodiments that are as expected and have specific uses. Adaptive alternative embodiments to the invention will be apparent to those skilled in the art without departing from the spirit and scope of the invention. Therefore, the scope of protection of the invention is defined by the appended claims rather than by the foregoing description and the exemplary embodiments described.

[0135] References cited and discussed in the description of this invention may include patents, patent applications, and various publications. Such citations and / or discussions are for the purpose of clearly describing this invention only and do not constitute an admission that any such reference is "prior art." All references cited and discussed in this specification are incorporated herein by reference in their entirety to the same extent as if each reference were cited individually.

[0136] References

[0137] [1]. Sun, Jiaming et al., “LoFTR: Detectorless Local Features Matched to Transformers”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021.

[0138] [2]. Dai, Angela et al., “Scannet: 3D reconstruction of indoor scenes with extensive annotations”, IEEE Conference Proceedings on Computer Vision and Pattern Recognition, 2017.

[0139] [3]. Li, Zhengqi and Noah Snavely, “Megadepth: Learning single-viewpoint depth prediction from Internet photos”, IEEE Conference Proceedings on Computer Vision and Pattern Recognition, 2018.

Claims

1. A system for three-dimensional geometry-guided teacher-student feature matching (3DG-STFM), comprising: A multimodal teacher model is used to determine feature matching between a pair of RGB-D images, wherein each RGB-D image is a combination of an RGB image and its corresponding depth image, the depth image being an image channel, and each pixel in the image channel being related to the distance between the image plane in the RGB image and the corresponding object; A unimodal student model is used to determine feature matches from RGB image pairs and the teacher model, wherein the teacher model guides the student model to learn RGB-induced depth information for coarse and fine feature matching; A coarse-level knowledge transfer loss function is used to determine the loss for transferring coarse-level matching knowledge from the teacher model to the student model; and A fine-grained knowledge transfer loss function is used to determine the loss of transferring fine-grained matching knowledge from the teacher model to the student model, and to guide the student model to prioritize learning the fine-grained prediction distribution.

2. The system as claimed in claim 1, wherein, Each of the teacher models and the student models includes a Feature Pyramid Network (FPN) for extracting coarse local features with 1 / 8 of the original image resolution and fine features with 1 / 2 of the original image resolution from the image pairs.

3. The system as described in claim 2, wherein, The FPN of the teacher model is a four-channel input of the RGB-D image, and the FPN of the student model is a three-channel input of the RGB image, where D represents depth information.

4. The system as described in claim 2, wherein, Each of the teacher model and the student model further includes a coarse-level local feature transformation module, a coarse-level matching module, and a fine-level matching module.

5. The system as described in claim 4, wherein, The coarse-level local feature transformation module is used for: The extracted coarse feature map is flattened into a one-dimensional vector; Positional encoding is performed on the flattened one-dimensional vector to assign a unique embedding vector in sinusoidal format to each feature entry, thereby encoding spatial information; as well as The encoded local feature vectors are processed by a coarse-level matching transformer that includes self-attention and cross-attention layers to generate a logical layer containing a correlation matrix.

6. The system of claim 5, wherein, The self-attention layer and the cross-attention layer are interleaved in the coarse-level matching converter. c Second-rate.

7. The system as claimed in claim 5, wherein, The coarse-level matching module is used for: By adjusting the correlation matrix S in the row and column directions c Applying a double softmax operation to generate the confidence score matrix P c ,in: Let be the matching probability at position (i, j); and Selecting a probability score higher than the threshold The matching pairs are used to generate coarse-level matching predictions.

8. The system of claim 7, wherein, Given the truth matrix derived from the corresponding labels, the cross-entropy loss, which measures the performance of the coarse-level local feature transformation module and the coarse-level matching module, is calculated using the following formula: in The corresponding set generated for the true labels, with FL having a predicted probability p, is the focus loss term used to address the imbalance between matching and non-matching pairs.

9. The system of claim 7, wherein, The fine-level matching module is used for: Project the coarse matching position (i,j) onto the position in the fine-level feature map. At point ), features are extracted from adjacent windows of size w×w as matching candidates; Sample the selected coarse-level features and associate the sampled coarse-level features with the fine-level features; as well as The associated fine-grained features are processed by a fine-grained matching converter to predict sub-pixel-level correspondences.

10. The system of claim 9, wherein, The fine-level matching converter includes a self-attention layer L. f And cross-attention layers to aggregate fine-grained local information and generate heatmap distributions for location refinement.

11. The system of claim 9, wherein, The final loss based on direct supervision is calculated using the following formula: in( () represents the ground truth location for converting an image solution into a fine-grained heatmap legend. For predictions associated with a rough location, The total variance of the heatmap distribution. This is the fine-grained matching set predicted by the module.

12. The system of claim 11, wherein, The total variance of the similarity distribution is randomly assigned weights to each fine-grained match, where the larger total variance is the unknown prediction associated with the lower weights.

13. The system of claim 7, wherein, The coarse-level knowledge transfer loss function is suitable for dividing the logic layer into multiple independent query distributions and guiding the student model to learn these distributions.

14. The system of claim 13, wherein, The mutual query deviation (MQD) loss is determined by the coarse-level knowledge transfer loss function, which utilizes all mutual query distributions for knowledge transfer using the following formula: in and The query distributions for students and teachers were extracted at temperature T, respectively. An additional focus loss weight FL was added to balance the match / non-match truth pairs. The coarse total MQD loss is the standard confidence score predicted by the student model. Let be the average of the knowledge distillation (KD) loss for all n distributions, where n equals 2 × hw, h is the height of the feature map, and w is the width of the feature map.

15. The system of claim 9, wherein, The fine-grained knowledge transfer loss function is applicable to assigning weights to the predictions of the teacher model, with predictions having higher certainty being assigned greater weights.

16. The system of claim 15, wherein, The heatmaps of the teacher model and the student model are respectively... ( )and ( The Gaussian distribution of ).

17. The system of claim 16, wherein, The attention loss used to help the student model learn the Gaussian distribution is determined through the detailed knowledge transfer loss function. ): in, and Output the expected values ​​of the distributions for students and teachers, respectively, and their corresponding sets at the finer levels. (in) (Corresponding match) 18. The system of claim 17, wherein, The attention loss is an L2 distance loss, which focuses more on high attention weights. Related predictions.

19. The system of claim 17, wherein, Teacher predictions with smaller total variance indicate that the teacher model is very specific to the corresponding location, and the loss is given a larger weight to guide the student model to preferentially learn certain predictions from the teacher model.

20. The system of claim 17, wherein, The final losses of the teacher model and the student model are respectively: ;and 。

Images