A descriptor invariance selection method and system based on attention mechanism
Through multi-scale feature extraction and parallel self-attention module based on attention mechanism, the problem of insufficient invariance selection in the prior art is solved, and more efficient invariance selection and positioning accuracy are achieved.
Patent Information
- Application Number
- CN202111537515.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-15
- Publication Date
- 2025-01-17
- Estimated Expiration
- 2041-12-15
AI Technical Summary
The existing local descriptive sub-methods lack effective invariance selection under changes in external factors such as lighting, rotation, and scale, resulting in limited positioning accuracy and lack of capture and perception of global scene information.
Using the descriptor invariance selection method based on attention mechanism, local descriptors with multiple invariances such as rotation invariance and lighting invariance are generated through the multi-scale feature extraction module and the parallel self-attention module, and the local descriptors with multiple invariances are generated, and the triple loss is achieved. Function and metric learning loss calculation, select the appropriate invariance.
It improves the performance and information volume of local descriptors, enables more accurate selection of appropriate invariance, and improves positioning accuracy and robustness.
Smart Images
Figure CN114492562B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and in particular to a method and system for selecting descriptor invariance based on an attention mechanism. Background Art
[0002] Local feature detection and description is a fundamental problem in computer vision, and has received extensive attention in the fields of visual localization, 3D reconstruction, robotics, virtual reality, and augmented reality. Given two images, the classic approach usually uses a two-stage pipeline: first, key points (interest points) are detected from each image, and then the corresponding local descriptors are extracted from the key points for downstream localization tasks.
[0003] In the real world, there are often huge changes in external factors such as illumination, rotation, and scale, which greatly limit the performance of local descriptors and seriously affect the accuracy of downstream tasks. Therefore, the local descriptor methods that have emerged in recent years usually have as many invariances as possible, such as illumination invariance, rotation invariance, scale invariance, etc.
[0004] However, although more invariance will make the local descriptor more robust, it will also make the local descriptor provide less information, that is, in scenes where external factors do not change drastically, it will limit the improvement of positioning accuracy. In order to make a trade-off between robustness and accuracy, some methods for selecting invariance of local descriptors have emerged recently. However, these methods lack the capture and perception of the global information of the scene, so when selecting invariance, they often cannot obtain appropriate invariance. Summary of the invention
[0005] The present invention aims to solve the technical problem that the existing methods fail to effectively select appropriate invariance for local descriptors.
[0006] The technical solution adopted by the present invention is: a descriptor invariance selection method based on attention mechanism, comprising the following steps:
[0007] S1, input the prepared image sequence, and randomly extract image I from it a , generate the invariant image I through perspective transformation i , Transformed Image I v ;
[0008] S2, image I a , unchanged image I i and the transformed image I v Input the multi-scale feature extraction module to generate a training local descriptor, and input the obtained training local descriptor into the parallel self-attention module to obtain a training meta-descriptor;
[0009] S3, using image I a , unchanged image I i and the transformed image I v Construct a triplet loss function, calculate the metric learning loss A between the training local descriptors in S2 and the metric learning loss B between the training meta-descriptors in S3 through the triplet loss function, and add the metric learning losses A and B to get the total loss function;
[0010] S4, repeat S1 to S3 until the total loss function converges and the training network is obtained;
[0011] S5, inputting the image sequence to be predicted into the training network to obtain the local descriptor to be predicted and the meta descriptor to be predicted;
[0012] S6, select two images I from the sequence to be predicted q ,I k , using the meta-descriptors of both parties to select the invariant local descriptors and obtain the weights;
[0013] S7, calculate the L2 distance of the local descriptors to be predicted of the two images in S7, and use the weights in S7 as coefficients to weight the descriptor distance, thereby completing the process of selecting the invariance of the local descriptors.
[0014] Furthermore, in S1, the image size needs to be scaled first, and then image enhancement is performed. Finally, a key point extraction algorithm needs to be used to extract key points from each image.
[0015] Furthermore, the multi-scale feature extraction module includes a feature pyramid module and a four-head module. The multi-scale features are extracted by the feature pyramid module, and then passed through the four-head module to obtain four training local descriptors, each of which corresponds to a combination of invariance.
[0016] Furthermore, in S2, the parallel self-attention module includes four parallel self-attention layers, each of which generates a training meta-descriptor with specific invariance, such as rotation invariance and illumination invariance. The global receptive field of self-attention can fully mine the global information of the image, so that the meta-descriptor can better select invariance for the local descriptor.
[0017] Furthermore, the total loss function L in S5 is calculated as:
[0018]
[0019] in, is the local descriptor metric loss function, Metric loss function for meta-descriptors;
[0020] Said The calculation formula is:
[0021]
[0022] Among them, l is a local descriptor and L is a set of l. is the invariance descriptor loss function, is the loss function of the descriptor that is not invariant. If the local descriptor is I in S1 a with I i All of them are immutable, so use Otherwise use
[0023] Said The calculation formula is:
[0024]
[0025] Where f is the image I in S1 a With image I i is the distinguishing factor, and M is the marginal factor.
[0026] Said The calculation formula is:
[0027]
[0028] in, is the triplet edge loss function commonly used in metric learning, l a For image l a The local descriptor of i For image I i The local descriptor of .
[0029] Said The calculation formula is:
[0030]
[0031] in, is the triplet edge loss function commonly used in metric learning, m a For image I a The meta-descriptor, m i For image I i The meta-descriptor of .
[0032] Furthermore, in S9, it is necessary to find the two images I in S8 q ,I k The L2 distances of the four local descriptors are calculated, and the weights in S8 are used as coefficients to obtain the final descriptor distance of the key points. The specific formula is:
[0033]
[0034] in, For image I q The meta-descriptor of For image Ik 的 Meta descriptor, For image I q The local descriptor of For image I k The local descriptor of .
[0035] A descriptor invariance selection system based on an attention mechanism comprises a multi-scale feature extraction module and a parallel self-attention module; the multi-scale feature extraction module is used to extract multi-scale features of an image and generate four local descriptors with different invariances; the parallel self-attention module is used to extract four meta-descriptors with different invariances, which are used to select appropriate invariance for the local descriptors.
[0036] Preferably, the multi-scale feature extraction module consists of a feature pyramid module and a four-head module; the parallel self-attention module has four parallel branches, each of which is implemented through a self-attention layer.
[0037] Compared with the prior art, the present invention has the following advantages and effects:
[0038] 1. The multi-scale feature extraction module involved in the model of the present invention can extract more useful information from the image and improve the performance of the local descriptor.
[0039] 2. The parallel self-attention module involved in the model of the present invention utilizes the global receptive field of self-attention and fully mines the global information of the image. The obtained meta-descriptor can more accurately select appropriate invariance for the local descriptor and improve the comprehensive performance of the local descriptor. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0041] Figure 1 It is a schematic diagram of the model of the present invention;
[0042] Figure 2 It is a multi-scale feature extraction module;
[0043] Figure 3 It is the self-attention layer structure;
[0044] Figure 4 Schematic diagram of the process of matching a pair of key points;
[0045] Figure 5 The figure shows the comparison between the image matching results of the present invention and those of other methods. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0047] Embodiment 1:
[0048] A descriptor invariance selection system based on an attention mechanism includes a multi-scale feature extraction module and a parallel self-attention module; the multi-scale feature extraction module is used to extract multi-scale features of an image and generate four local descriptors with different invariances; the parallel self-attention module is used to extract four meta-descriptors with different invariances, which are used to select appropriate invariances for local descriptors. The multi-scale feature extraction module is composed of a module similar to a feature pyramid (referred to as an MS module in this embodiment) and a four-head module (referred to as a 4-head module in this embodiment); the parallel self-attention module has four parallel branches, each of which is implemented through a self-attention layer.
[0049] like Figure 1 As shown, a descriptor invariance selection method based on attention mechanism includes the following steps.
[0050] Step 1, data preparation, is divided into two steps:
[0051] 1.1. Prepare a dataset with sufficient scenes. This example uses three datasets as training datasets: Microsoft's COCO, Multi-Illumination Images in the Wild dataset, and VIDIT dataset. These images are randomly enhanced, such as translation, scaling, rotation, and perspective distortion. After that, this example scales the image resolution to 240×320.
[0052] 1.2, using the key point extraction algorithm, extract the key points from each image. This embodiment uses the Superpoint key point extraction algorithm.
[0053] Step 2: Obtaining local descriptors. Input the image in step 1.1 into the multi-scale feature extraction module. Figure 2As shown in the figure, the image is first passed through a module similar to a feature pyramid to extract multi-scale features, and the output size of each feature map is [H / 8, W / 8, 256]. After that, the feature map is passed through a four-head module to obtain four local descriptors, each with an output size of [H / 8, W / 8, 128].
[0054] Step 3: Obtaining meta-descriptors. The four local descriptors obtained in step 2 are sent to the parallel self-attention module to obtain meta-descriptors. The parallel self-attention module has four parallel branches, each of which is implemented through a self-attention layer. The structure of this layer is shown in Figure 3 The core operation is scale dot-product attention, and finally four meta-descriptors are obtained, each with an output size of [H / 8, W / 8, 128].
[0055] Step 4: Calculate the sum of the losses of the local descriptor and the meta-descriptor as a constraint for network training.
[0056]
[0057] Said The calculation formula is:
[0058]
[0059] Among them, l is a local descriptor and L is a set of l. is the invariance descriptor loss function, is the loss function of the descriptor that is not invariant. If the local descriptor is I in S1 a with I i All of them are immutable, so use Otherwise use
[0060] Said The calculation formula is:
[0061]
[0062] Where f is the image I in S1 a With image I i is the distinguishing factor, and M is the marginal factor.
[0063] Said The calculation formula is:
[0064]
[0065] in, is the triplet edge loss function commonly used in metric learning, l a For image Ia The local descriptor, l i For image I i The local descriptor of .
[0066] Said The calculation formula is:
[0067]
[0068] in, is the triplet edge loss function commonly used in metric learning, m a For image I a The meta-descriptor, m i For image I i The meta-descriptor of .
[0069] When the sum of the losses becomes stable, the network training is completed.
[0070] Step 5: Key point matching. The test set used in this embodiment is the RDNIM dataset, which has nothing to do with the training dataset, in order to prove the generalization of the algorithm. First, the key points are extracted from the two images in the test set using the key point extraction algorithm. The specific embodiment uses the Superpoint key point extraction algorithm. For a pair of extracted key points, Figure 4 The process of selecting the invariant local descriptor of the key point is as follows. Finally, the specific formula of the descriptor distance of a pair of key points is:
[0071]
[0072] in, For image I q The meta-descriptor of For image I k The meta-descriptor of For image I q The local descriptor of For image I k The local descriptor of .
[0073] Figure 5 The image matching results of the present invention are compared with other methods. It can be found that in the two tracks on the RDNIM dataset, various evaluation indicators are better than the current mainstream algorithms, which also proves that the invariance selection algorithm of the present invention can greatly improve the comprehensive performance of the local descriptor.
[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A descriptor invariance selection method based on attention mechanism, characterized in that: The following steps are involved: S1, input the prepared image sequence, and randomly extract image I from it a , generate the invariant image I through perspective transformation i , Transformed Image I v ; S2, image I a , unchanged image I i and the transformed image I v Input the multi-scale feature extraction module to generate a training local descriptor, and input the obtained training local descriptor into the parallel self-attention module to obtain a training meta-descriptor; S3, using image I a , unchanged image I i and the transformed image I v Construct a triplet loss function, calculate the metric learning loss A between the training local descriptors in S2 and the metric learning loss B between the training meta-descriptors in S2 through the triplet loss function, and add the metric learning losses A and B to get the total loss function; S4, repeat S1 to S3 until the total loss function converges and the training network is obtained; S5, inputting the image sequence to be predicted into the training network to obtain the local descriptor to be predicted and the meta descriptor to be predicted; S6, select two images I from the sequence to be predicted q ,I k , using the meta-descriptors of both parties to select the invariant local descriptors and obtain the weights; S7, calculating the L2 distance of the local descriptors to be predicted of the two images in S6, and using the weights in S6 as coefficients, weighting to obtain the descriptor distance, completing the invariance selection of the local descriptors; The formula for calculating the descriptor distance is: in, For image I q The meta-descriptor of For image I k The meta-descriptor of For image I q The local descriptor of For image I k The local descriptor of .
2. The descriptor invariance selection method based on the attention mechanism according to claim 1, characterized in that: In S1, the prepared image is first scaled, then image enhancement is performed, and then key points are extracted from the image using a key point extraction algorithm.
3. The descriptor invariance selection method based on the attention mechanism according to claim 1, characterized in that: In S2, the multi-scale feature extraction module includes a feature pyramid module and a four-head module. Multi-scale features are extracted by the feature pyramid module, and then passed through the four-head module to obtain four training local descriptors.
4. The descriptor invariance selection method based on the attention mechanism according to claim 1, characterized in that: In S2, the parallel self-attention module includes four parallel self-attention layers, and each self-attention layer generates a training meta-descriptor with specific invariance.
5. The descriptor invariance selection method based on the attention mechanism according to claim 1, characterized in that: The total loss function L in S3 is calculated as follows: in, To train the local descriptor, we measure the loss function: Measuring loss function for training meta-descriptors; Said The calculation formula is: Among them, l is the training local descriptor, L is a set of l, is the invariance descriptor loss function, is the loss function of the descriptor that does not have invariance. If the local descriptor is trained on I in S1 a with I i If both are immutable, use Otherwise use Said The calculation formula is: Where f is the image I in S1 a With image I i The distinguishing factor, M is the marginal factor; Said The calculation formula is: in, is the triplet edge loss function commonly used in metric learning, l a For image I a The local descriptor of i For image I i The local descriptor of Said The calculation formula is: in, is the triplet edge loss function commonly used in metric learning, m a For image I a The meta-descriptor, m i For image I i The meta-descriptor of .
6. The descriptor invariance selection method based on the attention mechanism according to claim 2, characterized in that: The image enhancement includes modifying brightness, contrast and saturation, and homography transformation.
7. A descriptor invariance selection system based on an attention mechanism, used to implement the descriptor invariance selection method based on an attention mechanism according to any one of claims 1 to 6, characterized in that: It includes a multi-scale feature extraction module and a parallel self-attention module; the multi-scale feature extraction module is used to extract multi-scale features of the image and generate four local descriptors with different invariances; the parallel self-attention module is used to extract four meta-descriptors with different invariances, which are used to select appropriate invariance for the local descriptors.
8. The descriptor invariance selection system based on the attention mechanism according to claim 7, characterized in that: The multi-scale feature extraction module consists of a feature pyramid module and a four-head module; the parallel self-attention module has four parallel branches, each of which is implemented through a self-attention layer.