Feature detection and description method for global context and local information aggregation

Abstract

The invention discloses a global context and local information aggregation feature detection and description method, and the method comprises a feature pyramid module which obtains feature maps of different receptive fields through a plurality of convolution layers, thereby achieving the perception of information of different scales; the self-adaptive global context module is used for segmenting the deep feature map into 16 * 16 blocks, encoding the blocks, inputting the blocks into a Transform layer, learning a global context and then adaptively adding the global context into a descriptor; the local feature enhancement module more effectively captures context information around feature points through a multi-branch convolution structure and a cavity convolution operation, and enhances the expression ability of the network for local information; the feature consistency weighting module is used for introducing feature consistency weighting in optimization and matching of local descriptors and improving the consistency of different-source features; and the multi-scale feature detection module improves the positioning precision of the key points. The global context is introduced for the local features, and the positioning precision of feature point detection and the consistency and specificity of descriptors are improved.

Description

Technical Field

[0001] The present invention relates to the field of feature detection and description, and more specifically, to a feature detection and description method that aggregates global context and local information. Background Art

[0002] Image feature description and matching is a basic technology in computer vision and is widely used in image stitching, SLAM, 3D reconstruction, visual positioning and other fields. This task usually includes three main steps: feature detection, feature description and feature matching. In the feature detection stage, a reliable feature detector is first required to accurately extract key points with stability in the image, especially to ensure the consistency of feature points in the face of different illumination changes or geometric transformations. Next, in the feature description stage, the local image information of the neighborhood of each feature point is encoded and mapped into a high-dimensional feature vector (i.e., feature descriptor) to characterize the uniqueness of the feature point. The design of feature descriptors is crucial. It needs to be robust to common photometric transformations (such as illumination changes, noise, compression, blur, etc.) and geometric transformations (such as scaling, rotation, translation and affine transformation, etc.) to ensure the stability of feature descriptors under different transformation conditions. Finally, in the feature matching step, the matching algorithm relies on these highly discriminative feature descriptors to accurately identify the same feature points in the image.

[0003] As the basis of computer vision applications, domestic and foreign research teams have proposed a large number of feature detectors and feature descriptor methods. Early traditional methods mainly focused on carefully designed methods to identify unique key points using descriptors that are robust to changes such as viewpoints and illumination. Handcrafted techniques such as Harris corners, SIFT, and ORB use explicit geometric concepts such as corners, gradients, and scale-space extrema. With the rise of deep learning and the rapid development of GPU parallel computing, learning-based methods have shown great potential. Methods such as LIFT use CNN to extract more robust and discriminative key point descriptors, making significant progress in handling large viewpoint changes and local feature illumination changes.

[0004] Limited by the locality of convolutional neural networks, most existing local feature description methods can only learn local descriptors with local information, but lack the understanding of the global and surrounding spatial background. The present invention discloses a feature detection and description method that aggregates global context and local information. Based on the local information mined by convolutional neural networks, the global context is introduced into local features by using context enhancement and spatial consistency weighting mechanism, thereby improving the positioning accuracy of feature point detection and the consistency and specificity of descriptors. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a feature detection and description method that aggregates global context and local information to achieve feature point detection and description of an image.

[0006] In order to achieve the above object, the embodiment of the present invention provides the following solution:

[0007] A feature detection and description method for global context and local information aggregation, characterized by comprising the following steps:

[0008] Step 1. Feature extraction

[0009] The feature extraction network used in the present invention is the FPN feature pyramid, which is based on two convolutional layers, two ReLu layers, and one maximum pooling layer. The convolution kernels are all 3×3, the step size is 1, and the padding is 1. According to the size of the feature map extracted by the feature extraction network, the feature extraction network is divided into 4 layers, where the height and width of the feature map output by c1 are the same as the original Figure 1 The height and width of the feature map of c2 is 1 / 2 of the original image and the number of channels is 64, the height and width of the feature map of c3 is 1 / 4 of the original image and the number of channels is 128, and the height and width of the feature map of c4 is 1 / 8 of the original image and the number of channels is 128.

[0010] Step 2. Get the global context

[0011] First, the c4 feature map extracted in step 1 is adjusted to the size of (64,64) through an adaptive average pooling layer, and then the feature map is divided into 16 blocks with a 16×16 window. The pixel features of each block are directly superimposed on one dimension, recorded as X p . Use a learnable linear projection to transform X p Mapped to a potential 128-dimensional space. In order to make the local descriptor obtain the spatial position information relative to the global one, a specific position encoding is learned and added to X p In order to preserve the position information. After reshaping by n Transformer layers, a region-level descriptor with global context is obtained. In the method of the present invention, n=12.

[0012] Step 3. Enhance local information

[0013] The local feature enhancement module is used to fully collect rich local information. The local feature enhancement module uses a multi-branch convolution structure to extract a variety of discriminative semantic information, and uses dilated convolution to obtain richer local contextual information. The local information enhancement module has four branches, each of which first performs a 1×1 convolution operation on the input feature map to preliminarily adjust the number of channels for subsequent processing. Next, the first branch is a residual structure that does not perform additional operations to form an equivalent graph to retain the surrounding key feature information. The other three branches perform cascaded standard convolution operations with kernel sizes of 1×3, 3×1, and 3×3, respectively. Additional dilated convolutions are added to the two middle branches so that the extracted feature maps can retain more contextual information.

[0014] Step 4. Global and local feature fusion

[0015] A method based on a gating mechanism is used to adaptively adjust the degree of introduction of global context information, and intelligently adjust it according to the actual needs of each local area, so as to achieve the effect of optimizing feature expression. First, the c4 feature map extracted in step 1 is adjusted to a size of (64, 64) through an average pooling operation through an adaptive average pooling layer. Then it is input into a 3×3 convolution layer, the output dimension is adjusted to 1, and then a ReLU activation function is passed, and its output is used as the weight map of feature fusion. The global context obtained in step 2 and the weight map are upsampled to (1 / 4H, 1 / 4W) using bilinear interpolation, and the matrix elements are multiplied to obtain the adaptively adjusted context information. Finally, it is added to the local enhanced features obtained in step 3, and the output is the description map after the adaptive fusion of global and local features.

[0016] Step 5. Multi-scale feature detection

[0017] The method of the present invention predicts key points in different scale spaces by using feature pyramids for local feature detection. Specifically, five detection heads are set to predict key point heat maps on feature maps of different sizes and the global feature map extracted in step 2. In order to merge the prediction results, the predicted heat maps of different scales are upsampled to the image size h×w using bilinear interpolation, and then the weighted summation is performed using learnable parameters to output the final prediction result heat map.

[0018] Step 6. Feature consistency weighting to obtain descriptors:

[0019] In order to further overcome the limitation that key point description is based only on local information, the method of the present invention obtains cognition and insight into spatial information through a feature consistency weighting mechanism. First, the feature maps of different scales obtained from the feature pyramid are aligned by up and down sampling, spliced ​​together, and then averaged in the channel dimension. Finally, the weighted score map W is predicted from this average feature map through 3×3 convolution and SoftPlus activation function.

[0020] After obtaining the weighted score map W, feature description map D, and key point probability map H, first obtain the coordinates of the points with scores greater than the detection threshold from the key point probability map H and normalize them to [-1, 1] according to the size of the image. Then sample from the feature description map D according to the normalized coordinates, and multiply them with the weights corresponding to the weighted score map W after L2 normalization to obtain the final descriptor x.

[0021] Preferably, the number of images per batch is set to 12 during algorithm training.

[0022] Preferably, the algorithm iterates for a total of 40 epochs.

[0023] Preferably, the number of Transformer layers in the algorithm is set to 12.

[0024] Preferably, the initial value of the learning rate is 0.001, the learning rate remains unchanged in the first five epochs, and linearly decays to 0 starting from the sixth epoch.

[0025] The present invention discloses a feature detection and description method for global context and local information aggregation. On the basis of local information mined by convolutional neural network, context enhancement and spatial consistency weighting mechanism are used to obtain non-local awareness. First, through the feature pyramid module, multiple convolutional layers are used to obtain feature maps of different receptive fields, so as to realize the perception of information of different scales. Then, through the local feature enhancement module, through the multi-branch convolution structure and the hole convolution operation, the context information around the feature point is more effectively captured, and the network's expression ability for local information is enhanced. Then, through the adaptive global context module, the deep feature map is cut into 16×16 blocks, encoded and input into the Transformer layer, the global context is learned, and then adaptively added to the descriptor. Finally, through the feature consistency weighting module, spatial consistency weighting is introduced in the optimization and matching of local descriptors to improve the consistency of heterogeneous features. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A schematic diagram of a feature detection and description method for global context and local information aggregation provided by an embodiment of the present invention;

[0028] Figure 2A flowchart of a feature detection and description method for global context and local information aggregation provided by an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of a global context module provided by an embodiment of the present invention;

[0030] Figure 4 A schematic diagram of a local information enhancement module provided in an embodiment of the present invention;

[0031] Figure 5 A schematic diagram of a multi-scale feature detection module provided by an embodiment of the present invention;

[0032] Figure 6 A training flow chart of a feature detection and description method for global context and local information aggregation provided by an embodiment of the present invention;

[0033] Figure 7 This is a quantitative comparison chart of various methods on the HPatches dataset under different thresholds provided by the embodiments of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Limited by the locality of convolutional neural networks, most existing local feature description methods can only learn local descriptors with local information, but lack the understanding of the global and surrounding spatial background. Figure 1 As shown in the figure, by using context enhancement and spatial consistency weighting mechanism, the feature descriptor gains global awareness. First, through the feature pyramid module, multiple convolutional layers are used to obtain feature maps with different receptive fields, thereby realizing the perception of information at different scales. Then, through the local feature enhancement module, the multi-branch convolution structure and the hole convolution operation are used to more effectively capture the context information around the feature points and enhance the network's ability to express local information. Then, through the adaptive global context module, the deep feature map is divided into 16×16 blocks, encoded and input into the Transformer layer, the global context is learned, and then adaptively added to the descriptor. Finally, through the feature consistency weighting module, spatial consistency weighting is introduced in the optimization and matching of local descriptors to improve the consistency of heterogeneous features.

[0036] The embodiment of the present invention discloses a feature detection and description method that aggregates global context and local information to achieve feature point detection and description of an image. Figure 2 , the above method includes at least the following 6 steps.

[0037] Step 1. Feature extraction

[0038] The feature extraction network used in the present invention is the FPN feature pyramid, which is based on two convolutional layers, two ReLu layers, and one maximum pooling layer. The convolution kernels are all 3×3, the step size is 1, and the padding is 1. According to the size of the feature map extracted by the feature extraction network, the feature extraction network is divided into 4 layers, where the height and width of the feature map output by c1 are the same as the original Figure 1 The height and width of the feature map of c2 is 1 / 2 of the original image and the number of channels is 64, the height and width of the feature map of c3 is 1 / 4 of the original image and the number of channels is 128, and the height and width of the feature map of c4 is 1 / 8 of the original image and the number of channels is 128.

[0039] Step 2. Get the global context

[0040] CNNs have limitations in capturing long-range dependencies or global context in images due to their local receptive fields and hierarchical structures. In contrast, the Transformer is able to consider the relationship between all positions in the entire input sequence at once through the self-attention mechanism, which makes it particularly suitable for tasks that require understanding global information. Therefore, to introduce global context to feature descriptors, using the Transformer is a natural choice. However, the computational complexity of the self-attention mechanism is O(n 2 ), where n is the length of the input sequence. This means that as the length of the sequence grows, the amount of computation increases rapidly. And since the self-attention layer needs to store the attention weight matrix between all positions, this results in huge video memory consumption. So Figure 3 As shown, the method of the present invention first adjusts the c4 feature map extracted in step 1 to a size of (64,64) through an average pooling operation through an adaptive average pooling layer, and then divides the feature map into 16 blocks with a 16×16 window. The pixel features of each block are directly superimposed on one dimension, which is recorded as X p . Use a learnable linear projection to transform X p Mapped to a potential 128-dimensional space. In order to make the local descriptor obtain the spatial position information relative to the global one, a specific position encoding is learned and added to X p To preserve the location information, as follows:

[0041]

[0042] Where U is the linear coding projection, U posis the position code. After Z0 passes through the Transformer layer, it gradually becomes a hidden feature with global background. The structure of the Transformer layer is as follows Figure 3 As shown. Among them, MSA represents multi-head self-attention and MLP represents multi-layer perceptron. After reshaping by n Transformer layers, a regional level descriptor with global context is obtained. In the method of the present invention, n=12.

[0043] Step 3. Enhance local information

[0044] In the feature description task, local information is the first priority, and the most obvious feature of a feature point comes from its nearby pixels. However, the extraction ability of ordinary convolution is limited. The features extracted at this stage contain less semantic information and a narrow receptive field, making it difficult to fully utilize the rich local information. Therefore, the proposed local information enhancement module considers enhancing pixel features from two aspects. From the perspective of increasing feature richness, a multi-branch convolution structure is used to extract a variety of discriminative semantic information. From the perspective of expanding the receptive field, a dilated convolution is used to obtain richer local context information. The overall structure of the local information enhancement module is shown in the figure. Figure 4 As shown. The local information enhancement module has four branches, each of which first performs a 1×1 convolution operation on the input feature map to preliminarily adjust the number of channels for subsequent processing. Next, the first branch is a residual structure that does not perform additional operations to form an equivalent graph to retain the surrounding key feature information. The other three branches perform cascaded standard convolution operations with kernel sizes of 1×3, 3×1, and 3×3, respectively. The two middle branches add additional hole convolutions so that the extracted feature maps can retain more contextual information. The mathematical expression of the local information enhancement module is as follows:

[0045]

[0046] in, and They represent standard convolution operations with kernel sizes of 1×1, 1×3, 3×1, and 3×3, respectively. represents the element-wise addition operation on the feature map. F is the input feature map. F1, F2 and F3 represent the output feature maps of the first three branches after standard convolution and dilated convolution. Y is the output feature map of the local information enhancement module.

[0047] The structure of the local information enhancement module is very lightweight. Through multi-branch and hole convolution, the model can learn richer local context features, thereby improving the ability to represent local information around feature points.

[0048] Step 4. Global and local feature fusion

[0049] In feature description, different regions have different requirements for global context information. For those areas where local features are not enough to provide sufficient descriptive power, such as areas with sparse textures or repeated patterns, the introduction of global context can significantly enhance its representation ability, increase the richness of spatial information, and thus improve the discrimination. However, for local feature areas that already have good distinguishability, blindly adding global context may introduce unnecessary noise and weaken the effectiveness of the original features. Therefore, the present invention adopts a method based on a gating mechanism, which can adaptively adjust the degree of introduction of global context information, and intelligently adjust it according to the actual needs of each local area, so as to achieve the effect of optimizing feature expression. This method not only improves the feature extraction accuracy in complex scenes, but also ensures the overall robustness and generalization ability of the model.

[0050] Similar to step 2, first adjust the c4 feature map extracted in step 1 to a size of (64, 64) through an average pooling operation through an adaptive average pooling layer. Then input it into a 3×3 convolution layer, adjust the output dimension to 1, and then pass it through a ReLU activation function, and its output is used as the weight map for feature fusion. The global context obtained in step 2 and the weight map are upsampled to (1 / 4H, 1 / 4W) using bilinear interpolation, and the matrix elements are multiplied to obtain the adaptively adjusted context information. Finally, it is added to the local enhanced features obtained in step 3, and the output is the description map after the adaptive fusion of global and local features. The mathematical expression of the whole process is as follows:

[0051]

[0052] Among them, f i represents bilinear interpolation, Represents a standard convolution operation with a kernel size of 3×3. Indicates the element-wise addition operation on the feature map. D represents the final feature description map, Y is the local enhanced feature map, and Z is the global feature map reshaped by Transformer. It means to perform element addition operation on the feature map. Represents element-wise multiplication of feature maps.

[0053] Step 5. Multi-scale feature detection

[0054] The method of the present invention predicts key points in different scale spaces by using feature pyramids for local feature detection. Specifically, five detection heads are set to predict key point heat maps on feature maps of different sizes and the global feature map extracted in step 2, such as Figure 5As shown in Figure 2, in order to merge the prediction results, the prediction heat maps of different scales are upsampled to the image size h×w using bilinear interpolation, and then the weighted summation is performed using learnable parameters to output the final prediction result heat map.

[0055] Step 6. Feature consistency weighting to obtain descriptors:

[0056] In order to further overcome the limitation that key point description is based only on local information, the method of the present invention obtains cognition and insight into spatial information through a feature consistency weighting mechanism. First, the feature maps of different scales obtained from the feature pyramid are aligned by up and down sampling, spliced ​​together, and then averaged in the channel dimension. Finally, the weighted score map W is predicted from this average feature map through 3×3 convolution and SoftPlus activation function.

[0057] After obtaining the weighted score map W, feature description map D, and key point probability map H, first obtain the coordinates of the points with scores greater than the detection threshold from the key point probability map H and normalize them to [-1, 1] according to the size of the image. Then sample from the feature description map D according to the normalized coordinates, and multiply them with the weights corresponding to the weighted score map W after L2 normalization to obtain the final descriptor x.

[0058] During the model training phase, Figure 6 As shown. The model is trained on a subset of MegaDepth, with a batch size of 12 and a combination of two images as input. First, follow steps 1 to 6 in order to obtain the descriptors x, x' and key point probability maps H, H' of the two images respectively. Then use the loss function to calculate the loss function value of this iteration, and use the Adam algorithm to update the network parameters. When the maximum iteration cycle is reached or the iteration converges, stop the iteration and save the network model.

[0059] The loss function of this method is composed of the loss function L of the feature descriptor d And the loss function L for feature detection p It consists of two parts. The loss function L of the feature descriptor d It is used to supervise that the descriptors corresponding to the two images remain close, using the triplet loss (Triplet Loss) commonly used in contrastive learning. The specific expression is as follows:

[0060]

[0061] The loss function L for feature detection p Using weighted binary cross entropy loss, the specific expression is as follows:

[0062] L p =-λglog(k)-(1-g)log(1-k) (7)

[0063] The final loss function L = L d +L p .

[0064] Preferably, when the algorithm is trained, the number of images in each batch is set to 12, a total of 40 epochs are iterated, the number of Transformer layers in the adaptive global context module is set to 12, the initial value of the learning rate is 0.001, the learning rate remains unchanged for the first 5 epochs, and linearly decays to 0 from the 6th epoch.

[0065] The present invention uses the average matching accuracy MMA as a quantitative evaluation indicator. MMA is the average percentage of correct matches in image pairs under different matching error thresholds. The higher the MMA, the better the effect of the feature descriptor. Table 1 shows the MMA values ​​of the method disclosed in the present invention and D2-Net, SuperPoint, R2D2, PoSFeat, DISK, ASLFeat and MTLDesc on the HPatches dataset. Figure 7 The MMA performance of each method is reported at thresholds from 0 to 10. As can be seen from Table 1, the MMA@1px value of the method disclosed in the present invention is 44.33%, MMA@2px is 71.00%, and MMA@3px is 82.04%, which is better than other models in the table.

[0066] Table 1 Quantitative results of the method disclosed in the present invention and other methods

[0067]

[0068] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined in the present invention may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in the present invention, but will conform to the widest scope consistent with the principles and novel features disclosed in the present invention.

Claims

1. A feature detection and description method that aggregates global context and local information, characterized in that: It includes the following 6 steps: Step 1. Feature extraction; The feature extraction network is an FPN feature pyramid, which is based on two convolutional layers, two ReLu layers, and one maximum pooling layer. The convolution kernels are all 3×3, the step size is 1, and the padding is 1. According to the size of the feature map extracted by the feature extraction network, the feature extraction network is divided into 4 layers, where the height and width of the feature map output by c1 are consistent with the original image and the number of channels is 64, the height and width of the feature map of c2 is 1 / 2 of the original image and the number of channels is 64, the height and width of the feature map of c3 is 1 / 4 of the original image and the number of channels is 128, and the height and width of the feature map of c4 is 1 / 8 of the original image and the number of channels is 128. Step 2. Get the global context; First, the c4 feature map extracted in step 1 is adjusted to the size of (64, 64) through an adaptive average pooling layer, and then the feature map is divided into 16 blocks with a 16×16 window. The pixel features of each block are directly superimposed on one dimension, recorded as X p ; Use a learnable linear projection to transform X p Mapped to a potential 128-dimensional space; in order to make the local descriptor obtain spatial position information relative to the global, a specific position encoding is learned and added to X p In order to preserve the position information; after reshaping through n Transformer layers, a region-level descriptor with global context is obtained; Step 3. Enhance local information; The local feature enhancement module is used to fully collect rich local information; the local feature enhancement module uses a multi-branch convolution structure to extract a variety of discriminative semantic information, and uses a hollow convolution to obtain richer local context information; the local information enhancement module has four branches, each of which first performs a 1×1 convolution operation on the input feature map to preliminarily adjust the number of channels for subsequent processing; next, the first branch is a residual structure, without additional operations, to form an equivalent graph to retain the surrounding key feature information; The other three branches perform cascaded standard convolution operations with kernel sizes of 1×3, 3×1, and 3×3 respectively; the middle two branches add additional dilated convolutions so that the extracted feature maps can retain more contextual information; Step 4. Global and local feature fusion; The gating mechanism method is used to adaptively adjust the degree of introduction of global context information, and intelligently adjust it according to the actual needs of each local area, so as to achieve the effect of optimizing feature expression; first, the c4 feature map extracted in step 1 is adjusted to the size of (64, 64) through an adaptive average pooling layer through an average pooling operation; it is input into a 3×3 convolution layer, the output dimension is adjusted to 1, and then a ReLU activation function is passed, and its output is used as the weight map of feature fusion; the global context obtained in step 2 and the weight map are upsampled to (1 / 4H, 1 / 4W) using bilinear interpolation, and the matrix elements are multiplied to obtain the adaptively adjusted context information; finally, it is added to the local enhanced feature obtained in step 3, and the output is the description map after the adaptive fusion of global and local features; Step 5. Multi-scale feature detection; The key points of different scale spaces are predicted by using feature pyramid for local feature detection. Specifically, five detection heads are set to predict key point heat maps on feature maps of different sizes and the global feature map extracted in step 2. In order to merge the prediction results, the predicted heat maps of different scales are upsampled to the image size h×w using bilinear interpolation, and then the final prediction result heat map is output by weighted summation using learnable parameters. Step 6. Feature consistency weighting to obtain descriptors: The recognition and insight of spatial information is obtained through the feature consistency weighting mechanism; first, the feature maps of different scales obtained from the feature pyramid are aligned in size using up and down sampling, spliced ​​together, and then averaged in the channel dimension; finally, the weighted score map W is predicted from this average feature map through 3×3 convolution and SoftPlus activation function; after obtaining the weighted score map W, feature description map D and key point probability map H, first obtain the coordinates of the points with scores greater than the detection threshold from the key point probability map H and normalize them to [-1,1] according to the size of the image; then sample from the feature description map D according to the normalized coordinates, and multiply them with the weights corresponding to the weighted score map W after L2 normalization to obtain the final descriptor x.

2. The feature detection and description method of global context and local information aggregation as claimed in claim 1, characterized in that: The number of images per batch is set to 12 during algorithm training.

3. The feature detection and description method of global context and local information aggregation as claimed in claim 1, characterized in that: The algorithm iterates for 40 epochs in total.

4. The feature detection and description method of global context and local information aggregation as claimed in claim 1, characterized in that: The number of Transformer layers in the algorithm is set to 12.

5. The feature detection and description method of global context and local information aggregation as claimed in claim 1, characterized in that: The initial value of the learning rate is 0.

001. The learning rate remains unchanged in the first five epochs and linearly decays to 0 starting from the sixth epoch.

Images