A feature point detection and description method based on global features and guidance information

Abstract

The present invention discloses a feature point detection and description method based on global features and guidance information. This method improves and optimizes the Transformer framework, performs detection and description on global features, and avoids the performance limitations brought by local receptive fields. At the same time, by introducing feature point guidance information, feature point detection and description become more intelligent and robust. On the one hand, the position of feature points is estimated in advance to provide the network with prior position information, thereby improving training efficiency. On the other hand, the guidance information is beneficial to the network's robustness to factors such as scale and rotation. The present invention combines the Transformer framework with the feature point detection task, and the guidance information generated by mature traditional methods can avoid most of the false detections. At the same time, it reduces the amount of redundant calculations in some details, which is of great help to the application of feature point detection and description algorithms in practical engineering.

Description

Technical Field

[0001] The present invention belongs to the fields of image processing and computer vision, and in particular relates to a feature point detection and description method based on global features and guidance information. Background Art

[0002] Feature point detection and description is a key research issue in computer vision, fundamental to visual tasks such as 3D reconstruction, structure from motion estimation, and simultaneous localization and mapping. Feature point detection and description involves detecting sparse feature points in the same scene across different images under different viewing angles and imaging conditions, and describing them as vectors. Key challenges lie in ensuring detection stability and description robustness.

[0003] Feature point detection and description methods have evolved from traditional methods to deep learning approaches. In the early days of computer vision, researchers, drawing on their own experience and domain knowledge, proposed numerous methods, such as the classic SIFT algorithm, to extract stable feature points and generate robust descriptors. While these methods possessed excellent characteristics, such as scale invariance, rotation invariance, and robustness to noise and illumination, they were limited by researchers' prior knowledge and the receptive field of detection, resulting in limited detection accuracy and descriptor performance. In recent years, thanks to the powerful feature extraction capabilities of convolutional neural networks (CNNs), CNN-based feature point detection and description algorithms have rapidly developed. In the paper "LIFT: Learned Invariant Feature Transform [C], Yi KM, Trulls E, Lepetit V, et al., European conference on computer vision. 2016: 467-483," Yi et al. proposed a trainable version of SIFT, the LIFT algorithm, which utilizes a CNN for feature point detection, orientation estimation, and computation of feature point descriptors. To learn these three subtasks within a single model framework, a Spatial Transformer Network (STN) is introduced to modify image patches to obtain feature point detection and orientation estimation results. A soft argmax strategy is used instead of the traditional non-maximum suppression strategy. The LIFT algorithm uses the detection results of the SIFT algorithm as the ground truth of feature point locations for supervised training, achieving superior performance compared to previous algorithms. However, because LIFT's ground truth depends on the SIFT detection results, its performance is significantly limited by the supervisory signal, preventing the network from learning feature points that are not present in the supervisory signal. To avoid relying on supervisory signals, DeTone et al. ("Superpoint: Self-supervised interest point detection and description [C], DeTone, D., Malisiewicz, T., & Rabinovich, A., In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2018: 224-236") proposed a fully convolutional network, SuperPoint, which jointly computes pixel-level feature point locations and associated descriptors.In SuperPoint, a synthetic dataset was constructed for pseudo-truth generation and pre-training. The introduced homography adaptation module enables self-supervised training, allowing the network to learn to distinguish good and bad feature points from the data. However, in areas with repetitive patterns (such as windows on tall buildings) and under extreme lighting conditions, the descriptor performs poorly due to the inherent contradiction between the increase in receptive field and the decrease in resolution caused by convolutional network downsampling. To improve the robustness of the descriptor, Revaud et al. proposed the R2D2 algorithm in the reference "R2d2: Reliable and repeatable detector and descriptor [J], Revaud, Jerome, et al., Advances in neural information processing systems. 2019: 12405-12415." Although the R2D2 algorithm still uses a fully convolutional network, it eliminates downsampling layers to maintain feature map resolution, preventing the map from becoming coarse. It also uses dilated convolutions instead of strided convolutions to increase the receptive field of the feature map. In addition to the feature point location branch and the descriptor vector branch, a feature point reliability branch is added to improve the robustness of feature points in areas with repeated patterns in the image. However, while the feature map resolution is maintained, the receptive field gain achieved by dilated convolutions is small, limiting R2D2's performance improvement. In the reference "DenserNet: Weakly supervised visual localization using multi-scale feature aggregation[C],Liu D, Cui Y, Yan L, et al.,Proceedings of the AAAI Conference on Artificial Intelligence.2021:6101-6109", Liu et al. proposed the DenserNet algorithm, which, based on the existing fully convolutional network, aggregates feature maps at different semantic levels so that the feature maps contain information of different resolutions, thereby improving the performance of the feature maps; DenserNet performs end-to-end training without pixel-level annotation, and uses weakly supervised triple ranking loss to learn discriminative features, thereby enhancing detection accuracy and descriptor performance.

[0004] Existing algorithms use fully convolutional networks for feature point detection and description, but CNN-based feature calculations face the contradiction between feature map resolution and receptive field, which limits the performance of feature point detection and description. This impact is difficult to avoid by expanding the data set or iterative training. At the same time, existing algorithms rely solely on image pixel information and network structure for detection during training and testing, without using any prior information, which may limit the speed and accuracy of detection. In response to the shortcomings of existing methods, the present invention designs a feature point detection and description method based on global features and guidance information. On the one hand, global features can provide a global receptive field and full resolution for the detection and description process, thereby improving the performance of detection and description; on the other hand, guidance information can help the algorithm intelligently and quickly pre-estimate the positions of possible feature points without preventing the network from searching for other feature points. Summary of the Invention

[0005] The present invention belongs to the fields of image processing and computer vision, and involves a series of methods such as the detection and description of feature points, the generation of guidance information for feature points, and global feature extraction. By improving and optimizing the Transformer framework, detection and description are performed on global features, avoiding the performance limitations brought by the local receptive field. By introducing feature point guidance information for feature-assisted decoding, on the one hand, the position of feature points is estimated in advance, providing the network with prior position information, improving training efficiency and detection accuracy; on the other hand, the guidance information helps the network improve its robustness to factors such as scale and rotation, which is of great help to the application of the method proposed by the present invention in practical engineering projects such as homography estimation and three-dimensional reconstruction.

[0006] The present invention proposes a feature point detection and description method based on global features and guidance information, which includes the following steps:

[0007] 1. Construct a training dataset: Collect a series of image data to construct a training dataset D. For any image I∈D in training dataset D, the size is W×H×C, where W, H, and C represent the width, height, and number of channels of training image I, respectively. Input I into the network N constructed in step 2.

[0008] 2. Build an end-to-end feature point detection and description network N: The network N consists of an encoder network N E , guide information generation network N G and decoder network N D Among them, the encoder network N E By block layer N EB , linear embedding layer N EE , position encoding layer N EP and feature encoding layer N EF Cascade composition; guide information generation network N G By the detection layer NGD , block layer N GB and a linear embedding layer N GE Cascade composition; decoder network N D By feature decoding layer N DF , feature point position decoding layer N DP and descriptor decoding layer N DD composition.

[0009] 2-1. Design encoder network N E . N E By block layer N EB , linear embedding layer N EE , position encoding layer N EP and feature encoding layer N EE Composition. Block layer N EB The input image I is divided into M non-overlapping blocks of size S×S×C, where S and C represent the spatial size and number of channels of the block respectively. The relationship between the number of blocks and the block size is: Each block is flattened into a 1-dimensional column vector of size (S·S·C)×1, and the block vector sequence {b (1) , b (2) ,...,b (M)},in Indicates the i b block vector, i b ∈{1, 2, ..., M}.

[0010] 2-2. The block vector sequence {b (1) , b (2) ,...,b (M) Input linear embedding layer N EE , get the embedding vector sequence {e (1) , e (2) ,...,e (M)},in Indicates the i e embedding vector, i e ∈{1, 2, ..., M}. Among them, the linear embedding layer N EE It consists of n layers of multi-layer perceptron cascade, which transforms the block vector Mapped to embedding vector by formula (1) The size is N e ×1, where i be ∈{1, 2, ..., M}, N e Indicates the number of elements in the embedding vector.

[0011]

[0012] Among them, N EE(·) represents the operation performed by the linear embedding layer, W EE and b represent the weights and bias of the linear embedding layer.

[0013] 2-3. Embed the vector sequence {e (1) , e (2) ,...,e (M)}Input position encoding layer N EP , output position embedding vector sequence Among them, the position encoding layer N EP Embed the vector With the i ep Learnable position parameters of block regions Add together to get the position embedding vector and The size is N e ×1, where i ep ∈{1, 2, ..., M}.

[0014]

[0015] Among them, N EP (·) represents the operation performed by the position encoding layer.

[0016] 2-4. Embed positions into vector sequences Input feature encoding layer N EF , output encoding feature sequence in The size is N f ×1, N f The number of elements representing the encoded features. Feature encoding layer N EF It consists of L transformer basic blocks repeatedly connected in cascade. Each transformer basic block consists of a multi-headed self-attention layer (MSA), a multi-layer perceptron (MLP) and two layer norms (LN). Let the input sequence of the j-th transformer basic block be Where j∈{1, 2, ..., L}. Let the vector sequence With the matrix One-to-one correspondence, where the matrix Z j By vector sequence Concatenate by column to get the vector sequence The matrix Z j The j+1th transformer basic block converts the sequence Mapping encoding to sequence

[0017]

[0018] Among them, LN(·) represents the layer normalization operation, MSA(·) represents the multi-head attention operation, and Z j and Z j+1 denote the matrix corresponding to the input sequence and the matrix corresponding to the output sequence of the j-th transformer basic block, j∈{1, 2, ..., L}. The calculation of the MSA(·) function is as follows:

[0019]

[0020] Among them, X represents the input matrix of MSA(·), head k represents the output of the kth head of the multi-head self-attention layer, Concat represents the concatenation operation in the row dimension, h represents the number of heads in the multi-head self-attention layer, and W O represents the output mapping matrix of the multi-head self-attention layer, Denote the query mapping matrix, key mapping matrix, and value mapping matrix of the kth head, respectively, k∈{1, 2, ..., h}, d X Represents the number of rows of the input matrix X. The calculation of the MLP(·) function is as follows:

[0021] Y=MLP(X′)=W M ·X′+b M (5)

[0022] Where X′ represents the input matrix of the MLP(·) function, W M and b M are the weights and biases of the MLP.

[0023] Will Recorded as The encoded feature sequence is the final output of the feature encoding layer.

[0024] 2-5. Design of guidance information generation network N G . N G By the detection layer N GD , block layer N GB and a linear embedding layer N GE Cascade composition. Detection layer N GD Use any other feature point detection algorithm A to detect feature points on the input image I and obtain the mask matrix M mask , size is W×H×C, where the matrix M mask The values of the C channels at the non-feature point positions are all 0, and the values of the C channels at the feature point positions are all 1.

[0025] 2-6. Block layer NGB Mask matrix M mask Divide the blocks into blocks, each block size is S×S×C, and the number of blocks is M; flatten each block into a 1-dimensional column vector to obtain the mask vector sequence {m (1) , m (2) ,...,m (M)}, where the i m mask vectors The size of is (S·S·C)×1, i m ∈{1, 2, ..., M}.

[0026] 2-7. The mask vector sequence {m (1) , m (2) ,...,m (M) Input linear embedding layer N GE , get the boot information sequence Linear embedding layer N GE It is composed of m layers of multi-layer perceptron cascade, the i-th ge The multilayer perceptron transforms the mask vector Mapping to boot information The size is N f ×1, where i ge ∈{1, 2, ..., M}. The information sequence is guided The coding feature sequence obtained in steps 2-4 Merge by splicing to get the decoder network N D The input sequence

[0027] 2-8. Design decoder network N D . N D By feature decoding layer N DF , feature point position decoding layer N DP and descriptor decoding layer N DD Composition. Feature decoding layer N DF It consists of T transformer basic blocks cascaded as described in steps 2-4, with the input feature sequence Output decoding feature sequence in and By N DF Then the corresponding output and will sequence with sequence The corresponding addition results in the characteristic sequence {t (1) , t (2) ,...,t (M)},Right now in Represents the i-th feature sequence a feature vectors, i a ∈{1, 2, ..., M}.

[0028] 2-9. The feature sequence {t (1) , t (2) ,...,t (M)} Input feature point position decoding layer N DP In the output position response map I P , size is W×H. Feature point position decoding layer N DP It is composed of u layers of multi-layer perceptron cascade, N DP First, the block feature vector t (i) Mapped to position vector p (i) , p (i) The size is (S·S)×1, where i∈{1, 2, ..., M}; then the position response vector p (i) Arrange rows into blocks of size S×S Get block position response sequence Finally, the sequence Combine the blocks in the order of step 2-1 (step 2-1 divides the image into blocks and forms a sequence, and here the reverse transformation is performed to reverse transform the sequence into a position response map in the shape of the image), and obtain the output position response map I P .

[0029] 2-10. The feature sequence {t (1) , t (2) ,...,t (M)} Input descriptor decoding layer N DD In the output descriptor response diagram I D , size is W×H×N F , where N F Descriptor decoding layer N DD It consists of v layers of multi-layer perceptron cascade, N DD First, the block feature vectors t (i) Mapped to block descriptor vector d (i) , d (i) The size of (S·S·N F )×1, where i∈{1, 2, ..., M}; then d (i) Rearrange to size S×S×N F Block Traverse i=1, 2, ..., M to get the block descriptor vector sequence Finally, the sequence Combine the blocks in the order of step 2-1 (how to block in step 2-1 is how to transform back here) to get the output descriptor vector image I D .

[0030] 3. Add random noise δ to image I, perform random translation, rotation, and color change, etc. to obtain augmented image I * , the size is W×H×C, where W, H, and C represent the augmented image I * The width, height, and number of channels of . Let the homography matrix ( represents the field of real numbers, represents all 3×3 real matrices) for the original image I and the augmented image I * The pixel position correspondence of the overlapping area between the augmented images I * Input the network N constructed in step 2 to obtain the position response map and descriptor vector graph The sizes are W×H and W×H×N respectively F .

[0031] 4. Construct the loss function L(I, I * )(L(I,I * ) represents the combination of I and I * is a function of the input). Loss function L(I, I * ) is lost by the feature point position L P (I,I * ) and descriptor loss L D (I,I * )composition:

[0032] L(I,I * )=L P (I,I * )+βL D (I,I * ) (6)

[0033] Among them, the hyperparameter β is used to balance the ratio of feature point position loss and descriptor loss.

[0034] 4-1. Constructing feature point position loss L P (I,I * ). Feature point position loss uses local cosine loss L cos (I,I * )(see formula 7) and peak loss L peaky (I,I * )(see formula 8) is described, L cos (I,I * ) and L peaky (I,I *) are used to measure the consistency of feature point positions and the local peak distribution of feature points. Assume that the value obtained in step 3 is After the homography matrix Homo transformation, we get Will I P and Divide into non-overlapping M P Block, block size is S P ×S P (The image size is W×H×C, and the block size is S×S×C in space. The position response map size here is W×H, and the block size is S in space. P ×S P ), the division method is the same as the block division method in step 2-1, where the relationship between the number of blocks and the block size is For I P and The column vectors formed by flattening the i′th block are denoted as P[i′] and where i′∈{1, 2, ..., M P}.

[0035]

[0036] Where ||·||2 represents the vector norm, and the superscript T represents the transpose of the vector.

[0037]

[0038] Among them, P[i′](j) represents the j-th element of the column vector P[i′], Represents the maximum value of all elements of the column vector P[i′]. represents the average value of all elements of the column vector P[i′]; Represents a column vector The jth element of Represents a column vector The maximum value of all elements, Represents a column vector The average value of all elements.

[0039] Finally, the two types of losses are balanced by the balance parameter λ to obtain the total loss L of the feature point position P (I,I * ).

[0040] L P (I,I * )=L cos (I,I * )+λL peaky (I,I * ) (9)

[0041] 4-2. Constructing the descriptor loss L D (I,I * ). The descriptor loss is described using the hinge loss (see formula 10). Let the position (i, j) in image I correspond to the augmented image I * Position in (i * ,j * ), descriptor vector graph I D The descriptor vector corresponding to position (i, j) is f i，j , vector graphics Middle position (i * ,j * )The corresponding descriptor vector is f i，j * . Remember the position (i, j) in I and I * The distance between the positions (k, l) in the ij，kl , by comparing the distance threshold t (t is a manually set parameter, in the following specific implementation, the t value is set to 9) (see formula 11), we get c ij，kl .

[0042]

[0043]

[0044] Among them, δ, m p and m n are hyperparameters, representing weight coefficients, positive boundaries, and negative boundaries respectively. The superscript T represents transposition.

[0045] 5. Train the feature point detection and description network N (see the training flow chart Figure 8 ).

[0046] 5-1. Implement step 1 to obtain the training data set D.

[0047] 5-2. For an image I∈D in the training dataset D, perform step 3 to obtain the augmented image I * . Image I and augmented image I * Input them into the network N constructed in step 2 respectively to obtain the position response graph I P , descriptor vector graph L D , Position Response Plot and descriptor vector graph

[0048] 5-3. For image I and augmented image I * , implement step 4 and calculate the loss function L(I, I *). The training parameters of the network N are updated by the Stochastic Gradient Descent (SGD) algorithm, where the learning rate in the SGD algorithm is α1 and the momentum is α2.

[0049] 5-4. Traverse all images in the training dataset D and implement steps 5-2 to 5-3 to complete one round of training.

[0050] 5-5. Implement step 5-4 for a total of e rounds to obtain the trained feature point detection and description network N.

[0051] 6. Test: For test image I t , size is W t ×H t ×C t , where W t 、H t 、C t Represent the test pictures I t The width, height, and number of channels are calculated, and finally a set of feature points is obtained.

[0052] 6-1.Change I t Input the network N built and trained in steps 2 to 5, and output the position response map and descriptor vector graph The sizes are W t ×H t and W t ×H t ×N F , where N F Indicates the number of channels of the descriptor vector map (the number of channels of the descriptor vector map is the same as the number of channels of the sub-feature map).

[0053] 6-2. Calculate the feature point position set P. For the position response map Perform non-maximum suppression processing to obtain the feature point position set P = {(x1, y1), ..., (x np ,y np )}, where np represents the number of feature points. Non-maximum suppression uses a preset w×w size window in Slide up, the sliding step is s, calculate The local maximum position on the w×w window size The local maximum position As the feature point position (in The maximum value position in the w×w window is the detected feature point position), where

[0054] 6-3. Calculate the feature point set FP. i″,y i″ )∈P, select the descriptor vector map Middle position (x i″ ,y i″ ) vector As the position (x i″ ,y i″ ) at the descriptor vector. Perform the descriptor vector selection operation for all (i″, j″) = (1, 1), ..., (p, p) to obtain the feature point set

[0055] The present invention improves and optimizes on the basis of the Transformer framework, performs detection and description on global features, and avoids the performance limitations brought by local receptive fields. At the same time, by introducing feature point guidance information, feature point detection and description become more intelligent and robust. On the one hand, the position of feature points is estimated in advance to provide the network with prior position information, thereby improving training efficiency; on the other hand, the guidance information helps the network improve its robustness to factors such as scale and rotation. The present invention combines the Transformer framework with the feature point detection task, and the guidance information generated by mature traditional methods can avoid most of the false detections. At the same time, the redundant calculation amount is reduced in some details. The entire system is of great help to the application of feature point detection and description algorithms in practical engineering. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Flowchart of the feature point detection and description method based on global features and guidance information of the present invention;

[0057] Figure 2 The structural diagram of the feature point detection and description network N;

[0058] Figure 3 is the encoder network N E Structural diagram of

[0059] Figure 4 Generate network N for guidance information G Structural diagram of

[0060] Figure 5 is the decoder network N D Structural diagram of

[0061] Figure 6 N is the block layer EB Schematic diagram of an example of the division method;

[0062] Figure 7 This is the structural diagram of the basic block of Transformer;

[0063] Figure 8 This is the network training flow chart;

[0064] Figure 9 This is a network testing flowchart. DETAILED DESCRIPTION

[0065] The following is a more detailed description of the embodiments of the present invention with reference to the accompanying drawings and reference numerals, so that those skilled in the art can implement the embodiments according to the present invention after studying the present description. It should be understood that the embodiments described in this example are only for explaining the present invention and are not intended to limit the present invention.

[0066] Example 1

[0067] like Figure 1 As shown, the present invention proposes a feature point detection and description method based on global features and guidance information, which includes the following steps:

[0068] 1. Construct a training dataset: Use the COCO 2017 training set as training dataset D. For any image I∈D in training dataset D, the size is W×H×3, where W and H represent the width and height of training image I, respectively, and 3 represents the number of channels in training image I. Input I into the network N constructed in step 2.

[0069] 2. Construct an end-to-end feature point detection and description network N (its structure is as follows Figure 2 As shown): Network N consists of encoder network N E (Its structure is as Figure 3 As shown), guide information generation network N G (Its structure is as Figure 4 as shown) and the decoder network N D (Its structure is as Figure 5 As shown). Among them, the encoder network N E By block layer N EB , linear embedding layer N EE , position encoding layer N EP and feature encoding layer N EF Cascade composition; guide information generation network N G By the detection layer N GD , block layer N GB and a linear embedding layer N GE Cascade composition; decoder network N D By feature decoding layer N DF , feature point position decoding layer N DP and descriptor decoding layer N DD composition.

[0070] 2-1. Design encoder network N E (Its structure is as Figure 3 N E By block layer NEB , linear embedding layer N EE , position encoding layer N EP and feature encoding layer N EF Composition. Block layer N EB Divide the input image I into M non-overlapping blocks (see Figure 6 ), the block size is 16×16×3, and the relationship between the number of blocks and the block size is Each block is flattened into a 1-dimensional column vector of size 768×1, and the block vector sequence {b (1) , b (2) ,...,b (M)},in Indicates the i b block vector, i b ∈{1, 2, ..., M}.

[0071] 2-2. The block vector sequence {b (1) , b (2) ,...,b (M) Input linear embedding layer N EE , get the embedding vector sequence {e (1) , e (2) ,...,e (M)},in Indicates the i e embedding vector, i e ∈{1, 2, ..., M}. Among them, the linear embedding layer N EE It consists of n layers of multi-layer perceptron cascade, which transforms the block vector Mapped to embedding vector by formula (1) The size of is 768×1, where i be ∈{1, 2, ..., M}, N e Indicates the number of elements in the embedding vector.

[0072]

[0073] Among them, N EE (·) represents the operation performed by the linear embedding layer, W EE and b represent the weights and bias of the linear embedding layer.

[0074] 2-3. Embed the vector sequence {e (1) , e (2) ,...,e (M)}Input position encoding layer N EP , output position embedding vector sequence Among them, the position encoding layer N EP Embed the vector With the i epLearnable position parameters of block regions Add together to get the position embedding vector and The size is 768×1, where i ep ∈{1, 2, ..., M}.

[0075]

[0076] Among them, N EP (·) represents the operation performed by the position encoding layer.

[0077] 2-4. Embed positions into vector sequences Input feature encoding layer N EF , output encoding feature sequence in The size is 768×1, N f The number of elements representing the encoded features. Feature encoding layer N EF It consists of 4 transformer basic blocks (its structure is as follows Figure 7 Each transformer basic block consists of a multi-headed self-attention layer (MSA), a multi-layer perceptron (MLP), and two layers of regularization (LN). Let the input sequence of the j-th transformer basic block be Where j∈{1, 2, ..., L}. Let the vector sequence With the matrix One-to-one correspondence, where the matrix Z j By vector sequence Concatenate by column to get the vector sequence The matrix Z j The j+1th transformer basic block converts the sequence Mapping encoding to sequence

[0078]

[0079] Among them, LN(·) represents the layer normalization operation, MSA(·) represents the multi-head attention operation, and Z j and Z j+1 denote the matrix corresponding to the input sequence and the matrix corresponding to the output sequence of the j-th transformer basic block, j∈{1, 2, 3, 4}. The calculation of the MSA(·) function is as follows:

[0080]

[0081] Among them, X represents the input matrix of MSA(·), head k Represents the output of the k-th head of the multi-head self-attention layer, Concat represents the splicing operation in the row dimension, and the number of heads in the multi-head self-attention layer is 8. represents the output mapping matrix of the multi-head self-attention layer, Denote the query mapping matrix, key mapping matrix, and value mapping matrix of the kth head, respectively, k∈{1, 2, ..., 8}, d X Represents the number of rows of the input matrix X. The calculation of the MLP(.) function is as follows:

[0082] Y=MLP(X′)=W M ·X′+b M (5)

[0083] in, represents the input matrix of the MLP(·) function, and are the weights and biases of the MLP.

[0084] Will Recorded as The encoded feature sequence is the final output of the feature encoding layer.

[0085] 2-5. Design of guidance information generation network N G . N G By the detection layer N GD , block layer N GB and a linear embedding layer N GE Composition. Detection layer N GD Use the SIFT algorithm to detect feature points on the input image I and obtain the mask matrix M mask , size is W×H×3, where the matrix M mask The values of the three channels at the non-feature point position are all 0, and the values of the three channels at the feature point position are all 1.

[0086] 2-6. Block layer N GB Mask matrix M mask Divide the blocks into 16×16×3 blocks, and the number of blocks is M. Flatten each block into a 1-dimensional column vector to obtain the mask vector sequence {m (1) , m (2) ,...,m (M)}, where the i m mask vectors The size of is 768×1, i∈{1, 2, ..., M}.

[0087] 2-7. The mask vector sequence {m (1) , m(2) ,...,m (M) Input linear embedding layer N GE , get the boot information sequence Linear embedding layer N GE It consists of 1 layer of multi-layer perceptron, the i-th ge The multilayer perceptron transforms the mask vector Mapping to boot information The size of is 768×1, where i∈{1, 2, ..., M}. The information sequence is guided The coding feature sequence obtained in steps 2-4 Merge by splicing to get the decoder network N D The input sequence

[0088] 2-8. Design decoder network N D . N D By feature decoding layer N DF , feature point position decoding layer N DP and descriptor decoding layer N DD Composition. Feature decoding layer N DF It consists of two transformer basic blocks cascaded as described in steps 2-4, with the input feature sequence Output decoding feature sequence in and By N DF Then the corresponding output and will sequence with sequence The corresponding addition results in the characteristic sequence {t (1) , t (2) ,...,t (M)}, that is in Represents the i-th feature sequence a feature vectors, i a ∈{1, 2, ..., M}.

[0089] 2-9. The feature sequence {t (1) , t (2) ,...,t (M)} Input feature point position decoding layer N DP In the output position response map I P , size is W×H. Feature point position decoding layer N DP It consists of 1 layer of multi-layer perceptron, N DP First, the block feature vector t (i) Mapped to position vector p(i) , p (i) The size is 256×1, where i∈{1, 2, ..., M}; then the position response vector p (i) Arranged in rows into blocks of size 16×16 Get block position response sequence Finally, the sequence Combine the blocks in the order of step 2-1 to obtain the output position response map I P .

[0090] 2-10. The feature sequence {t (1) , t (2) ,...,t (N)} Input descriptor decoding layer N DD In the output descriptor response diagram I D , size is W×H×256. Descriptor decoding layer N DD It consists of 1 layer of multi-layer perceptron, N DD First, the block feature vectors t (i) Mapped to block descriptor vector d (i) , the size is 65536×1, where i∈{1, 2, ..., M}; then d (i) Rearrange into blocks of size 16×16×256 Traverse i=1, 2, ..., M to get the block descriptor vector sequence Finally, the sequence Combine the blocks in the order of step 2-1 to get the output descriptor vector image I D .

[0091] 3. Add Gaussian noise δ with a mean of 0 and a variance of 30 to image I, and perform random translation, rotation, and color changes to obtain the augmented image I * , the size is W×H×3, where W and H represent the augmented image I * The width and height of . Let the homography matrix The original image I and the augmented image I * The pixel position correspondence of the overlapping area between the augmented images I * Input the network N constructed in step 2 to obtain the position response map and descriptor vector graph The sizes are W×H and W×H×256 respectively.

[0092] 4. Construct the loss function L(I, I * ). Loss function L(I,I * ) is lost by the feature point position L P (I,I * ) and descriptor loss LD (I,I * )composition:

[0093] L(I,I * )=L P (I,I * )+0.001·L D (I,I * ) (6)

[0094] 4-1. Constructing feature point position loss L P (I,I * ). Feature point position loss uses local cosine loss L cos (I,I * )(see formula 7) and peak loss L peaky (I,I * )(see formula 8) together describe, L cos (I,I * ) and L peaky (I,I * ) are used to measure the consistency of feature point positions and the local peak distribution of feature points. Assume that the value obtained in step 3 is After the homography matrix Homo transformation, we get Will I P and Divide into non-overlapping M P Block (division method see Figure 6 ), the block size is 8×8, where the relationship between the number of blocks and the block size is For I P and The column vectors formed by flattening the i′th block are denoted as P[i′] and where i′∈{1, 2, ..., M P}.

[0095]

[0096] Where ||·||2 represents the vector norm, and the superscript T represents the transpose of the vector.

[0097]

[0098] Among them, P[i′](j) represents the j-th element of the column vector P[i′], Represents the maximum value of all elements of the column vector P[i′]. represents the average value of all elements of the column vector P[i′]; Represents a column vector The jth element of Represents a column vector The maximum value of all elements, Represents a column vector The average value of all elements.

[0099] Finally, the two types of losses are balanced by the balance coefficient 0.1, and the total loss L of the feature point position is obtained. P (I,I * ).

[0100] L P (I,I * )=L cos (I,I * )+λL peaky (I,I * ) (9)

[0101] 4-2. Constructing the descriptor loss L D (I,I * ). The descriptor loss is described using the hinge loss (see formula 10). Let the position (i, j) in image I correspond to the augmented image I * Position in (i * ,j * ), descriptor vector graph I D The descriptor vector corresponding to position (i, j) is f i，j , vector graphics Middle position (i * ,j * )The corresponding descriptor vector is f i，j * . Remember the position (i, j) in I and I * The distance between the positions (k, l) in the ij，kl , by comparing the distance threshold (see formula 11), we get c ij，kl .

[0102]

[0103]

[0104] 5. Train the feature point detection and description network N (see the training flow chart Figure 8 ).

[0105] 5-1. Implement step 1 to obtain the training data set D.

[0106] 5-2. For an image I∈D in the training dataset D, perform step 3 to obtain the augmented image I * . Image I and augmented image I * Input them into the network N constructed in step 2 respectively to obtain the position response graph I P , descriptor vector graph I D , Position Response Plot and descriptor vector graph

[0107] 5-3. For image I and augmented image I * , implement step 4 and calculate the loss function L(I, I * ). The training parameters of the network N are updated by the Stochastic Gradient Descent (SGD) algorithm, where the learning rate in the SGD algorithm is 10 -4 , the learning rate is reduced by a factor of 10 every 10 rounds, the weight decay coefficient is 0.0005, and the momentum is 0.9.

[0108] 5-4. Traverse all images in the training dataset D and implement steps 5-2 to 5-3 to complete one round of training.

[0109] 5-5. Implement step 5-4 for a total of 100 rounds to obtain the trained feature point detection and description network N.

[0110] 6. Test: For test image I t , size is W t ×H t ×3, where W t 、H t Represent the test picture I t Width and height (see the test flow chart Figure 9 ).

[0111] 6-1.Change I t Input the trained network N obtained in step 5 and output the position response map and descriptor vector graph The sizes are W t ×H t and W t ×H t ×256.

[0112] 6-2. Calculate the feature point position set P. For the position response map Perform non-maximum suppression processing to obtain the feature point position set P = {(x1, y1), ..., (x p ,y p )}, where p represents the number of feature points. Non-maximum suppression uses a 3×3 window in Slide up, the sliding step is 1, calculate The local maximum position on the 3×3 window size The local maximum position As the detected feature point position (in There are 9 numbers in the 3×3 window, and the position of the maximum value among the 9 numbers is the position of the detected feature point).

[0113] 6-3. Calculate the feature point set FP. For (x i″ ,y i″ )∈P, select the descriptor vector map Middle position (x i″ ,y i″ ) vector As the position (x i″ ,y i″ ) at the descriptor vector. Perform the descriptor vector selection operation for all (i″, j″) = (1, 1), ..., (p, p) to obtain the feature point set (The final result of the task of the present invention is to obtain a set of feature points, which is the basis for other image tasks (such as three-dimensional reconstruction, etc.)

[0114] The embodiments described above are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

Claims

1. A feature point detection and description method based on global features and guidance information, characterized in that: The method comprises the following steps: S1) Construct a training dataset: Collect a series of image data to construct a training dataset D. For any image I∈D in the training dataset D, the size is W×H×C, where W, H, and C represent the width, height, and number of channels of image I respectively; input I into the network N constructed in step 2; S2) Construct an end-to-end feature point detection and description network N, input the image I into the constructed network N, and obtain the descriptor vector graph of the image I, wherein the network N consists of the encoder network N E , guide information generation network N G and decoder network N D composition; S3) Add random noise δ to the image I, and perform random translation, rotation and color change to obtain the augmented image I * , the size is W×H×C, where W, H, and C represent the augmented image I * The width, height, and number of channels of For image I and augmented image I * The pixel position correspondence between the overlapping areas, represents the field of real numbers, Represents all 3×3 real matrices; the augmented image I * Input the network N constructed in step S2) to obtain the position response map and descriptor vector graph The sizes are W×H and W×H×N respectively F ; S4) Construct the loss function L(I,I) of the network N * ), loss function L(I,I * ) is lost by the feature point position L P (I,I * ) and descriptor loss L D (I,I * )composition: L(I,I * )=L P (I,I * )+βL D (I,I * ) (1) Among them, the hyperparameter β is used to balance the ratio of feature point position loss and descriptor loss; S5) Train the feature point detection and description network N Traverse all images in the training data set D and implement steps S2) to S4) to complete one round of training; after completing the preset e rounds of training, obtain the trained feature point detection and description network N; S6) Test feature point detection and description network N Test image I t Input the trained feature point detection and description network N, and finally get the feature point set, where the test image I t The size of W t ×H t ×C t , W t 、H t 、C t Represent the test picture I t The width, height, and number of channels.

2. The feature point detection and description method based on global features and guidance information according to claim 1, characterized in that: Described step S2) specifically comprises: S21) Design encoder network N E , N E By block layer N EB , linear embedding layer N EE , position encoding layer N EP and feature encoding layer N EF Composition; Block layer N EB The input image I is divided into M non-overlapping blocks of size S×S×C, where S and C represent the spatial size and number of channels of the block respectively. The relationship between the number of blocks and the block size is: Each block is flattened into a 1-dimensional column vector of size (S·S·C)×1, and the block vector sequence {b (1) ,b (2) ,...,b (M) },in Indicates the i b block vector, i b ∈{1,2,...,M}; S22) The block vector sequence {b (1) ,b (2) ,...,b (M) Input linear embedding layer N EE , get the embedding vector sequence {e (1) ,e (2) ,...,e (M) },in Indicates the i e embedding vector, i e ∈{1,2,...,M}, where the linear embedding layer N EE It consists of n layers of multi-layer perceptron cascade, which transforms the block vector Mapped to embedding vector by formula (2) The size is N e ×1, where i be ∈{1,2,...,M},N e represents the number of elements in the embedding vector, Among them, N EE (·) represents the operation performed by the linear embedding layer, W EE and b represent the weights and biases of the linear embedding layer; S23) embed the vector sequence {e (1) ,e (2) ,...,e (M) }Input position encoding layer N EP , output position embedding vector sequence Among them, the position encoding layer N EP Embed the vector With the i ep Learnable position parameters of block regions Add together to get the position embedding vector and The size is N e ×1, where i ep ∈{1,2,...,M}, Among them, N EP (·) represents the operation performed by the position encoding layer; S24) embed the position into a vector sequence Input feature encoding layer N EF , output encoding feature sequence in The size is N f ×1, i=1,2,...,M,N f Indicates the number of elements of the coded features, feature coding layer N EF It consists of L transformer basic blocks repeatedly cascaded, where each transformer basic block consists of a multi-head self-attention layer MSA, a multi-layer perceptron MLP and two layers of regularization LN; let the input sequence of the j-th transformer basic block be Where j∈{1,2,...,L}, let the vector sequence With the matrix One-to-one correspondence, where the matrix Z j By vector sequence Concatenate by column to get the vector sequence By matrix Z j The j+1th transformer basic block obtains the sequence Mapping encoding to sequence Among them, LN(·) represents the layer normalization operation, MSA(·) represents the multi-head attention operation, and Z j and Z j+1 They represent the matrix corresponding to the input sequence and the matrix corresponding to the output sequence of the j-th transformer basic block respectively. The calculation of the MSA(·) function is as follows: Among them, X represents the input matrix of MSA(·), head k represents the output of the kth head of the multi-head self-attention layer, Concat represents the concatenation operation in the row dimension, h represents the number of heads in the multi-head self-attention layer, and W O represents the output mapping matrix of the multi-head self-attention layer, Denote the query mapping matrix, key mapping matrix, and value mapping matrix of the kth head, k∈{1,2,...,h}, d X represents the number of rows of the input matrix X; the calculation of the MLP(·) function is as follows: Y=MLP(X′)=W M ·X′+b M (6) Where X′ represents the input matrix of the MLP(·) function, W M and b M are the weights and biases of MLP; Will Recorded as The encoded feature sequence as the final output of the feature encoding layer; S25) Design guidance information generation network N G , N G By the detection layer N GD , block layer N GB and a linear embedding layer N GE Cascade composition, detection layer N GD Use any other feature point detection algorithm A to detect feature points on the input image I and obtain the mask matrix M mask , size is W×H×C, where the matrix M mask The values of the C channels at the non-feature point positions are all 0, and the values of the C channels at the feature point positions are all 1; S26) Block layer N GB Mask matrix M mask Divide the blocks into blocks, each block size is S×S×C, and the number of blocks is M; flatten each block into a 1-dimensional column vector to obtain the mask vector sequence {m (1) ,m (2) ,...,m (M) }, where the i m mask vectors The size of is (S·S·C)×1, i m ∈{1,2,...,M}; S27) The mask vector sequence {m (1) ,m (2) ,...,m (M) Input linear embedding layer N GE , get the boot information sequence Linear embedding layer N GE It consists of m layers of multi-layer perceptron cascade, the i-th ge The multilayer perceptron transforms the mask vector Mapping to boot information The size is N f ×1, where i ge ∈{1,2,...,M}, which guides the information sequence The coding feature sequence obtained in step S24) Merge by splicing to get the decoder network N D The input sequence S28) Design decoder network N D , N D By feature decoding layer N DF , feature point position decoding layer N DP and descriptor decoding layer N DD Composition, feature decoding layer N DF It consists of T transformer basic blocks cascaded as described in step S24), and the input feature sequence Output decoding feature sequence in and By N DF Then the corresponding output and i df ∈{1,2,...,M}, the sequence with sequence The corresponding addition results in the characteristic sequence {t (1) ,t (2) ,...,t (M) },Right now in Represents the i-th feature sequence a feature vectors, i a ∈{1,2,...,M}; S29) The feature sequence {t (1) ,t (2) ,...,t (M) } Input feature point position decoding layer N DP In the output position response map I P , size is W×H, feature point position decoding layer N DP It is composed of u layers of multi-layer perceptron cascade, N DP First, the block feature vector t (i) Mapped to position vector p (i) , p (i) The size is (S·S)×1, where i∈{1,2,...,M}; then the position response vector p (i) Arrange rows into blocks of size S×S Get block position response sequence Finally, the sequence According to the block order combination in step S21), the output position response image I is obtained. P ; S210) The feature sequence {t (1) ,t (2) ,...,t (M) } Input descriptor decoding layer N DD In the output descriptor response diagram I D , size is W×H×N F , where N F Denotes the dimension of the descriptor vector; descriptor decoding layer N DD It consists of v layers of multi-layer perceptron cascade, N DD First, the block feature vectors t (i) Mapped to block descriptor vector d (i) , d (i) The size of (S·S·N F )×1; then d (i) Rearrange to size S×S×N F Block Traverse i=1,2,...,M to get the block descriptor vector sequence Finally, the sequence According to the block order combination in step S21), the output descriptor vector image I is obtained. D .

3. The feature point detection and description method based on global features and guidance information according to claim 2, characterized in that: The step S4) specifically includes: S41) Construct feature point position loss L P (I,I * ), the feature point position loss uses the local cosine loss L in formula (7) cos (I,I * ) and the peak loss L in formula (8) peaky (I,I * ) to describe, L cos (I,I * ) and L peaky (I,I * ) are used to measure the consistency of the feature point position and the local peak distribution of the feature point, respectively. Suppose that the value obtained in step S3) After the homography matrix Homo transformation, we get Will I P and Divide into non-overlapping M P Block, block size is S P ×S P The division method is the same as the block division method in step S21), where the relationship between the number of blocks and the block size is For I P and The column vectors formed by flattening the i′th block are denoted as P[i′] and where i′∈{1,2,...,M P }, Where ||·||2 represents the vector two-norm, and the superscript T represents the transpose of the vector; Among them, P[i′](j) represents the j-th element of the column vector P[i′], represents the maximum value of all elements of the column vector P[i′], represents the average value of all elements of the column vector P[i′]; Represents a column vector The jth element of Represents a column vector The maximum value of all elements, Represents a column vector The average value of all elements; Finally, the two types of losses are balanced by the balance parameter λ to obtain the total loss L of the feature point position P (I,I * ), THE P (I,I * )=L cos (I,I * )+λL peaky (I,I * ) (9) S42) Construct descriptor loss L D (I,I * ), the descriptor loss is described using the hinge loss in formula (10), and the position (i, j) in image I corresponds to the augmented image I * Position in (i * ,j * ), descriptor vector graph I D The descriptor vector corresponding to position (i, j) is f i,j , vector graphics Middle position (i * ,j * )The corresponding descriptor vector is f i,j * , remember the position (i,j) in I and I * The distance between the positions (k, l) in the ij,kl , by comparing the distance threshold t, we can get c in formula (11) ij,kl , Among them, δ, m p and m n are hyperparameters, representing weight coefficients, positive boundaries, and negative boundaries respectively. The superscript T represents transposition.

4. The feature point detection and description method based on global features and guidance information according to claim 3, characterized in that: The step S6) specifically includes: S61) I t Input the trained feature point detection and description network N and output the position response map and descriptor vector graph The sizes are W t ×H t and W t ×H t ×N F , where N F Indicates the number of channels of the descriptor vector image; S62) Calculate the feature point position set P, for the position response map Perform non-maximum suppression processing to obtain the feature point position set P = {(x1, y1), ..., (x np ,y np )}, where np represents the number of feature points, and non-maximum suppression uses a preset w×w size window in Slide up, the sliding step is s, calculate The local maximum position on the w×w window size The local maximum position As the feature point location, (i t ,j t )∈{(1,1),(2,2),...,(np,np)}; S63) Calculate the feature point set FP, for (x i″ ,y i″ )∈P, select the descriptor vector map Middle position (x i″ ,y i″ ) vector As the position (x i″ ,y i″ ), perform the descriptor vector selection operation for all (i″,j″)=(1,1),...,(p,p) to obtain the feature point set

Images