An end-to-end rotationally robust feature point detection and description method

Abstract

The application discloses an end-to-end rotation robust feature point detection and description method. The method adds a direction decoding network on the basis of feature point position calculation, estimates the direction of the block where all the feature points are located in one forward calculation, greatly reduces the storage and calculation cost, and provides the direction information of the feature points for the calculation of the descriptor. Meanwhile, the rotation isovariant convolution network is adopted to reduce the calculation cost of the direction vector and improve the reusability of the feature map. The application combines the feature point detection and description network, guides the description process through the detection process, and reduces the number of dense descriptors through the interpolation method. Meanwhile, the end-to-end method is adopted, which is beneficial to increase the coupling of the detection and description, reduce the calculation amount, and greatly help the feature point detection and description algorithm in the practical engineering application.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing and computer vision, and particularly relates to an end-to-end rotationally robust feature point detection and description method. BACKGROUND

[0002] Establishing the pixel-level correspondence between images is the basis of visual tasks such as three-dimensional reconstruction, motion estimation structure, and simultaneous localization and mapping. As an effective method for estimating pixel-level correspondence, image feature points aim to detect and describe feature points between two or more images. The core problem of image feature points is how to detect stable feature point positions and generate robust descriptor vectors between multiple images under the interference of factors such as changes in view angle, changes in illumination, scale, rotation, and noise.

[0003] Feature point detection and description methods can be divided into traditional methods and deep learning-based methods. In the early stage of research, researchers designed scale-invariant and rotation-invariant detectors based on prior knowledge in related fields, but these traditional feature point detectors performed poorly on many datasets due to the limitations of prior knowledge. In recent years, due to the powerful ability of deep learning to learn from data, deep learning-based feature point detection methods have made great progress, and many studies on deep learning-based feature point detection and description methods have emerged, especially end-to-end methods have become the mainstream. However, these methods only focus on training strategies and the impact of view and illumination changes, and fail to address the performance degradation caused by rotation changes in end-to-end networks. DeTone et al. proposed the SuperPoint algorithm (see "Superpoint: Self-supervised interest point detection and description [C]", DeTone, D., Malisiewicz, T., & Rabinovich, A., In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2018: 224-236), which treats the feature point detection problem as a classification problem and applies self-supervised learning methods to the feature point task, avoiding the problem of missing true values or being unable to label true values. In the ordinary self-supervised learning framework, the Homographic Adaptation strategy is added to improve the geometric consistency of the detector. A fully convolutional network is constructed to process full-size input images, and the feature point position and corresponding descriptor vector are calculated in one forward propagation, improving the detection efficiency. However, the training process of the SuperPoint algorithm is complex and tedious, the number of detected feature points is small, the descriptor is not robust in areas with repeated patterns (such as windows of high-rise buildings) and extreme lighting, and it is sensitive to image rotation.To simplify the training process and improve the robustness of the descriptor, Revaud et al. proposed the R2D2 algorithm (see "R2d2: Reliable and repeatable detector and descriptor [J]", Revaud, Jerome, et al., Advances in neural information processing systems. 2019: 12405-12415) to treat the feature point detection problem as a regression problem, using data augmentation to obtain the homography true value between two images for self-supervised training, which not only greatly simplifies the training process, but also enhances the controllability of training; Using dilated convolution instead of step convolution to maintain spatial resolution and avoid feature maps becoming rough; On the basis of feature point position branch and descriptor vector branch, the feature point reliability branch is added to improve the robustness of feature points in image style repeated areas. However, the performance of the detector is still poor under image rotation and extreme illumination. Reference "D2-net: A trainable cnn for joint description and detection of local features [C]", Dusmanu M, Rocco I, Pajdla T, et al., Proceedings of the IEEE conference on computer vision and pattern recognition. 2019: 8092-8101" proposes a D2Net method to solve the problem of extreme illumination, which uses the deep feature map of the VGG network as the corresponding map of the feature point, and extracts the feature point at the local maximum position of the corresponding map. D2Net has certain effect under severe illumination, but only uses deep features, which has the disadvantages of low positioning accuracy, slow speed and sensitivity to rotation and scale. The invention patent with publication number CN114067100A proposes a feature point method for detecting and generating descriptors simultaneously under difficult conditions, which regards each channel of the feature map as the response map of different detection functions, and selects the position with the maximum channel dimension and the local maximum spatial dimension as the feature point detection result. The results show that the effect is significant under extreme conditions.Reference "D2d: Keypoint extraction with describe to detect approach [C], Tian Y, Balntas V, Ng T, et al., Proceedings of the Asian Conference on Computer Vision. 2020: 223-240" proposes a D2D method, defines the absolute and relative significance of the feature points, and obtains the position and descriptor of the feature points in the describe-to-detect strategy, which only needs to extract the dense descriptor of the image, and then completes the detection stage at a very low cost, but this method is sensitive to scale and rotation, etc.

[0004] The existing end-to-end method only completes the feature point position detection and descriptor vector generation in the network, and the robustness to the rotation factor is only obtained through data enhancement in the training process, without designing a special structure for learning, resulting in that the algorithm performance seriously depends on the data set and the generalization ability is poor. The present application proposes an end-to-end rotation robust feature point detection and description method, which effectively estimates the direction of the feature point by introducing a direction estimation decoding module, and avoids multiple forward calculations by using a rotation-invariant feature extraction network to generate feature maps of various rotation angles once. The above modules are embedded in the end-to-end network, which reduces the amount of calculation while improving the detection accuracy and the robustness of the descriptor. SUMMARY

[0005] The present application belongs to the field of image processing and computer vision, and provides an end-to-end feature point detection and description method which is robust to rotation, relates to a series of methods such as feature point detection and description, feature encoding and decoding, and direction estimation in deep learning, and is used to solve the influence of the rotation factor of the image on the performance of the feature point.

[0006] In order to achieve the above purpose, the present application provides an end-to-end rotation robust feature point detection and description method, which specifically comprises the following steps:

[0007] 1. Construct a training data set: collect a series of picture data to construct a training data set D. For any picture I in the training data set D, the size is WxHxC, wherein W, H and C respectively represent the width, height and channel number of the training picture I.

[0008] 2. Construct an end-to-end feature point detection and description network N, input I into the constructed network N to obtain the descriptor vector of the picture I. Wherein, the network N is composed of a feature point detection network N P and a feature point description network N D in parallel. N P is composed of an encoding network N PEand decoding network N PD in series, where N PD is a position decoding network PDP and a direction decoding network N PDO in parallel; N D is a rotation equivariant convolution network N DR and a selection network N DS in series.

[0009] 2-1. Construct a feature point detection network N P . N P is an encoding network N PE and a decoding network N PD in series. The encoding network N PE is a fully convolutional network, which includes l convolutional groups, each of which is composed of a convolutional layer, a batch normalization layer, and an activation layer in series. The picture I is input into the encoding network N PE , and an encoded feature map E with a size of W E ×H E ×N E is obtained, where W E , H E , and N E represent the width, height, and channel number of the encoded feature map E, respectively.

[0010] 2-2. The encoded feature map E is input into a position decoding network N PDP , and a position response map I P with a size of W×H is obtained, where N PDP is composed of m convolutional layers in series. The value I P at position (i, j) in I P (i, j) represents the feature point response value at position (i, j) in the picture I, where i ∈ {1,..., H} and j ∈ {1,..., W}.

[0011] 2-3. The encoded feature map E is input into a direction decoding network N PDO , and a direction estimation map I O with a size of W×H is obtained, where N PDO is composed of n convolutional layers in series. The value I O at position (i, j) in I O (i, j) represents the direction estimation value at position (i, j) in the picture I, where i ∈ {1,..., H} and j ∈ {1,..., W}.

[0012] 2-4. Construct a feature point description network N D . N D is a rotation equivariant convolution network N DR and a selection network N DSCascaded composition. Rotationally equivariant convolutional network N DR It consists of u rotationally equivariant convolutional groups and v rotationally equivariant max-pooling downsampling layers. Each rotationally equivariant convolutional group is composed of a cascaded rotationally equivariant convolutional layer, a rotationally equivariant batch regularization layer, and a rotationally equivariant activation layer. Input image I into N. DR In the process, the output feature map F is of size K×W. F ×H F ×N F F consists of K sub-feature maps of the same size, with each sub-feature map having a size of W. F ×H F ×N F (The sub-feature map represents a portion of feature map F), W F H F N F Let F[k] represent the width, height, and number of channels of the sub-feature map, respectively. Let F[k] be the k-th sub-feature map of F, where k∈{1,2,...,K}. The spatial downsampling rate ε of the sub-feature map F[k] relative to the input image I is ε = W / W. F .

[0013] 2-5. For the orientation estimation map I obtained in step 2-3 O The value at position (i,j) is I. O (i,j). Input the feature map F obtained in steps 2-4 into the selection network N. DS In the middle, select network N DS Based on value I O (i,j), calculate I based on formula (1) O (i,j) corresponds to the index k of the sub-feature map in F. Select the sub-feature map F[k].

[0014]

[0015] Where 360 ​​represents 360 degrees, I O (i,j) represents the direction estimation diagram I. O The value at position (i,j) in the middle. This indicates rounding down; based on formula (2), according to the coordinate values ​​of position (i,j), calculate the four position coordinates (p1,q1), (p2,q2), (p3,q3), and (p4,q4) corresponding to position (i,j) in the sub-feature map F[k];

[0016]

[0017] in, denotes down-sampling, and ε denotes the spatial down-sampling rate of the sub-feature map F[k] with respect to the input picture I; the vectors at positions (p1, q1), (p2, q2), (p3, q3), (p4, q4) in the sub-feature map F[k] are denoted as The four vectors are denoted as The vectors are interpolated by formula (3) to obtain the vector

[0018]

[0019] The vectors are denoted as The L2 regularization is performed by formula (4) to obtain the descriptor vector f at position (i, j) i,j , which has a size of N F × 1.

[0020]

[0021] where ||·||2denotes the vector two-norm.

[0022] 2-6. Steps 2-5 are performed for all i = 1,..., H and j = 1,..., W to obtain the descriptor vector map I D , which has a size of W × H × N F , where the vector at position (i, j) in I D is f i,j , which denotes the descriptor vector at position (i, j) in the input picture I.

[0023] 3. Random noise δ is added to the picture I to perform random translation, rotation, color change, etc. to obtain the augmented picture I * , which has a size of W × H × C, where W, H, and C respectively denote the width, height, and channel number of the augmented picture I * . Let the homography matrix be the pixel position correspondence relationship between the original picture I and the overlapping region of the augmented picture I * , denote the real number field, denote all 3 × 3 real matrices. The augmented picture I * is input into the network N constructed in step 2 to obtain the position response map and the descriptor vector map , which have sizes of W × H and W × H × N F , respectively.

[0024] 4. The loss function L(I, I * ) (L(I, I * ) denotes a function with I and I * as inputs) of the network N is constructed. The loss function L(I, I* ) by feature point position loss L P (I, I * ) and descriptor loss L D (I, I * ) consists of:

[0025] L(I, I * ) = L P (I, I * ) + βL D (I, I * ) (5)

[0026] where the hyper-parameter β is used to balance the proportion of feature point position loss and descriptor loss.

[0027] 4-1. Constructing feature point position loss L P (I, I * ). The feature point position loss is described using local cosine loss L cos (I, I * ) (see equation 6) and peak loss L peaky (I, I * ) (see equation 7), L cos (I, I * ) and L peaky (I, I * ) are used to measure the consistency of feature point position and the peak distribution of local feature points, respectively. Let the After the homography matrix Homo transformation, we get I P and are divided into M non-overlapping blocks (see Figure 6 ), respectively, and the block size is SxS, where the relationship between the number of blocks and the block size is where W, H represent the width and height of the picture I, respectively. For I P and The column vectors formed by flattening the i'th block region are denoted as P[i'] and where i' ∈ {1, 2,..., N}.

[0028]

[0029] where ||·||2 represents the vector 2-norm, and the superscript T represents the transpose of the vector.

[0030]

[0031] where P[i'](j) represents the jth element of the column vector P[i'], and represents the maximum value of all elements of the column vector P[i']. denotes the average of all elements of column vector P[i'].

[0032] Finally, the feature point position loss L P (I,I * ) is obtained by balancing the two types of loss through the balance parameter λ.

[0033] L P (I,I * ) = L cos (I,I * ) + λL peaky (I,I * ) (8)

[0034] 4-2. Construct the descriptor loss L D (I,I * ). The descriptor loss is described using the hinge loss (see formula 9). Let the position (i, j) in the picture I correspond to the position (i * , j * ) in the augmented picture I * , and the descriptor vector corresponding to the position (i, j) in the vector map I D is f i,j , and the descriptor vector corresponding to the position (i * , j D ) in the vector map I * is f * i,j * . Let the distance between the position (i, j) in I and the position (k, l) in I * be g ij,kl , and by comparing the distance threshold t (t is a manually set parameter, and in the specific implementation below, the value of t is set to 9) (see formula 10), c ij,kl is obtained.

[0035]

[0036]

[0037] where δ, m p and m n are hyperparameters, representing weight coefficients, positive boundaries and negative boundaries respectively, and the superscript T represents transposition.

[0038] 5. Train the feature point detection and description network N.

[0039] 5-1. Implement step 1 to obtain the training data set D.

[0040] 5-2. For a picture I ∈ D in the training data set D, implement step 3 to obtain the augmented picture I *. Input picture I and augmented picture I * into the network N constructed in step 2, respectively, to obtain position response map I P and descriptor vector map I D . and descriptor vector map I

[0041] 5-3. For picture I and augmented picture I * , implement step 4 to calculate the loss function L(I, I * ) of network N. Update the training parameters of network N by Stochastic Gradient Descent (SGD) algorithm, wherein the learning rate of SGD algorithm is a1, and the momentum is a2.

[0042] 5-4. Traverse all pictures in the training data set D, implement steps 5-2 to 5-3, and complete one round of training.

[0043] 5-4. Implement step 5-4 for e rounds to obtain the trained feature point detection and description network N.

[0044] 6. Test: input test picture I t into the trained feature point detection and description network N to finally obtain a feature point set. For test picture I t , the size is W t × H t × C t , wherein W t , H t , and C t represent the width, height, and channel number of test picture I t , respectively.

[0045] 6-1. Input I t into the trained network N obtained in step 5 to output position response map I and descriptor vector map I with sizes of W t × H t and W t × H t × N F , respectively, wherein N F represents the channel number of descriptor vector map (the channel number of descriptor vector map is the same as the channel number of sub-feature map).

[0046] 6-2. Calculate the feature point position set P. For position response map I , perform non-maximum suppression processing to obtain the feature point position set P = {(x1, y1),..., (x np , y np)}, where np represents the number of feature points. Non-maximum suppression uses a preset window of size w×w. The slider moves upwards (w is a manually set parameter; in the detailed implementation later, w is set to 3), with a slider step size of s. The calculation... Location of local maximum value on the upper w×w window size Location of local maximum As the location of feature points, where (i t ,j t )∈{(1,1),(2,2),...,(np,np)}.

[0047] 6-3. Calculate the feature point set FP. For (x i″ ,y i″ )∈P, select descriptor vector graph Mid position (x) i″ ,y i″ The vector at position ) As position (x) i″ ,y i″ For each (i″,j″) = (1,1),...,(p,p), perform the descriptor vector selection operation on the descriptor vector at position (i″,j″) to obtain the feature point descriptor set.

[0048] This invention provides directional information with minimal computational cost through a direction decoding network. Building upon feature point location calculation, a direction decoding network is added to estimate the orientation of all feature point blocks in a single forward pass, significantly reducing storage and computational costs and providing directional information for feature points in descriptor calculation. Simultaneously, a rotation-equivariant convolutional network greatly reduces the computational cost of orientation vectors. By using this network, a single forward pass can obtain feature maps of various rotation angles of the image, significantly reducing computational load and improving feature map reusability compared to performing rotation and forward pass calculations for each feature point block separately. This invention combines feature point detection and description networks, using the detection process to guide the description process, and can reduce the number of dense descriptors through interpolation. Furthermore, the end-to-end approach increases the coupling between detection and description, reducing computational load. The entire network greatly facilitates the practical application of feature point detection and description algorithms in engineering. Attached Figure Description

[0049] Figure 1 This is a flowchart of the end-to-end rotation-robust feature point detection and description method of the present invention;

[0050] Figure 2 The structure diagram of the end-to-end feature point detection and description network N;

[0051] Figure 3 Fig. 1 is a structural diagram of the feature point detection network N P ;

[0052] Figure 4 Fig. 2 is a structural diagram of the feature point description network N D ;

[0053] Figure 5 Fig. 3 is a structural diagram of the feature point selection network N DS ;

[0054] Figure 6 Fig. 4 is a division method of the position response map;

[0055] Figure 7 Fig. 5 is a network training flowchart;

[0056] Figure 8 Fig. 6 is a network testing flowchart. DETAILED DESCRIPTION

[0057] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and the accompanying drawings and the accompanying drawings, so that those skilled in the art can implement the present application after reading the description. It should be understood that the embodiments described in the examples are only used to explain the present application and do not limit the present application.

[0058] Embodiment one

[0059] As shown in Figure 1 , the present application provides an end-to-end rotation robust feature point detection and description method, which specifically comprises the following steps:

[0060] 1. Constructing a training data set: using COCO 2017 training set as a training data set D. For any one picture I in the training data set D, the size is WxH, wherein W and H represent the width and height of the training picture I respectively. Input I into the network N constructed in step 2.

[0061] 2. Constructing an end-to-end feature point detection and description network N (see Figure 2 ): the network N is composed of a feature point detection network N P (see Figure 3 ) and a feature point description network N D (see Figure 4 ). N P is composed of an encoding network N PE and a decoding network N PD . Among them, N PD is composed of a position decoding network N PDP and a direction decoding network N PDO ; N D is composed of a rotation equivariant convolution network N DR and a selection network NDS cascade.

[0062] 2-1. Constructing feature point detection network N P (see Figure 3 ) N P is composed of an encoding network N PE and a decoding network N PD . The encoding network N PE is a full convolutional network, including 11 convolutional groups, each of which is composed of a convolutional layer, a batch normalization layer, and an activation layer. Among them, each convolutional layer in the convolutional group performs a 3x3 convolution operation with a step size of 1, the dilation size of the convolutional layer is [1, 1, 2, 2, 1, 1, 2, 2, 4, 8, 16] in turn, the padding size of the convolutional layer is [1, 1, 2, 2, 1, 1, 2, 2, 4, 8, 16] in turn, and the number of convolution kernels in the convolutional layer is [16, 16, 32, 32, 64, 64, 128, 128, 256, 256, 256] in turn. The picture I is input into the encoding network N PE , to obtain an encoded feature map E with a size of WxHx256 (the same as the W and H of the training picture size).

[0063] 2-2. Input the encoded feature map E into the position decoding network N PDP , to obtain a position response map I P with a size of WxH, where N PDP is composed of 1 convolutional layer with a kernel size of 1x1 and a number of 1.

[0064] 2-3. Input the encoded feature map E into the direction decoding network N PDO , to obtain a direction estimation map I O with a size of WxH, where N PDO is composed of 1 convolutional layer with a kernel size of 1x1 and a number of 1.

[0065] 2-4. Constructing feature point description network N D (see Figure 4 ) N D is composed of a rotation equivariant convolutional network N DR and a selection network N DS . The rotation equivariant convolutional network N DRconsists of 8 groups of rotation equivariant convolutions and 3 groups of rotation equivariant max-pooling down-sampling layers. Each group of rotation equivariant convolutions consists of a rotation equivariant convolution layer, a rotation equivariant batch normalization layer, and a rotation equivariant activation layer, and each two groups of convolutions are followed by a rotation equivariant max-pooling down-sampling layer. Each convolution layer is implemented by e2cnn, and the convolution layer performs a convolution operation with a kernel size of 3, a stride of 1, and a padding size of 1. The number of convolution kernels in the 8 groups of rotation equivariant convolutions is [32, 32, 64, 64, 128, 128, 256, 256] in turn. Each rotation equivariant max-pooling down-sampling layer performs a max-pooling operation with a window size of 2x2 and a stride of 2, and the output result is equivalent to a down-sampling with a coefficient of 2. The picture I is input into the N DR , and an output feature map F with a size of 36xW / 8xH / 8x256 is obtained. The k-th sub-feature map of F is denoted as F[k], where k e {1, 2,..., K}, and the spatial down-sampling rate ε of the sub-feature map F[k] relative to the input picture I is 8.

[0066] 2-5. For the direction estimation map I O obtained in step 2-3, the value of the position (i, j) is I O (i, j). The feature map F obtained in step 2-4 is input into a selection network N DS (the structure of which is shown in Figure 5 ), and the selection network N DS calculates I O (i, j) based on formula (1) according to the value I O (i, j), where k is the index number of the sub-feature map F corresponding to I O (i, j), and the sub-feature map F[k] is selected.

[0067]

[0068] where 360 represents 360 degrees, I O (i, j) represents the value at the position (i, j) in the direction estimation map I i,j , and represents the floor value; based on formula (2), the four position coordinates (p1, q1), (p2, q2), (p3, q3), and (p4, q4) corresponding to the position (i, j) in the sub-feature map F[k] are calculated according to the coordinate value of the position (i, j).

[0069] where

[0070] represents the floor value, and ε represents the spatial down-sampling rate of the sub-feature map F[k] relative to the input picture I; the vectors at the positions (p1, q1), (p2, q2), (p3, q3), and (p4, q4) in the sub-feature map F[k] are denoted as For the four vectors The vector

[0071]

[0072] For the vector The descriptor vector f at position (i, j) is obtained by L2 regularization of formula (4) i,j , with size 256 x 1.

[0073]

[0074] where ||·||2 denotes the vector two norm.

[0075] 2-6. Steps 2-5 are performed for all i = 1,... H, j = 1,... W, respectively, to obtain the descriptor vector map I D with size W x H x 256, where I D is the vector at position (i, j) in I i,j denotes the descriptor vector at position (i, j) in the input picture I.

[0076] 3. A Gaussian noise δ with mean 0 and variance 30 is added to the picture I to obtain the augmented picture I * with size W x H x 3, where W and H denote the width and height of the augmented picture I * , respectively. Let the homography matrix be the pixel position correspondence relationship between the original picture I and the overlapping region of the augmented picture I * . The augmented picture I * is input into the network N constructed in step 2 to obtain the position response map and the descriptor vector map with sizes W x H and W x H x 256, respectively.

[0077] 4. The loss function L(I, I * ) of the network N is constructed. The loss function L(I, I * ) is composed of the feature point position loss L P (I, I * ) and the descriptor loss L D (I, I * ):

[0078] L(I, I * ) = L P (I, I * ) + 0.001·L D (I, I * ) (5)

[0079] 4-1. Construction of feature point position loss L P (I,I * ). The feature point position loss is jointly described by the local cosine loss L cos (I,I * ) (see equation 6) and the peak loss L peaky (I,I * ) (see equation 7), L cos (I,I * ) and L peaky (I,I * ) are used to measure the consistency of feature point positions and the peak distribution of local feature points, respectively. Let the step 3 get After the homography matrix Homo transformation, we get Divide I P and into M non-overlapping blocks (see Figure 6 for the division method), and the block size is 8x8. The relationship between the number of blocks and the block size is where W and H represent the width and height of the picture I, respectively. For I P and The column vectors formed by flattening the i'th (i' = 1, 2,..., N) block region are denoted as P[i'] and

[0080]

[0081] where ||·||2 represents the vector 2-norm, and the superscript T represents the transpose of the vector.

[0082]

[0083] where P[i'] (j) represents the jth element of the column vector P[i'], and represents the maximum value of all elements of the column vector P[i']. represents the average value of all elements of the column vector P[i'].

[0084] Finally, by balancing the coefficients 0.1, the total loss of feature point position L P (I,I * ) is obtained.

[0085] L P (I,I * ) = L cos (I,I * ) + 0.1 x L peaky (I,I * ) (8)

[0086] 4-2. Constructing the descriptor loss L D (I,I * The descriptor loss is calculated using the hinge loss (see Equation 9). Let position (i,j) in image slice I correspond to augmented image I. * The position in (i) * ,j * ), Descriptor Vector Graph I D The descriptor vector corresponding to position (i,j) is f i,j Vector graph Mid position (i * ,j * The corresponding descriptor vector is f. i,j * Let I denote the position (i,j) and I. * The distance between positions (k, l) in the middle is g ij,kl By comparing with a distance threshold of 9 (see Formula 10), c is obtained. ij,kl .

[0087]

[0088]

[0089] 5. Training the feature point detection and description network N (training flowchart see...) Figure 7 ).

[0090] 5-1. Implement step 1 to obtain the training dataset D.

[0091] 5-2. For an image I∈D in the training dataset D, perform step 3 to obtain the augmented image I. * Combine image I and augmented image I * The inputs are respectively fed into the network N constructed in step 2 to obtain the position response map I. P Descriptor Vector Graph I D Location response diagram and descriptor vector graph

[0092] 5-3. Regarding Image I and Augmented Image I * Step 4 is performed to calculate the loss function L(I,I) of network N. * The training parameters of network N are updated using the Stochastic Gradient Descent (SGD) algorithm, where the learning rate in the SGD algorithm is 10. -4 The learning rate decreases by a factor of 10 every 10 rounds, the weight decay coefficient is 0.0005, and the momentum is 0.9.

[0093] 5-4. Traverse all images in the training dataset D, and implement steps 5-2 to 5-3 to complete one round of training.

[0094] 5-5. Implement steps 5-4 for a total of 100 rounds to obtain the trained feature point detection and description network N.

[0095] 6. Testing: For test image I t Size is W t ×H t ×3, where W t H t These represent test images I and I respectively. t Width and height. (See test flowchart) Figure 8 )

[0096] 6-1.Change I t Input the trained network N obtained in step 5 and output the position response map. and descriptor vector graph Sizes are W t ×H t and W t ×H t ×256.

[0097] 6-2. Calculate the set P of feature point locations. For the location response map... Non-maximum suppression is performed to obtain the feature point location set P = {(x1, y1), ..., (x p ,y p )}, where p represents the number of feature points. Non-maximum suppression uses a 3×3 window in Slide upwards with a step size of 1, calculate Location of local maximum value in the upper 3x3 window size Location of local maximum As the detected feature point location (in) Above, within a 3x3 window, there are 9 numbers in total. The location of the maximum value among these 9 numbers is the location of the detected feature point. (i t ,j t )∈{(1,1),(2,2),...,(np,np)}.

[0098] 6-3. Calculate the feature point set FP. For (x i″ ,y i″ )∈P, select descriptor vector graph Mid position (x) i″ ,y i″ The vector at position ) As position (x) i″ ,y i″) the descriptor vector at (i", j") is obtained by performing the select descriptor vector operation on the descriptor vector at (i", j") and the descriptor vector at (i", j") for all (i", j") = (1, 1),..., (p, p). The set of feature points is obtained by performing the select descriptor vector operation on the descriptor vector at (i", j") for all (i", j") = (1, 1),..., (p, p). The final result of the task is to obtain the set of feature points, which is the basis for other image tasks, such as three-dimensional reconstruction.

[0099] The above-described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

Claims

1. A method for end-to-end rotationally robust feature point detection and description, characterized in that, The method includes the following steps: S1) Constructing the training dataset: Collect a series of image data to construct the training dataset D. For any image in the training dataset D... Size is Where W, H, and C represent training images respectively. Width, height, and number of channels; S2) Construct an end-to-end feature point detection and description network , the image Input to build the network From the image, The descriptor vector, where the network By feature point detection network and feature point description network Parallel connection; step S2) specifically includes: S21) Construct a feature point detection network , By coding network and decoding network Cascaded configuration, Location decoding network and direction decoding network Parallel connection, coding network It is a fully convolutional network, including Each convolutional group consists of a concatenated convolutional layer, a batch regularization layer, and an activation layer; the image... Input encoding network , thus obtaining the encoded feature map Its size is ,in , , These represent the encoded feature maps respectively. Width, height, and number of channels; S22) Encode the feature map Input position decoding network In the middle, the position response map is obtained. Its size is ,in Depend on It consists of a series of convolutional layers. Middle position value at Image Middle position The response values ​​of the feature points at the location, where , ; S23) Encode the feature map Input direction decoding network In the middle, the direction estimation map is obtained. Its size is ,in Depend on It consists of a series of convolutional layers. Middle position value at Image Middle position The direction estimate at, where , ; S24) Construct a feature point description network , Rotationally variable convolutional networks and choosing a network Cascaded, rotationally variable convolutional networks Depend on A rotating equivariant convolution group and The image is composed of a rotationally isovariant max-pooling downsampling layer, wherein each rotationally isovariant convolutional group consists of a cascaded rotationally isovariant convolutional layer, a rotationally isovariant batch regularization layer, and a rotationally isovariant activation layer; enter In the middle, output feature map Size is ,in, Depend on It consists of several sub-feature maps of the same size, with each sub-feature map having a size of [missing value]. , , , Let the width, height, and number of channels of the sub-feature map be represented respectively; Let be... The The individual feature map is ,in Sub-feature map Compared to the input image Spatial downsampling rate ; S25) For the orientation estimation map obtained in step S23) ,Location The value is The feature map obtained in step S24) Input Select Network In the middle, select network. According to the value Calculate based on formula (2) For the sub-feature map index k in F, select sub-feature map F[k]. （2） 360 represents 360 degrees. Direction estimation diagram Middle position Value, This indicates rounding down; based on formula (3), according to position The coordinates are used to calculate the position. In sub-feature map The four corresponding position coordinates , , , ; （3） in, Indicates rounding down. Sub-feature map Compared to the input image Spatial downsampling rate; sub-feature map Middle position , , , The vectors at each point are denoted as follows: , , , For these four vectors , , , The vector is obtained by interpolation using formula (4). ; （4） For vectors L2 regularization is performed using formula (5) to obtain the position. Descriptor vector at location Its size is , （5） in, Represents the 2-norm of a vector; S26) for all , Perform step S25 respectively to obtain the descriptor vector graph. Size is ,in, Middle position The vector at is , indicating that an image is being input. Middle position The descriptor vector at the location; S3) Image Add random noise Random translation, rotation, and color changes are performed to obtain the augmented image. Size is ,in , , These represent augmented images. Width, height, and number of channels; Let the homography matrix be... For image I and augmented image The pixel position correspondence in the overlapping areas. Represents the real number field. Representing all 3×3 real matrices, this will augment the image. Input the network constructed in step S2) In the middle, the position response map is obtained. and descriptor vector graph The sizes are respectively and ; S4) Building the network loss function loss function Loss due to feature point location and descriptor loss composition: （1） Among them, hyperparameters The ratio used to balance feature point location loss and descriptor loss; S5) Training Feature Point Detection and Description Network Traverse all images in the training dataset D, performing steps S2) to S4) to complete one round of training; when the preset... After one round of training, the trained feature point detection and description network is obtained. ; S6) Test Feature Point Detection and Description Network Test image Input the trained feature point detection and description network Finally, a set of feature points is obtained, including the test image. The size is , , , These represent the test images. Width, height, and number of channels.

2. The end-to-end rotationally robust feature point detection and description method according to claim 1, characterized in that, Step S4) specifically includes: S41) Construct feature point location loss The feature point location loss uses the local cosine loss in formula (6). And the peak loss in formula (7) Describe it. and These are used to measure the consistency of feature point locations and the local peak distribution of feature points, respectively. Let's assume that the result obtained in step S3) After homography matrix After transformation, we get ,Will and Divided into non-overlapping Block, block size is The relationship between the number of blocks and the block size is as follows: ,for and , No. The column vectors formed by flattening the block region are denoted as follows: and ,in , （6） in This represents the L2 norm of a vector, and the superscript T indicates the transpose of the vector. （7） in, Represents column vectors The One element, Represents column vectors The maximum value of all elements. Represents column vectors The average of all elements; Finally, by balancing parameters By balancing the two types of losses, we obtain the feature point location loss. , （8） S42) Construction Descriptor Loss The descriptor loss is described using the hinge loss in formula (9), assuming the image... The position in the middle Corresponding augmented image The position in the middle Descriptor Vector Graph Middle position The corresponding descriptor vector is Vector graph Middle position The corresponding descriptor vector is ,remember The position in the middle and The position in the middle The distance between them is By setting a preset distance threshold By comparison, we obtain the result in formula (10). , （9） （10） in, , and These are hyperparameters, representing the weighting coefficient, positive boundary, and negative boundary, respectively. The superscript T indicates transpose.

3. The end-to-end rotationally robust feature point detection and description method according to claim 2, characterized in that, Step S6) specifically includes: S61) Test image Input the trained feature point detection and description network obtained in step S5) In the middle, output position response diagram and descriptor vector graph The sizes are respectively and ,in This represents the number of channels in the descriptor vector graph; S62) Calculate the set of feature point locations For the location response diagram Non-maximum suppression is performed to obtain the set of feature point locations. ,in This indicates the number of feature points; non-maximum suppression uses a preset value. Window size in Slide upwards, with a step size of [missing information]. ,calculate superior Location of local maximum value on window size Position the local maximum value As the location of feature points, where , , ; S63) Calculate the set of feature points for Select descriptor vector graph Middle position Vector at As a position The descriptor vector at the location, for all Perform the descriptor vector selection operation separately to obtain the feature point descriptor set. .

Images