Camera Pose Estimation Method, Device, Electronic Device and Storage Medium

Abstract

The present invention provides a camera pose estimation method, device, electronic device and storage medium, wherein the method includes: determining a first image and a second image; performing feature extraction on the first image and the second image respectively, and determining a sparse feature matching set between the first image and the second image based on the first sparse features and the second sparse features obtained by the feature extraction; performing dense feature matching on the first image and the second image based on a dense matching network to obtain a dense feature matching set between the first image and the second image; determining the camera pose between the first image and the second image based on the sparse feature matching set and the dense feature matching set, thereby solving the problem of poor accuracy and reliability of camera pose estimation in traditional solutions under drastic changes in lighting and viewing angle. Through feature matching and feature fusion, sufficient correct matches are established between the two images, thereby achieving a dual improvement in the precision and accuracy of camera pose estimation.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a camera pose estimation method, device, electronic device and storage medium. Background Art

[0002] In the field of computer vision, camera pose estimation is particularly important, especially in visual SLAM (Simultaneous Localization and Mapping) tasks and visual positioning tasks. Camera pose estimation methods play a vital role, and image feature point matching is a key step in image-based camera pose estimation methods.

[0003] In current visual SLAM, the optical flow method is usually used to match key points of two images. However, the premise of using the optical flow method is that the relative motion between the two frames of images is small. In other words, optical flow estimation is only applicable to the case where the relative motion between the two frames of images is not large. For situations where the lighting and perspective change drastically, optical flow estimation is not applicable. It cannot establish sufficient correct matches for the image-based camera pose estimation task, which greatly reduces the accuracy and reliability of the estimated camera pose. Summary of the Invention

[0004] The present invention provides a camera pose estimation method, device, electronic device and storage medium to address the defects in the prior art of being unable to establish sufficient correct matches for the camera pose estimation task under conditions of drastic changes in illumination and viewing angle, resulting in poor accuracy and reliability of the estimated camera pose.

[0005] The present invention provides a camera pose estimation method, comprising:

[0006] determining a first image and a second image;

[0007] performing feature extraction on the first image and the second image respectively, and determining a sparse feature matching set between the first image and the second image based on first sparse features and second sparse features obtained by the feature extraction;

[0008] performing dense feature matching on the first image and the second image based on a dense matching network to obtain a dense feature matching set between the first image and the second image;

[0009] determining a camera pose between the first image and the second image based on the sparse feature matching set and the dense feature matching set;

[0010] The dense matching network is trained based on a sample first image, a sample second image, and a sample dense feature matching set between the sample first image and the sample second image.

[0011] According to a camera pose estimation method provided by the present invention, determining the camera pose between the first image and the second image based on the sparse feature matching set and the dense feature matching set includes:

[0012] fusing the sparse feature matching set and the dense feature matching set to obtain an overall feature matching set;

[0013] False matches are eliminated from the overall feature matching set, and a camera pose between the first image and the second image is determined based on the overall feature matching set after false matches are eliminated.

[0014] According to a camera pose estimation method provided by the present invention, the dense matching network includes a multi-scale feature extraction layer, a similarity judgment layer and a dense feature matching layer;

[0015] The multi-scale feature extraction layer is used to perform multi-scale feature extraction on the first image and the second image respectively to obtain a first feature map set of the first image and a second feature map set of the second image;

[0016] The similarity decision layer is used to determine a first similarity between a first scale feature map in the first feature map set and a first scale feature map in the second feature map set;

[0017] The dense feature matching layer is used to determine a dense feature matching set between the first image and the second image based on the first similarity and a second similarity, where the second similarity is the similarity between the second scale feature map in the first feature map set and the second scale feature map in the second feature map set.

[0018] According to a camera pose estimation method provided by the present invention, the first similarity is determined based on the following steps:

[0019] Determining an initial similarity between a first scale feature map in the first feature map set and a first scale feature map in the second feature map set;

[0020] Performing data cleaning on the initial similarity based on a preset cleaning method to obtain a first similarity between a first scale feature map in the first feature map set and a first scale feature map in the second feature map set;

[0021] The preset cleaning method includes a nearest neighbor filtering method and / or a neighborhood consistency filtering method.

[0022] According to a camera pose estimation method provided by the present invention, determining a dense feature matching set between the first image and the second image based on the first similarity and the second similarity includes:

[0023] Determining, based on the second similarity, a second matching score map between each descriptor vector in the second scale feature map in the first feature map set and the second scale feature map in the second feature map set;

[0024] Based on the first similarity, determining, by bilinear interpolation, a first matching score map between each descriptor vector in the second scale feature map in the first feature map set and the first scale feature map in the second feature map set;

[0025] Based on the first matching score map and the second matching score map, a dense feature matching set between the first image and the second image is determined.

[0026] According to a camera pose estimation method provided by the present invention, determining, based on the first similarity, by bilinear interpolation, a first matching score map between each descriptor vector in the second scale feature map in the first feature map set and the first scale feature map in the second feature map set, includes:

[0027] Based on the first similarity, obtaining, by bilinear interpolation, a description similarity between each descriptor vector in the second scale feature map in the first feature map set and the first scale feature map in the second feature map set;

[0028] Based on the description similarity, a first matching score map is determined between each descriptor vector in the second scale feature map in the first feature map set and the first scale feature map in the second feature map set.

[0029] According to a camera pose estimation method provided by the present invention, determining a dense feature matching set between the first image and the second image based on the first matching score map and the second matching score map includes:

[0030] Upsampling the first matching score map, and determining a target matching score map based on the upsampled first matching score map and the second matching score map;

[0031] Based on the object matching score map, a set of dense feature matches between the first image and the second image is determined.

[0032] The present invention also provides a camera pose estimation device, comprising:

[0033] an image determining unit, configured to determine a first image and a second image;

[0034] a sparse feature matching unit, configured to perform feature extraction on the first image and the second image respectively, and determine a sparse feature matching set between the first image and the second image based on the first sparse features and the second sparse features obtained by the feature extraction;

[0035] a dense feature matching unit, configured to perform dense feature matching on the first image and the second image based on a dense matching network to obtain a dense feature matching set between the first image and the second image; the dense matching network is trained based on a sample first image, a sample second image, and a sample dense feature matching set between the sample first image and the sample second image;

[0036] A camera pose estimation unit is configured to determine a camera pose between the first image and the second image based on the sparse feature matching set and the dense feature matching set.

[0037] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the camera pose estimation method as described above is implemented.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described camera pose estimation methods.

[0039] The camera pose estimation method, device, electronic device and storage medium provided by the present invention can fuse the sparse feature matching set obtained by sparse feature matching with the dense feature matching set obtained by dense feature matching, so that the overall feature matching set obtained by fusion can more completely reflect the matching relationship between the first image and the second image, thereby making the camera pose solution process based on this overall feature matching set more accurate, and solve the problem in traditional solutions that in the case of drastic changes in lighting and viewing angle, and weak texture, it is impossible to establish sufficient correct matches for the camera pose estimation task, resulting in poor accuracy and reliability of the estimated camera pose. Through sparse feature matching and dense feature matching, sufficient correct matches can be established between the two images, thereby achieving a dual improvement in the precision and accuracy of camera pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 1 is a flow chart of the camera pose estimation method provided by the present invention;

[0042] Figure 2 It is a schematic diagram of the structure of the sparse feature extraction network provided by the present invention;

[0043] Figure 3 is a schematic structural diagram of a dense matching network provided by the present invention;

[0044] Figure 4 This is the overall framework diagram of the camera pose estimation method provided by the present invention;

[0045] Figure 5 Schematic diagram of the camera pose estimation method provided by the present invention;

[0046] Figure 6 Schematic diagram of camera pose estimation for a large jitter video sequence provided by the present invention;

[0047] Figure 7 1 is a schematic structural diagram of a camera pose estimation device provided by the present invention;

[0048] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0050] As one of the commonly used methods in the field of computer vision, camera pose estimation plays a vital role in visual SLAM tasks and visual positioning tasks, and image feature point matching is a key step in image-based camera pose estimation methods.

[0051] Most current visual SLAMs use the optical flow method to match key points between two images. However, the premise of using the optical flow method is that the relative motion between the two frames is small. In other words, optical flow estimation is only applicable when the relative motion between the two frames is not large. For situations where the illumination and perspective changes between the two frames are large, optical flow estimation is not applicable, and it cannot establish sufficient correct matches for image-based camera pose estimation tasks.

[0052] Furthermore, in the case of drastic changes in lighting and perspective, feature matching methods can be used to track key points. However, current feature matching methods have obvious defects in the case of weak textures. For example, SuperGlue's sparse feature matching method is also unable to establish sufficient correct matches, resulting in low accuracy of camera pose.

[0053] The present invention provides a camera pose estimation method, which aims to establish sufficient correct matches between two images through sparse feature matching and dense feature matching in the case of drastic changes in illumination and perspective, as well as weak texture, thereby achieving a dual improvement in the precision and accuracy of camera pose estimation. Figure 1 : is a flow chart of the camera pose estimation method provided by the present invention, such as Figure 1 As shown, the method includes:

[0054] Step 110, determining a first image and a second image;

[0055] Specifically, before performing camera pose estimation, it is first necessary to determine two images, that is, an image pair, where the two images in the image pair are the first image and the second image, and there is a correspondence between the partial area in the first image and the partial area in the second image, that is, the partial area in the first image and the partial area in the second image actually refer to the same thing, or contain the same content.

[0056] The image pair here can be one or more. In the case of multiple image pairs, camera pose estimation needs to be performed for each image pair, that is, the camera pose between the first image and the second image in each image pair needs to be determined.

[0057] It should be noted that the first image and the second image in each image pair may be images of various types, such as landscape images, character images, or object images, which is not specifically limited in the embodiment of the present invention.

[0058] Step 120: performing feature extraction on the first image and the second image respectively, and determining a sparse feature matching set between the first image and the second image based on the first sparse features and the second sparse features obtained by the feature extraction;

[0059] Specifically, in step 110, based on the first image and the second image having been obtained, step 120 may be executed to perform feature extraction on the first image and the second image respectively, and determine a sparse feature matching set between the first image and the second image based on the first sparse features and the second sparse features obtained by the feature extraction. This process specifically includes the following steps:

[0060] First, feature extraction can be performed on the first image and the second image respectively, and sparse local feature points in the first image and the second image are extracted, thereby obtaining sparse features of the first image and sparse features of the second image, which are named first sparse features and second sparse features respectively;

[0061] Here, the feature extraction process for the first and second images can be implemented with the help of a sparse feature extraction network. That is, the first and second images can be input into the sparse feature extraction network to obtain first and second sparse features output by the sparse feature extraction network. Before inputting the first and second images into the sparse feature extraction network, a sample image and sample sparse features of the sample image can be used to pre-train the sparse feature extraction network.

[0062] Then, sparse feature matching can be performed on the first image and the second image based on the first sparse feature and the second sparse feature, so as to obtain a sparse feature matching set between the first image and the second image. Specifically, this process can be to establish a large number of sparse feature matches between the first image and the second image using a bidirectional nearest neighbor matching method based on the first sparse feature and the second sparse feature, and then apply a ratio test method to eliminate incorrect matches, and finally obtain a matching set, namely a sparse feature matching set.

[0063] Step 130: performing dense feature matching on the first image and the second image based on a dense matching network to obtain a dense feature matching set between the first image and the second image, wherein the dense matching network is trained based on a sample first image, a sample second image, and a sample dense feature matching set between the sample first image and the sample second image;

[0064] Specifically, after determining the first image and the second image in step 110, step 130 may be executed to perform dense feature matching on the first image and the second image using a dense matching network to obtain a dense feature matching set between the first image and the second image. The specific process includes the following steps:

[0065] First, the first image and the second image are input into a dense matching network. The dense matching network can take into account the global and local context information of the input image, and perform pixel-by-pixel matching on the input first image and the second image, and finally obtain a dense feature matching set between the first image and the second image output by the dense matching network. The specific process can be: first, multi-scale feature extraction is performed on the first image and the second image respectively, so as to obtain feature maps of the first image and the second image at different scales; then, according to the similarity between the feature maps of the first image and the second image at the corresponding scale, a matching score map at the corresponding scale is constructed; thereafter, the dense matching network can obtain the final matching score map by fusing the matching score maps at different resolutions corresponding to different scales, and on this basis, the dense feature matching set between the first image and the second image can be determined.

[0066] Before inputting the first image and the second image into the dense matching network, the MegaDepth dataset can be used to pre-train the dense matching network. The dense matching network training process includes the following steps: first, a large number of sample image pairs are obtained from the MegaDepth dataset, and a set of sample dense feature matches between the sample first image and the sample second image in the sample image pair is determined; then, based on the sample image pairs and the set of sample dense feature matches between the sample first image and the sample second image in the sample image pair, an initial dense matching network is trained to obtain a trained dense matching network.

[0067] Step 140 : Determine a camera pose between the first image and the second image based on the sparse feature matching set and the dense feature matching set.

[0068] Specifically, after steps 120 and 130, when a sparse feature matching set and a dense feature matching set are obtained between the first image and the second image, respectively, step 140 may be executed to determine the camera pose between the first image and the second image based on the sparse feature matching set and the dense feature matching set. This process specifically includes the following steps:

[0069] First, the sparse feature matching set between the first image and the second image and the dense feature matching set between the first image and the second image can be fused to obtain an overall feature matching set between the first image and the second image, which includes sparse feature matching and dense feature matching obtained by different feature matching methods;

[0070] Then, considering that there may be matching errors in the above feature matching process, before solving the camera pose based on this overall feature matching set, it is necessary to eliminate the error matches in the overall feature matching set, that is, to eliminate the error matches in the overall feature matching set to ensure the accuracy of the feature matching set;

[0071] After that, the overall feature matching set after false matching is eliminated can be applied to solve the camera pose between the first image and the second image, and finally the camera pose between the first image and the second image can be obtained.

[0072] The camera pose estimation method provided by the present invention can make the overall feature matching set obtained by fusion of the sparse feature matching set obtained by sparse feature matching and the dense feature matching set obtained by dense feature matching more completely reflect the matching relationship between the first image and the second image, thereby making the camera pose solution process based on this overall feature matching set more accurate. It solves the problem in which the traditional scheme cannot establish sufficient correct matches for the camera pose estimation task in the case of drastic changes in lighting and viewing angle, and weak texture, resulting in poor accuracy and reliability of the estimated camera pose. Through sparse feature matching and dense feature matching, sufficient correct matches can be established between the two images, thereby achieving a dual improvement in the precision and accuracy of camera pose estimation.

[0073] Based on the above embodiment, in step 120, the process of extracting features from the first image and the second image respectively can be implemented with the help of a sparse feature extraction network. Figure 2 It is a schematic diagram of the structure of the sparse feature extraction network provided by the present invention, such as Figure 2 As shown in the figure, the sparse feature extraction network includes multiple convolutional layers, and at the end of the sparse feature extraction network there are two heads, namely Keypoint Head and Descriptor Head.

[0074] In the sparse feature extraction network, the network structure of the convolution layer can be set according to actual needs. Preferably, in the embodiment of the present invention, the convolution layer is divided into three categories according to its function, one of which is:

[0075] Convolution_BatchNorm_ReLU(24, 3×3, 2)

[0076] Convolution_BatchNorm_ReLU(256, 3×3, 1)

[0077] Convolution_BatchNorm_ReLU(128, 3×3, 1)

[0078] Taking "Convolution_BatchNorm_ReLU(24, 3×3, 2)" as an example, the parameters are explained:

[0079] Among them, Convolution represents convolution, BatchNorm (Batch Normalization) represents batch normalization, ReLU (Rectified Linear Unit) represents activation, Convolution_BatchNorm_ReLU represents convolution + batch normalization + activation, 24 is the channel, 3×3 is the kernel size, and 2 is the stride.

[0080] The second is:

[0081] MobileNet_Expanded_Block(16,3×3,1)

[0082] MobileNet_Expanded_Block(24, 3×3, 2)

[0083] MobileNet_Expanded_Block(24, 3×3, 1)

[0084] MobileNet_Expanded_Block(24, 3×3, 2)

[0085] MobileNet_Expanded_Block(48, 3×3, 1)

[0086] MobileNet_Expanded_Block(96, 3×3, 1)

[0087] Taking "MobileNet_Expanded_Block(16, 3×3, 1)" as an example, the following parameters are explained:

[0088] Among them, MobileNet represents a lightweight network, Expanded represents expansion, MobileNet_Expanded_Block represents a lightweight and expanded network framework, 16 is the channel (channels), 3×3 is the size of the convolution kernel (kernelsize), and 1 is the stride (stride).

[0089] The third is:

[0090] Convolution(256, 1×1, 1)

[0091] Convolution(65, 1×1, 1)

[0092] Taking "Convolution(256, 1×1, 1)" as an example, the following parameters are explained:

[0093] Among them, 256 is the channel, 1×1 is the kernel size, and 1 is the stride.

[0094] In the sparse feature extraction network, the Keypoint Head at the end of the network is used to output the keypoint heatmap (KeypointHeatmap), and the Descriptor Head is used to output the local descriptors (Local Descriptors).

[0095] In the Keypoint Head, the feature map size before the Softmax operation is (W / 8 × H / 8), where W and H represent the width and height of the input image, respectively. This feature map has 65 channels. After the Softmax operation on the channel dimension, each of the first 64 channels represents the probability that each pixel in the 8×8 region of the input image is a keypoint, while the remaining channel represents the probability that there is no keypoint in the 8×8 region. A Depth-to-Space operation is then performed to obtain a keypoint heatmap.

[0096] Furthermore, based on the key point heat map, the non-maximum suppression (NMS) algorithm can be used to determine the key points in the first image and the second image. Each key point can only be matched with the key point of the other image if it is assigned a descriptor. Therefore, based on the key points in the first image and the second image, combined with the dense descriptor of size (W / 8×H / 8) output by the Descriptor Head, key point matching can be performed between the first image and the second image, that is, sparse feature matching is performed on the first image and the second image, thereby obtaining a sparse feature matching set between the first image and the second image; the dense descriptor here is the descriptor of each pixel in the image.

[0097] In the embodiment of the present invention, the process of extracting key points using the non-maximum suppression algorithm specifically includes the following steps:

[0098] First, a convolution kernel is constructed. The size of the convolution kernel is the same as that of the non-maximum suppression window, which is k×k. The convolution kernel has k×k channels, each of which is a k×k square matrix. In the i×jth (i and j are different, it is k / / 2, / / means rounding, that is, retaining On the integer part of the (int) channel, except for the i×j-th element which is 1, the rest of the elements are 0; and the square matrix on the (k / / 2)×(k / / 2)-th channel is a zero matrix;

[0099] Then, the convolution kernel is used to perform a convolution operation on the key point heat map to obtain a three-dimensional tensor with a channel number of k×k;

[0100] Then, in the channel dimension, the maximum value of the three-dimensional tensor is calculated to obtain a two-dimensional tensor with the same size as the key point heat map; and the value of the position (i, j) in the two-dimensional tensor is the maximum value of the k×k neighborhood around the position (i, j) in the key point heat map, excluding the position (i, j);

[0101] After that, the corresponding elements of the two-dimensional tensor and the key point heat map can be compared; if the value of the key point heat map at a certain position is larger, the value of that position is retained; otherwise, the value of that position is set to zero, so that the coordinates of the key point after non-maximum suppression can be obtained.

[0102] Based on the above embodiment, step 140 includes:

[0103] The sparse feature matching set and the dense feature matching set are fused to obtain the overall feature matching set;

[0104] False matches are eliminated from the overall feature matching set, and a camera pose between the first image and the second image is determined based on the overall feature matching set after false matches are eliminated.

[0105] Specifically, in step 140, the process of determining the camera pose between the first image and the second image based on the sparse feature matching set and the dense feature matching set between the first image and the second image includes the following steps:

[0106] First, a sparse feature matching set between the first image and the second image, and a dense feature matching set between the first image and the second image, obtained through different feature matching processes, can be fused to obtain an overall feature matching set between the first image and the second image, which includes sparse feature matching and dense feature matching obtained through different feature matching methods.

[0107] Here, in the case of fusing the above two images, dense feature matching can make up for the features lost in the sparse feature matching process, thereby ensuring that the overall feature matching set obtained can more completely reflect the matching relationship between the first image and the second image. The fusion method of the two images can be splicing, addition, weighted fusion, etc., which is not specifically limited in the embodiment of the present invention.

[0108] Then, considering that there may be matching errors in the process of performing the sparse feature matching and the dense feature matching, the wrong matches can be eliminated, that is, the wrong matches can be eliminated for the overall feature matching set to ensure the accuracy of the overall feature matching set;

[0109] After that, the camera pose estimation can be performed based on the overall feature matching set after the false matches are eliminated, so as to obtain the camera pose between the first image and the second image, that is, the overall feature matching set after the false matches are eliminated is applied to solve the camera pose between the first image and the second image.

[0110] Based on the above embodiment, the dense matching network includes a multi-scale feature extraction layer, a similarity judgment layer, and a dense feature matching layer;

[0111] The multi-scale feature extraction layer is used to perform multi-scale feature extraction on the first image and the second image respectively to obtain a first feature map set of the first image and a second feature map set of the second image;

[0112] The similarity decision layer is used to determine a first similarity between a first scale feature map in the first feature map set and a first scale feature map in the second feature map set;

[0113] The dense feature matching layer is used to determine a dense feature matching set between the first image and the second image based on a first similarity and a second similarity, where the second similarity is the similarity between the second scale feature map in the first feature map set and the second scale feature map in the second feature map set.

[0114] Specifically, the dense matching network in step 130 includes three modules, each of which can be regarded as a layer, namely a multi-scale feature extraction layer, a similarity judgment layer, and a dense feature matching layer.

[0115] The multi-scale feature extraction layer is used to perform feature extraction on the first image and the second image at different scales, thereby obtaining feature maps of the first image and the second image at different scales. In the embodiment of the present invention, a dual-scale feature extraction layer is actually used to perform feature extraction on the first image and the second image, respectively, to obtain a first feature map set of the first image and a second feature map set of the second image.

[0116] Here, the first feature map set includes the first scale feature map and the second scale feature map of the first image. Correspondingly, the second feature map set includes the first scale feature map and the second scale feature map of the second image. In other words, the first scale feature map and the second scale feature map obtained by dual-scale feature extraction can constitute the corresponding feature map set.

[0117] It is worth noting that in the embodiment of the present invention, the dual-scale feature extraction layer is essentially a convolutional neural network based on the feature pyramid. The backbone network used in the convolutional neural network is ResNet34 up to the conv4_6 (Convolution4 to 6) layer, that is, ResNet34 pre-trained on the ImageNet dataset is used as the feature extractor.

[0118] It should be noted that during the training of the dual-scale feature extraction layer, the ResNet34 network weights need to be fixed and the feature pyramid parameters are trainable parameters.

[0119] The similarity decision layer is used to determine the first similarity between the first scale feature map of the first image and the first scale feature map of the second image. Specifically, the similarity decision layer uses a correlation algorithm to calculate the similarity between all descriptor vectors in the first scale feature map of the input first image and all descriptor vectors in the first scale feature map of the input second image, and obtains the similarity between the two first scale feature maps. Since the similarity obtained at this time is a similarity without data cleaning, its accuracy is not high, so it is called the initial similarity. Subsequently, the initial similarity can be cleaned to increase the similarity of correct matches and reduce the similarity of incorrect matches to ensure its accuracy, and finally obtain the first similarity between the first scale feature map of the first image and the first scale feature map of the second image.

[0120] It should be noted that the data cleaning process of the initial similarity can be performed according to the nearest neighbor filtering method and / or the neighborhood consistency filtering method, that is, the nearest neighbor filtering method and / or the neighborhood consistency filtering method can be used to clean the data of the initial similarity to obtain the first similarity.

[0121] The dense feature matching layer includes two branches, namely the first matching branch and the second matching branch; wherein the second matching branch is used to calculate the similarity between all descriptor vectors in the second scale feature map of the first image and the second scale feature map of the second image using a correlation algorithm based on the second scale feature map of the input first image and the second scale feature map of the input second image. The similarity is the second similarity between the second scale feature map of the first image and the second scale feature map of the second image.

[0122] Furthermore, the second matching branch is used to obtain the similarity between any descriptor vector in the second scale feature map of the first image and all descriptor vectors in the second scale feature map of the second image from the second similarity, and construct a matching score map between the descriptor vector and the second scale feature map of the second image based on this similarity. By repeating this process, a second matching score map between each descriptor vector in the second scale feature map of the first image and the second scale feature map of the second image can be obtained.

[0123] The first matching branch is used to construct a first matching score map between each descriptor vector in the second scale feature map of the first image and the first scale feature map of the second image based on the first similarity. The specific process can be that since the first scale feature map and the second scale feature map of the first image are corresponding, the first scale feature map and the second scale feature map of the second image are also corresponding. Therefore, a bilinear interpolation method can be used to obtain the similarity between any descriptor vector in the second scale feature map of the first image and all descriptor vectors in the first scale feature map of the second image from the first similarity, and construct a matching score map between the descriptor vector and the first scale feature map of the second image based on this similarity. By repeating this process, the first matching score map between each descriptor vector in the second scale feature map of the first image and the first scale feature map of the second image can be obtained.

[0124] Furthermore, the dense feature matching layer is used to determine a target matching score map between the first image and the second image based on the first matching score map and the second matching score map, and to determine a dense feature matching set between the first image and the second image based on the target matching score map.

[0125] In addition, it should be noted that the dense matching network is trained using the MegaDepth dataset, and for each sample image pair in the MegaDepth dataset, 128 sets of corresponding points are selected for network training.

[0126] Based on the above embodiment, the loss function used in the dense matching network training process is:

[0127]

[0128] Among them, S AB ∈R N×M , indicating I A The N query points in I B of The matching probability of the corresponding pixel; I A is the first image, I B For the second image, Represents the second scale feature map of the second image, M represents The number of elements of The number of descriptor vectors in ; For S AB The corresponding Ground Truth, S BA Indicates I B The N query points in I A of The matching probability of the corresponding pixel, represents the second scale feature map of the first image, For S BA Corresponding GroundTruth.

[0129] In an embodiment of the present invention, an orthogonal loss function can also be used as a regularization term to enable the dense matching network to focus on learning the one-to-one matching relationship between the pixels of the two images. The formula of the orthogonal loss function is shown as follows:

[0130]

[0131] in, Indicates S AB The transpose of express The transpose of Then S BA The transpose of for The transpose of .

[0132] During the training of the dense matching network, the overall loss function can be expressed as:

[0133] L=L k +λL O

[0134] Among them, λ is the weight.

[0135] Based on the above embodiment, the first similarity is determined based on the following steps:

[0136] Determining an initial similarity between a first scale feature map in the first feature map set and a first scale feature map in the second feature map set;

[0137] Based on a preset cleaning method, data cleaning is performed on the initial similarity to obtain a first similarity between the first scale feature map in the first feature map set and the first scale feature map in the second feature map set;

[0138] The preset cleaning methods include nearest neighbor filtering and / or neighborhood consistency filtering.

[0139] Specifically, the calculation process of the first similarity between the first scale feature map of the first image and the first scale feature map of the second image is completed in the similarity decision layer in the dense matching network, and the specific process includes the following steps:

[0140] The similarity decision layer takes the first scale feature map of the first image and the first scale feature map of the second image as input. It first uses a correlation algorithm to calculate the similarity between all descriptor vectors in the first scale feature map of the first image and all descriptor vectors in the first scale feature map of the second image. Since the similarity obtained at this time is not a similarity that has been cleaned, its accuracy is not high, so it is called the initial similarity.

[0141] Subsequently, the initial similarity can be cleaned according to a preset cleaning method to increase the similarity of correct matches and reduce the similarity of incorrect matches, thereby ensuring its accuracy, thereby obtaining a first similarity between the first scale feature map of the first image and the first scale feature map of the second image. The preset cleaning method here can be a nearest neighbor filtering method, a neighborhood consistency filtering method, or a combination of the nearest neighbor filtering method and the neighborhood consistency filtering method, which is not specifically limited in the embodiments of the present invention.

[0142] Preferably, in the embodiment of the present invention, the above two data cleaning methods are alternately used to clean the initial similarity data. The specific process includes:

[0143] First, the initial similarity is input into the nearest neighbor filtering module in the similarity decision layer. The nearest neighbor filtering module uses the softened bidirectional nearest neighbor filtering method to clean the initial similarity data to increase the similarity of correct matches and reduce the similarity of incorrect matches, thus obtaining the similarity after the initial cleaning.

[0144] The similarity after the initial cleaning is then input into the neighborhood consensus filter module (NC module). The NC (Neighborhood Consensus) module uses four-dimensional convolution and the principle of neighborhood consistency to further enhance the similarity of correct matches in the similarity after the initial cleaning and reduce the similarity of incorrect matches, thus obtaining the similarity after the second cleaning. It is worth noting that the parameters in the four-dimensional convolution here are trainable parameters.

[0145] Afterwards, the softened bidirectional nearest neighbor filtering method is used again to clean the data for the similarity after the second round of cleaning to obtain the final similarity, that is, the first similarity between the first scale feature map of the first image and the first scale feature map of the second image.

[0146] Based on the above embodiment, the softened bidirectional nearest neighbor filtering method can be expressed as:

[0147]

[0148] Among them, C' represents the similarity obtained by the softened bidirectional nearest neighbor filtering method, that is, the similarity after the initial round of cleaning, C(i,j,k,l) represents the initial similarity, which is a four-dimensional tensor, i,j,k,l represents the data of each dimension respectively, and M(C) represents the use of softened bidirectional nearest neighbor filtering method to clean the initial similarity data. and Both are softened bidirectional filter coefficients.

[0149] and The calculation formula can be expressed as:

[0150]

[0151]

[0152] in, and Together suppress the similarity of incorrect matches in C(i,j,k,l); if C(i,j,k,l) is large, it means and The match is more reliable, so it can be determined and is larger, C′(i,j,k,l) is also larger, represents the first scale feature map of the first image, represents the descriptor vector in the first scale feature map of the first image, represents the first scale feature map of the second image, Represents the descriptor vector in the first-scale feature map of the second image.

[0153] Correspondingly, if C(i,j,k,l) is small, it means and The reliability of the match is not high, and it is very likely to be a wrong match, so it can be determined and is smaller, and C′(i,j,k,l) is also smaller.

[0154] Based on the above embodiment, determining a dense feature matching set between the first image and the second image based on the first similarity and the second similarity includes:

[0155] Determining, based on the second similarity, a second matching score map between each descriptor vector in the second scale feature map in the first feature map set and the second scale feature map in the second feature map set;

[0156] Based on the first similarity, obtaining, by bilinear interpolation, a first matching score map between each descriptor vector in the second scale feature map in the first feature map set and the first scale feature map in the second feature map set;

[0157] Based on the first matching score map and the second matching score map, a dense feature matching set between the first image and the second image is determined.

[0158] Specifically, in the above process, the process of determining a dense feature matching set between the first image and the second image based on the first similarity between the first scale feature map of the first image and the first scale feature map of the second image, and the similarity between the second scale feature map of the first image and the second scale feature map of the second image, specifically includes the following steps:

[0159] First, based on the second similarity, a second matching score map between each descriptor vector in the second scale feature map of the first image and the second scale feature map of the second image can be determined. The specific process can be: from the second similarity, the similarity between any descriptor vector in the second scale feature map of the first image and all descriptor vectors in the second scale feature map of the second image is obtained, and based on the similarity, a matching score map between the descriptor vector and the second scale feature map of the second image is constructed. This process is repeated to obtain a second matching score map between each descriptor vector in the second scale feature map of the first image and the second scale feature map of the second image;

[0160] Then, considering that the first scale feature map and the second scale feature map of the first image correspond to each other, and similarly, the first scale feature map and the second scale feature map of the second image also correspond to each other, a first matching score map between each descriptor vector in the second scale feature map of the first image and the first scale feature map of the second image can be determined by bilinear interpolation based on the first similarity;

[0161] After that, the dense feature matching set between the first image and the second image can be determined based on the first matching score map and the second matching score map. Specifically, the first matching score map and the second matching score map are fused, that is, the matching constraints are obtained by the first matching score map determined by the first similarity with higher reliability but lower resolution obtained through data cleaning, and then the erroneous matches in the second matching score map determined by the second similarity that has not been cleaned are suppressed based on the matching constraints, so as to obtain a high-resolution, accurate and reliable target matching score map, and based on the target matching score map, the dense feature matching set between the first image and the second image is determined.

[0162] Based on the above embodiment, based on the first similarity, determining a first matching score map between each descriptor vector in the second scale feature map in the first feature map set and the first scale feature map in the second feature map set by bilinear interpolation includes:

[0163] Based on the first similarity, obtaining, by bilinear interpolation, a description similarity between each descriptor vector in the second-scale feature map in the first feature map set and the first-scale feature map in the second feature map set;

[0164] Based on the description similarity, a first matching score map is determined between each descriptor vector in the second scale feature map in the first feature map set and the first scale feature map in the second feature map set.

[0165] Specifically, the process of determining the first matching score map between each descriptor vector in the second scale feature map of the first image and the first scale feature map of the second image by bilinear interpolation based on the first similarity specifically includes the following steps:

[0166] On the premise that the first scale feature map and the second scale feature map of the same image have a corresponding relationship, first, bilinear interpolation can be used to obtain the similarity between each descriptor vector in the second scale feature map of the first image and all descriptor vectors in the first scale feature map of the second image from the first similarity. This similarity is the description similarity;

[0167] Subsequently, based on the description similarity between each descriptor vector in the second scale feature map of the first image and the first scale feature map of the second image, a first matching score map between each descriptor vector in the second scale feature map of the first image and the first scale feature map of the second image can be constructed.

[0168] Based on the above embodiment, the process of determining the first matching score map by bilinear interpolation based on the first similarity can be expressed by the following formula:

[0169]

[0170] in, is the first matching score map, bilinear represents bilinear interpolation, represents the first similarity, Indicates rounding up. Indicates rounding down.

[0171] Based on the above embodiment, determining a dense feature matching set between the first image and the second image based on the first matching score map and the second matching score map includes:

[0172] Upsampling the first matching score map, and determining a target matching score map based on the upsampled first matching score map and the second matching score map;

[0173] Based on the target matching score map, a set of dense feature matches between the first image and the second image is determined.

[0174] Taking into account that the second similarity has not undergone data cleaning and the similarity of incorrect matches has not been reduced, it can be determined that the "noise" content in the second matching score graph determined based on the second similarity is significantly higher than the "noise" content in the first matching score graph determined based on the first similarity obtained after data cleaning.

[0175] The matching information contained in the first similarity extracted by the nearest neighbor filtering method and / or the neighborhood consistency filtering method is extremely reliable, and the first matching score map determined based on the first similarity is also extremely reliable. However, in the process of improving the reliability, its resolution is reduced, that is, the resolution of the first matching score map is too low. If a dense feature matching set is directly output from the first matching score map with too low a resolution, the coordinate accuracy of the matching points will be low.

[0176] In view of this, in an embodiment of the present invention, a process of determining a dense feature matching set between a first image and a second image based on a first matching score map and a second matching score map specifically includes the following steps:

[0177] First, upsampling the first matching score map to improve the resolution of the first matching score map, thereby obtaining an upsampled first matching score map;

[0178] Subsequently, the target matching score map can be determined based on the upsampled first matching score map and the second matching score map. That is, the upsampled first matching score map is used to suppress the "noise" in the second matching score map. It can also be understood that the matching constraints are obtained through the first matching score map with higher reliability but lower resolution, and then the matching constraints are used to suppress the erroneous matches in the second matching score map with higher resolution but lower reliability, thereby obtaining a high-resolution, accurate and reliable target matching score map.

[0179] After that, a dense feature matching set between the first image and the second image can be determined based on the target matching score map. Specifically, the pixel of the second image corresponding to the maximum value in the target matching score map can be used as the dense feature matching of the pixel of the first image corresponding to the corresponding descriptor vector in the second scale feature map of the first image. In this way, a dense feature matching can be determined for each descriptor vector in the second scale feature map of the first image, and finally a dense feature matching set between the first image and the second image can be established.

[0180] Based on the above embodiment, the process of determining the target matching score map according to the upsampled first matching score map and the second matching score map can be expressed by the following formula:

[0181]

[0182] in, is the target matching score map, represents the second matching score graph, represents the first matching score graph, It represents the first matching score map after upsampling, and ⊙ represents element-wise multiplication.

[0183] Based on the above embodiments, Figure 3 is a schematic diagram of the structure of the dense matching network provided by the present invention, such as Figure 3 As shown, the dense matching network includes a multi-scale feature extraction layer, a similarity judgment layer, and a dense feature matching layer. The overall process of performing dense feature matching on a first image and a second image according to the dense matching network to obtain a dense feature matching set between the first image and the second image may include the following steps:

[0184] First, the first image I A and the second image I B In the input dense matching network, in the multi-scale feature extraction layer, a convolutional neural network based on feature pyramid is used to extract the first image I A and the second image I B Perform multi-scale feature extraction to obtain the first image I A The first feature map set, and the second image I B A second feature map set of

[0185] The first feature map set includes the first image I A The first scale feature map and the second scale feature map The second feature map set includes the second image I B The first scale feature map and the second scale feature map

[0186] Contains H a ×W a C-dimensional descriptor vector, H a ×W a express The resolution of H a for High, W afor width.

[0187] Contains H b ×W b C-dimensional descriptor vector, H b ×W b express Resolution, H b for High, W b for width.

[0188] It should be noted that in the convolutional neural network based on the feature pyramid in the implementation of the present invention, the backbone network used is ResNet34 up to the conv4_6 (Convolution4 to 6) layer.

[0189] Then, and As the input of the similarity decision layer, the similarity decision layer first uses the correlation algorithm to calculate All descriptor vectors in The initial similarities between all descriptor vectors in , all the calculated initial similarities can form a four-dimensional tensor (4D Cost Volume, C). C(i,j,k,l) represents The descriptor vector in and The descriptor vector in The initial similarity between them is calculated; then, the initial similarity is input to the nearest neighbor filtering module, which uses the softened bidirectional nearest neighbor filtering method to clean the initial similarity data to increase the similarity of correct matches and reduce the similarity of incorrect matches, thereby obtaining the similarity C' after the initial cleaning; thereafter, C' is input to the neighborhood consistency filtering module (NC module), which uses four-dimensional convolution and the neighborhood consistency principle to further enhance the similarity of correct matches in C' and reduce the similarity of incorrect matches in C', thereby obtaining the similarity after the second round of cleaning. Finally, the softened bidirectional nearest neighbor filtering method is used again to Perform data cleaning to obtain and The first similarity between

[0190] It is worth noting that the parameters in the four-dimensional convolution here are trainable parameters.

[0191] Afterwards, in the dense feature matching layer, the second matching branch is and As input, use the relevant algorithm to calculate All descriptor vectors in The similarity between all descriptor vectors in is the second similarity. All similarities calculated can also form a four-dimensional tensor (4D Cost Volume, ); then, you can Get Any descriptor vector in and The similarity between all descriptor vectors in , and a two-dimensional matching score graph can be constructed based on this similarity. Repeat the above process to get Each descriptor vector in and The similarity between all descriptor vectors in Each descriptor vector in The similarity between them can be described based on this similarity to determine Each descriptor vector in The second matching score graph between

[0192] And in The position (i,j) in Under the premise that (i′, j′) in the dense feature matching layer corresponds to each other, the first matching branch in the dense feature matching layer can adopt the bilinear interpolation method to obtain Get Each descriptor vector in The description similarity between them can be used to determine Each descriptor vector in The two-dimensional first matching score map between

[0193] At the end of the dense feature matching layer, Upsampling is performed to improve the resolution of the first matching score map, thereby obtaining the upsampled first matching score map Then use To suppress In other words, the "noise" in the Get matching constraints and then suppress higher resolution but lower reliability based on these matching constraints The error matching in the target is eliminated, thus obtaining a high-resolution, accurate and reliable target matching score map. Afterwards, I corresponding to the maximum value B Pixels, as I A The corresponding descriptor vector The corresponding I A Dense feature matching of pixels, so it can be The descriptor vectors in the quantized feature vectors determine the dense feature matching, and finally we can establish I A and I B A dense feature matching set between .

[0194] Figure 4 This is the overall framework diagram of the camera pose estimation method provided by the present invention, such as Figure 4 As shown, first, a first image and a second image are determined;

[0195] Then, feature extraction is performed on the first image and the second image respectively to obtain first sparse features and second sparse features, and based on the first sparse features and the second sparse features, a large number of sparse feature matches are established between the first image and the second image using a bidirectional nearest neighbor matching method, and then a ratio test method is applied to eliminate false matches, and finally a sparse feature matching set is obtained;

[0196] At the same time, the first image and the second image can be input into a dense matching network, which can take into account the global and local context information of the input image, perform pixel-by-pixel matching on the input first image and the second image, and finally obtain a dense feature matching set between the first image and the second image output by the dense matching network;

[0197] Subsequently, the sparse feature matching set and the dense feature matching set can be fused to obtain the overall feature matching set;

[0198] After that, the random sampling consistency algorithm based on epipolar geometry can be used to eliminate false matches in the overall feature matching set, and the camera pose between the first image and the second image can be solved based on the overall feature matching set after false matches are eliminated.

[0199] The method provided by the embodiment of the present invention, through the fusion of a sparse feature matching set obtained by sparse feature matching and a dense feature matching set obtained by dense feature matching, can make the fused overall feature matching set more completely reflect the matching relationship between the first image and the second image, thereby making the camera pose solution process based on this overall feature matching set more accurate, and solves the problem that in traditional solutions, sufficient correct matches cannot be established for the camera pose estimation task in the case of drastic changes in lighting and viewing angle, and weak texture, resulting in poor accuracy and reliability of the estimated camera pose. Through sparse feature matching and dense feature matching, sufficient correct matches can be established between the two images, thereby achieving a dual improvement in the precision and accuracy of camera pose estimation.

[0200] Figure 5: is a schematic diagram of the effectiveness of the camera pose estimation method provided by the present invention, Figure 5 Specifically shown in the figure are the evaluation results of the camera pose estimation method provided by the present invention and the other four typical image matching methods on the HPatches dataset with drastic changes in illumination and viewing angle. It should be noted that the higher the MMA curve, the better the performance of the corresponding method. Figure 5 It can be seen that in terms of effectiveness, the present invention is significantly superior to the other four typical image matching methods.

[0201] The quantitative results on the HPatches dataset can be presented in the following table:

[0202] Ours D2-Net R2D2 SuperPoint SuperGlue MMA (%) 0.80 0.45 0.73 0.65 0.78

[0203] Among them, Ours is the camera pose estimation method provided by the present invention, D2-Net (ATrainable CNN for Joint Description and Detection of Local Features), R2D2 (Recurrent Replay Distributed DQN), SuperPoint, and SuperGlue are the other four typical image matching methods. The above table shows that the image matching effect achieved by the present invention is better than that of other methods.

[0204] Figure 6 : is a schematic diagram of camera pose estimation for a large jitter video sequence provided by the present invention, Figure 6 The camera pose estimation method provided by the present invention is specifically shown in FIG. , which is used to estimate the relative rotation of the camera pose for a video sequence with severe shaking. Figure 6 It can be seen that even in the face of severe shaking, the camera pose estimation method provided by the present invention can stably output the camera pose of the video frame.

[0205] The camera pose estimation device provided by the present invention is described below. The camera pose estimation device described below and the camera pose estimation method described above can be referenced to each other.

[0206] Figure 7 : is a structural diagram of the camera pose estimation device provided by the present invention, such as Figure 7 As shown, the device includes:

[0207] An image determining unit 710 is configured to determine a first image and a second image;

[0208] a sparse feature matching unit 720 configured to perform feature extraction on the first image and the second image, respectively, and determine a sparse feature matching set between the first image and the second image based on the first sparse features and the second sparse features obtained by the feature extraction;

[0209] A dense feature matching unit 730 is configured to perform dense feature matching on the first image and the second image based on a dense matching network to obtain a dense feature matching set between the first image and the second image; the dense matching network is trained based on a sample first image, a sample second image, and a sample dense feature matching set between the sample first image and the sample second image;

[0210] The camera pose estimation unit 740 is configured to determine a camera pose between the first image and the second image based on the sparse feature matching set and the dense feature matching set.

[0211] The camera pose estimation device provided by the present invention can make the overall feature matching set obtained by fusion of the sparse feature matching set obtained by sparse feature matching and the dense feature matching set obtained by dense feature matching more completely reflect the matching relationship between the first image and the second image, thereby making the camera pose solution process based on this overall feature matching set more accurate. It solves the problem in which the traditional scheme cannot establish sufficient correct matches for the camera pose estimation task in the case of drastic changes in lighting and viewing angle, and weak texture, resulting in poor accuracy and reliability of the estimated camera pose. Through sparse feature matching and dense feature matching, sufficient correct matches can be established between the two images, thereby achieving a double improvement in the precision and accuracy of camera pose estimation.

[0212] Based on the above embodiment, the camera pose estimation unit 740 is used to:

[0213] fusing the sparse feature matching set and the dense feature matching set to obtain an overall feature matching set;

[0214] False matches are eliminated from the overall feature matching set, and a camera pose between the first image and the second image is determined based on the overall feature matching set after false matches are eliminated.

[0215] Based on the above embodiment, the dense matching network includes a multi-scale feature extraction layer, a similarity judgment layer and a dense feature matching layer;

[0216] The multi-scale feature extraction layer is used to perform multi-scale feature extraction on the first image and the second image respectively to obtain a first feature map set of the first image and a second feature map set of the second image;

[0217] The similarity decision layer is used to determine a first similarity between a first scale feature map in the first feature map set and a first scale feature map in the second feature map set;

[0218] The dense feature matching layer is used to determine a dense feature matching set between the first image and the second image based on the first similarity and a second similarity, where the second similarity is the similarity between the second scale feature map in the first feature map set and the second scale feature map in the second feature map set.

[0219] Based on the above embodiment, the dense feature matching unit 730 is used to:

[0220] Determining an initial similarity between a first scale feature map in the first feature map set and a first scale feature map in the second feature map set;

[0221] Performing data cleaning on the initial similarity based on a preset cleaning method to obtain a first similarity between a first scale feature map in the first feature map set and a first scale feature map in the second feature map set;

[0222] The preset cleaning method includes a nearest neighbor filtering method and / or a neighborhood consistency filtering method.

[0223] Based on the above embodiment, the dense feature matching unit 730 is used to:

[0224] Determining, based on the second similarity, a second matching score map between each descriptor vector in the second scale feature map in the first feature map set and the second scale feature map in the second feature map set;

[0225] Based on the first similarity, determining, by bilinear interpolation, a first matching score map between each descriptor vector in the second scale feature map in the first feature map set and the first scale feature map in the second feature map set;

[0226] Based on the first matching score map and the second matching score map, a dense feature matching set between the first image and the second image is determined.

[0227] Based on the above embodiment, the dense feature matching unit 730 is used to:

[0228] Based on the first similarity, obtaining, by bilinear interpolation, a description similarity between each descriptor vector in the second scale feature map in the first feature map set and the first scale feature map in the second feature map set;

[0229] Based on the description similarity, a first matching score map is determined between each descriptor vector in the second scale feature map in the first feature map set and the first scale feature map in the second feature map set.

[0230] Based on the above embodiment, the dense feature matching unit 730 is used to:

[0231] Upsampling the first matching score map, and determining a target matching score map based on the upsampled first matching score map and the second matching score map;

[0232] Based on the object matching score map, a set of dense feature matches between the first image and the second image is determined.

[0233] Figure 8 An example of a physical structure diagram of an electronic device is shown below. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a camera pose estimation method, which includes: determining a first image and a second image; performing feature extraction on the first image and the second image respectively, and determining a sparse feature matching set between the first image and the second image based on the first sparse features and the second sparse features obtained by the feature extraction; performing dense feature matching on the first image and the second image based on a dense matching network to obtain a dense feature matching set between the first image and the second image; and determining the camera pose between the first image and the second image based on the sparse feature matching set and the dense feature matching set.

[0234] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0235] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the camera pose estimation method provided by the above methods, the method including: determining a first image and a second image; performing feature extraction on the first image and the second image respectively, and determining a sparse feature matching set between the first image and the second image based on the first sparse features and the second sparse features obtained by the feature extraction; performing dense feature matching on the first image and the second image based on a dense matching network to obtain a dense feature matching set between the first image and the second image; determining the camera pose between the first image and the second image based on the sparse feature matching set and the dense feature matching set.

[0236] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the camera pose estimation method provided by the above-mentioned methods, the method comprising: determining a first image and a second image; performing feature extraction on the first image and the second image respectively, and determining a sparse feature matching set between the first image and the second image based on the first sparse features and the second sparse features obtained by the feature extraction; performing dense feature matching on the first image and the second image based on a dense matching network to obtain a dense feature matching set between the first image and the second image; and determining the camera pose between the first image and the second image based on the sparse feature matching set and the dense feature matching set.

[0237] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0238] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0239] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A camera pose estimation method, characterized in that: include: determining a first image and a second image; performing feature extraction on the first image and the second image respectively, and determining a sparse feature matching set between the first image and the second image based on first sparse features and second sparse features obtained by the feature extraction; performing dense feature matching on the first image and the second image based on a dense matching network to obtain a dense feature matching set between the first image and the second image; determining a camera pose between the first image and the second image based on the sparse feature matching set and the dense feature matching set; The dense matching network is trained based on a sample first image, a sample second image, and a sample dense feature matching set between the sample first image and the sample second image; The dense matching network includes a multi-scale feature extraction layer, a similarity judgment layer and a dense feature matching layer; The multi-scale feature extraction layer is used to perform multi-scale feature extraction on the first image and the second image respectively to obtain a first feature map set of the first image and a second feature map set of the second image; The similarity decision layer is used to determine a first similarity between a first scale feature map in the first feature map set and a first scale feature map in the second feature map set; The dense feature matching layer is used to determine a dense feature matching set between the first image and the second image based on the first similarity and a second similarity, where the second similarity is the similarity between the second scale feature map in the first feature map set and the second scale feature map in the second feature map set.

2. The camera pose estimation method according to claim 1, wherein The determining, based on the sparse feature matching set and the dense feature matching set, a camera pose between the first image and the second image, includes: fusing the sparse feature matching set and the dense feature matching set to obtain an overall feature matching set; False matches are eliminated from the overall feature matching set, and a camera pose between the first image and the second image is determined based on the overall feature matching set after false matches are eliminated.

3. The camera pose estimation method according to claim 1, wherein The first similarity is determined based on the following steps: Determining an initial similarity between a first scale feature map in the first feature map set and a first scale feature map in the second feature map set; Performing data cleaning on the initial similarity based on a preset cleaning method to obtain a first similarity between a first scale feature map in the first feature map set and a first scale feature map in the second feature map set; The preset cleaning method includes a nearest neighbor filtering method and / or a neighborhood consistency filtering method.

4. The camera pose estimation method according to claim 1, wherein The determining, based on the first similarity and the second similarity, a dense feature matching set between the first image and the second image includes: Determining, based on the second similarity, a second matching score map between each descriptor vector in the second scale feature map in the first feature map set and the second scale feature map in the second feature map set; Based on the first similarity, determining, by bilinear interpolation, a first matching score map between each descriptor vector in the second scale feature map in the first feature map set and the first scale feature map in the second feature map set; Based on the first matching score map and the second matching score map, a dense feature matching set between the first image and the second image is determined.

5. The camera pose estimation method according to claim 4, wherein: The determining, based on the first similarity, by bilinear interpolation, a first matching score map between each descriptor vector in the second scale feature map in the first feature map set and the first scale feature map in the second feature map set includes: Based on the first similarity, obtaining, by bilinear interpolation, a description similarity between each descriptor vector in the second scale feature map in the first feature map set and the first scale feature map in the second feature map set; Based on the description similarity, a first matching score map is determined between each descriptor vector in the second scale feature map in the first feature map set and the first scale feature map in the second feature map set.

6. The camera pose estimation method according to claim 4, wherein: The determining, based on the first matching score map and the second matching score map, a dense feature matching set between the first image and the second image includes: Upsampling the first matching score map, and determining a target matching score map based on the upsampled first matching score map and the second matching score map; Based on the object matching score map, a set of dense feature matches between the first image and the second image is determined.

7. A camera pose estimation device, characterized in that: include: an image determining unit, configured to determine a first image and a second image; a sparse feature matching unit, configured to perform feature extraction on the first image and the second image respectively, and determine a sparse feature matching set between the first image and the second image based on the first sparse features and the second sparse features obtained by the feature extraction; a dense feature matching unit, configured to perform dense feature matching on the first image and the second image based on a dense matching network to obtain a dense feature matching set between the first image and the second image; the dense matching network is trained based on a sample first image, a sample second image, and a sample dense feature matching set between the sample first image and the sample second image; a camera pose estimation unit, configured to determine a camera pose between the first image and the second image based on the sparse feature matching set and the dense feature matching set; The dense matching network includes a multi-scale feature extraction layer, a similarity judgment layer and a dense feature matching layer; The multi-scale feature extraction layer is used to perform multi-scale feature extraction on the first image and the second image respectively to obtain a first feature map set of the first image and a second feature map set of the second image; The similarity decision layer is used to determine a first similarity between a first scale feature map in the first feature map set and a first scale feature map in the second feature map set; The dense feature matching layer is used to determine a dense feature matching set between the first image and the second image based on the first similarity and a second similarity, where the second similarity is the similarity between the second scale feature map in the first feature map set and the second scale feature map in the second feature map set.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the camera pose estimation method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the camera pose estimation method according to any one of claims 1 to 6 is implemented.

Images