Camera pose estimation methods, devices, electronic equipment and storage media

By using the initial camera pose as prior knowledge in panoramic image estimation, images with insufficient constraint information are filtered and fixed, and then bundled and adjusted. This solves the problem of camera pose estimation failure when lacking constraint information and achieves more accurate camera pose determination.

CN118864586BActive Publication Date: 2026-11-17DOUYIN VISION CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202310475025.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-04-27
Publication Date
2026-11-17
Estimated Expiration
2043-04-27

AI Technical Summary

Technical Problem

In existing technologies, camera pose estimation methods have strict requirements for camera location selection. The interval between adjacent cameras cannot be too large and they must share a common field of view, which leads to estimation failure when there is a lack of constraint information.

Method used

By acquiring the initial camera pose of the panoramic image as prior knowledge, target panoramic images with insufficient constraint information are filtered out, and their initial poses are fixed. The initial point cloud and feature points are bundled and adjusted to determine the target camera pose of other panoramic images.

Benefits of technology

This avoids pose estimation failures due to insufficient constraint information, thus improving the accuracy and success rate of camera pose estimation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118864586B_ABST
    Figure CN118864586B_ABST
Patent Text Reader

Abstract

This disclosure relates to the field of 3D video technology, specifically to camera pose estimation methods, apparatuses, electronic devices, and storage media. The method includes acquiring at least two panoramic images and their initial camera poses; extracting and matching feature points from any two panoramic images to obtain matching feature points in the panoramic images; constructing an initial point cloud of an object based on the matching feature points and the initial camera pose; selecting a target panoramic image from the at least two panoramic images based on the matching feature points, where the constraint information includes the number and / or distribution of matching feature points; fixing the initial camera pose of the target panoramic image, and adjusting the initial camera poses of other panoramic images based on the initial point cloud and the feature points to determine the target camera pose of the panoramic image. The initial camera pose serves as prior knowledge, avoiding pose estimation failures due to insufficient constraint information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of three-dimensional video technology, specifically to camera pose estimation methods, devices, electronic devices, and storage media. Background Technology

[0002] The camera pose is determined by estimating the camera's position and orientation at the time of capture based on the images captured by the camera, including the rotation matrix R and translation vector t. Specifically, feature extraction and matching are performed on the acquired images to calculate the geometric relationships between pairs of images. Then, point cloud reconstruction is performed, and finally, the pose and point cloud are bundled and optimized to obtain the camera pose. However, this method has certain requirements for the selection of camera points; the interval between adjacent cameras cannot be too large, and there must be a certain common viewing area between the images. Otherwise, it will be difficult to obtain the camera pose. Summary of the Invention

[0003] In view of this, embodiments of the present disclosure provide a camera pose estimation method, apparatus, electronic device, and storage medium to solve the problem of camera pose estimation.

[0004] According to a first aspect, embodiments of this disclosure provide a camera pose estimation method, including: Acquire at least two panoramic images and the initial camera pose of the panoramic images, wherein the at least two panoramic images are panoramic images obtained by taking pictures of the same object from different points; Feature point extraction and feature point matching are performed on any two of the panoramic images to obtain the matching feature points in the panoramic images; Based on the matched feature points and the initial camera pose, an initial point cloud of the object is constructed; Based on the matching feature points, target panoramic images with constraint information below a threshold are selected from at least two panoramic images. The constraint information includes the number of matching feature points and / or the distribution of the matching feature points. The initial camera pose of the target panoramic image is fixed, and the initial camera poses of other panoramic images are bundled and adjusted based on the initial point cloud and the feature points to determine the target camera pose of the panoramic image.

[0005] According to a second aspect, embodiments of this disclosure provide a camera pose estimation apparatus, including: The acquisition module is used to acquire at least two panoramic images and the initial camera pose of the panoramic images, wherein the at least two panoramic images are panoramic images obtained by taking pictures of the same object from different points. The matching module is used to extract and match feature points from any two panoramic images to obtain matching feature points in the panoramic images. A construction module is used to construct an initial point cloud of the object based on the matched feature points and the initial camera pose; A filtering module is used to filter out target panoramic images with constraint information below a threshold from at least two panoramic images based on the matching feature points, wherein the constraint information includes the number of matching feature points and / or the distribution of the matching feature points; An adjustment module is used to fix the initial camera pose of the target panoramic image, and to bundle and adjust the initial camera poses of other panoramic images based on the initial point cloud and the feature points, so as to determine the target camera pose of the panoramic image.

[0006] According to a third aspect, this disclosure provides an electronic device, including: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the camera pose estimation method described in the first aspect or any embodiment of the first aspect.

[0007] According to a fourth aspect, embodiments of this disclosure provide a computer-readable storage medium storing computer instructions for causing the computer to perform the camera pose estimation method described in the first aspect or any embodiment of the first aspect.

[0008] The camera pose estimation method provided in this embodiment obtains the initial camera pose of the panoramic image as prior knowledge, uses matching feature points to determine the target panoramic image with insufficient constraint information, and keeps the initial camera pose of these target panoramic images unchanged during the bundle adjustment process, thus avoiding the problem of pose estimation failure due to insufficient constraint information, and can obtain a more accurate target camera pose than the initial camera pose. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0010] Figure 1 This is a flowchart of a machine pose estimation method according to an embodiment of the present disclosure; Figure 2 This is a flowchart of a machine pose estimation method according to an embodiment of the present disclosure; Figure 3 This is a flowchart of a machine pose estimation method according to an embodiment of the present disclosure; Figure 4 This is a structural block diagram of a machine position and pose estimation device according to an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this disclosure. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0012] Camera pose estimation is commonly used for indoor modeling, such as in exhibition halls, museums, and rooms. Specifically, photographers use panoramic cameras to capture panoramic images from different points within the room. By estimating the camera pose of these panoramic images, the relative positional relationships between them are obtained, allowing for 3D reconstruction of the interior to create a 3D model. However, panoramic image camera pose estimation is performed by extracting and matching features from any two captured panoramic images. This requires consideration of the common viewing area between any two panoramic images. If the points where the panoramic images are captured are not chosen appropriately, a common viewing area may not exist. The absence of a common viewing area leads to insufficient constraints in the bundle adjustment, resulting in camera pose estimation failure.

[0013] Based on this, the camera pose estimation method provided in this disclosure acquires the initial camera pose of each panoramic image in addition to acquiring the panoramic image. The initial camera pose is used as the constraint for subsequent bundle constraints. That is, for panoramic images with insufficient constraint information, the initial camera pose is kept unchanged and used as the target camera pose. Then, the bundle is adjusted to obtain the target camera pose of other panoramic images.

[0014] According to an embodiment of this disclosure, a camera pose estimation method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0015] This embodiment provides a camera pose estimation method that can be used in electronic devices, such as computers and mobile terminals. Figure 1 This is a flowchart of a machine pose estimation method according to an embodiment of the present disclosure, as follows: Figure 1 As shown, the process includes the following steps: S11, acquire at least two panoramic images and the initial camera pose of the panoramic images.

[0016] The at least two panoramic images are panoramic images of the same object taken from different points.

[0017] To obtain at least two panoramic images, the same object is photographed from different locations. For example, photographing the same object from different locations within the same room will yield at least two panoramic images. The same object refers to the same shooting scene, such as a room, an exhibition hall, etc. The number of panoramic images is determined based on actual needs and is not limited here. The initial camera pose of the panoramic images can be obtained manually by the user or provided using a third-party AR module, etc. Taking manual stitching as an example, when the photographer uses a panoramic camera to capture panoramic images, they connect a mobile terminal to the panoramic camera. The panoramic images captured by the panoramic camera are synchronized to the mobile terminal in real time. After capturing panoramic images of the indoor space, the photographer interacts with the mobile terminal to drag and stitch the panoramic images. For example, panoramic images 1, 2, and 3 are captured. Panoramic image 1 is east of panoramic image 2, and panoramic image 3 is west of panoramic image 2. Therefore, the photographer drags and stitches panoramic images 1-3 to adjust their relative positions, thereby obtaining the initial camera pose of each panoramic image.

[0018] These initial poses contain significant errors. Although they can be used directly for subsequent 3D reconstruction, they can be used as prior information to overcome the problem of camera pose estimation failure under difficult conditions.

[0019] S12: Extract and match feature points from any two panoramic images to obtain matching feature points in the panoramic images.

[0020] Feature point extraction and matching are performed on any two panoramic images among all the acquired panoramic images. Continuing with the above example, the acquired panoramic images include panoramic images 1 to 3. Based on this, feature point extraction and matching are performed using panoramic images 1 and 2, 1 and 3, and 2 and 3.

[0021] Feature extraction includes, but is not limited to, SIFT features, SURF features, or ORB features, etc. SIFT stands for Scale-invariant feature transform, SURF for Speeded Up Robust Features, and ORB for Oriented Fast and Rotated BRIEF. In some implementations, this embodiment uses SIFT features because SIFT features are scale-invariant and rotation-invariant, thus achieving better feature matching results.

[0022] Feature point matching is performed between any two panoramic images. For example, if there are 100 feature points in panoramic image 1 and 50 feature points in panoramic image 2, then each feature point in panoramic image 1 needs to be matched with the 50 feature points in panoramic image 2. The feature point with the highest matching degree in panoramic image 2 is taken as the feature point corresponding to the feature point in panoramic image 1. The two matched feature points are called the matched feature points.

[0023] The matching method for feature points can be calculated by measuring the distance between feature points, or other methods can be used, etc., and no restrictions are imposed on it here.

[0024] S13: Construct the initial point cloud of the object based on the matched feature points and the initial camera pose.

[0025] The initial point cloud of the object is obtained using matching feature points from any two panoramic images and the initial camera pose. For example, the initial camera poses of panoramic image 1 are R1 and t1, and the initial camera poses of panoramic image 2 are R2 and t2. The normalized coordinates of the matching feature points in their respective camera coordinate systems are x1 and x2. Based on this, the following formula is obtained:

[0026] In the formula, P represents the coordinates of the 3D point in the initial point cloud in the world coordinate system, and s1 and s2 represent the depths of the 3D point in the camera's view, respectively. By applying a linear transformation to the above formula, the coordinates P of the 3D point in the world coordinate system can be obtained. Using this method, the coordinates corresponding to all matching feature points can be obtained, thus yielding the initial point cloud of the object.

[0027] S14, based on matching feature points, select target panoramic images from at least two panoramic images whose constraint information is below a threshold.

[0028] The constraint information includes the number of matching feature points and / or the distribution of matching feature points.

[0029] As mentioned above, due to the potentially large intervals between the acquisition points of panoramic cameras, the common viewing area between two panoramic images may be insufficient. This could result in some panoramic images lacking sufficient constraint information for camera poses, causing the constraint information to fall below a threshold. In such cases, without any constraints, the final optimized camera pose might drift to a clearly incorrect position. Therefore, this paper addresses this issue by fixing a portion of the initial camera poses with constraint information below the threshold and directly using these initial poses in subsequent bundle adjustment. Bundle adjustment transforms state estimation into a nonlinear least squares problem, and this bundle adjustment algorithm is used to calculate the target camera pose of the panoramic image.

[0030] The sufficiency of constraint information is determined by comparing the number of matching feature points in each panoramic image with a preset number. If the number of matching feature points is less than the preset number, the constraint information is insufficient. Alternatively, the spatial distribution of each matching feature point can be used. For example, for the same panoramic image, cluster analysis can be performed on the feature points among the matching feature points. If the distribution of the feature points among the matching feature points is relatively concentrated, the constraint information is insufficient.

[0031] It should be noted that using the number and / or distribution of matching feature points to determine whether the constraint information is sufficient is merely an example and does not limit the scope of protection of this disclosure. Other judgment factors may also be included, and are set according to actual needs. No limitations are made here.

[0032] For all panoramic images, based on whether the constraint information is sufficient, they are divided into two types: target panoramic images and other panoramic images. Target panoramic images are those with insufficient constraint information, while other panoramic images are all panoramic images excluding the target panoramic images.

[0033] S15: Fix the initial camera pose of the target panoramic image, and adjust the initial camera poses of other panoramic images based on the initial point cloud and matching feature points to determine the target camera pose of the panoramic image.

[0034] For the target panoramic image, since its constraint information is insufficient, its initial camera pose is fixed during the bundle adjustment. That is, the initial camera pose of the target panoramic image remains unchanged during the bundle adjustment process; after the bundle adjustment is completed, the initial camera pose of the target panoramic image is determined as the target camera pose of the target panoramic image.

[0035] For the initial camera pose of other panoramic images, it is obtained by bundling and adjusting the initial point cloud and feature points. Since the initial point cloud is obtained using the initial camera pose, and the initial camera pose has errors, the initial point cloud also has errors, and consequently, the 3D points in the initial point cloud also have deviations. In the absence of deviations, the obtained 3D points coincide with the feature points in the corresponding matching feature points in the camera coordinate system. Based on this, the bundling adjustment can be obtained by optimizing the reprojection error between the 3D points and the corresponding feature points.

[0036] The camera pose estimation method provided in this embodiment obtains the initial camera pose of the panoramic image as prior knowledge, uses matching feature points to determine the target panoramic image with insufficient constraint information, and keeps the initial camera pose of these target panoramic images unchanged during the bundle adjustment process, thus avoiding the problem of pose estimation failure due to insufficient constraint information, and can obtain a more accurate target camera pose than the initial camera pose.

[0037] This embodiment provides a camera pose estimation method that can be used in electronic devices, such as computers and mobile terminals. Figure 2 This is a flowchart of a machine pose estimation method according to an embodiment of the present disclosure, as follows: Figure 2 As shown, the process includes the following steps: S21, acquire at least two panoramic images and the initial camera pose of the panoramic images.

[0038] At least two of the panoramic images are panoramic images taken of the same object from different points. See details below. Figure 1 S11 of the illustrated embodiment will not be described again here.

[0039] S22, extract and match feature points from any two panoramic images to obtain matching feature points in the panoramic images.

[0040] Please see details Figure 1 S12 of the illustrated embodiment will not be described again here.

[0041] S23, construct the initial point cloud of the object based on the matched feature points and the initial camera pose.

[0042] Please see details Figure 1 S13 of the illustrated embodiment will not be described again here.

[0043] S24, based on matching feature points, select target panoramic images from at least two panoramic images whose constraint information is below a threshold.

[0044] The constraint information includes the number of matching feature points and / or the distribution of matching feature points.

[0045] Please see details Figure 1 S14 of the illustrated embodiment will not be described again here.

[0046] S25: Fix the initial camera pose of the target panoramic image, and adjust the initial camera poses of other panoramic images based on the initial point cloud and feature points to determine the target camera pose of the panoramic image.

[0047] Specifically, S25 includes: S251, based on the positions of each 3D point and its corresponding feature point in the initial point cloud in the camera coordinate system, determine the reprojection angle error between the 3D point and its corresponding feature point.

[0048] The position of each 3D point in the initial point cloud in the camera coordinate system is obtained by mapping the 3D points to the initial pose of the camera. The feature point corresponding to the 3D point is the feature point in the matching feature point, and its coordinates represent its position in the camera coordinate system.

[0049] Unlike planar images, panoramic images exhibit distortion, especially at the North and South Poles. Therefore, relying solely on pixel reprojection errors can introduce significant inaccuracies. Consequently, reprojection angle errors are used for bundle adjustment.

[0050] Specifically, the reprojection angle error is the angle error between the first ray of the 3D point in the camera coordinate system and the second ray of the feature point in the camera coordinate system. The first ray is determined by mapping the 3D point to the camera coordinate system using the initial camera pose, and then connecting the mapped coordinate points from the origin of the camera coordinate system. Similarly, the second ray is determined by connecting the feature points from the origin of the camera coordinate system. The angle between the first and second rays is the reprojection angle error.

[0051] For each 3D point, the reprojection angle error between it and its corresponding feature point is calculated. All the obtained reprojection angle errors are summed to obtain the total reprojection angle error between all 3D points and their corresponding feature points. For example, the reprojection angle error can be expressed by the following formula. :

[0052] In the formula, For angle, These are the coordinates of the feature point.

[0053] S252: The initial camera pose of the fixed target panoramic image is adjusted by bundling and adjusting the initial camera poses of other panoramic images with the minimum reprojection angle error as the optimization objective, in order to determine the target camera pose of the panoramic image.

[0054] The goal of bundle adjustment optimization is to minimize the reprojection angle error mentioned above. As described, during bundle adjustment, the initial camera pose of the target panoramic image is fixed, and the initial camera pose in the reprojection angle error is optimized. Based on this, the target camera pose of the panoramic image is determined after bundle adjustment optimization.

[0055] The camera pose estimation method provided in this embodiment addresses the issue that panoramic images represent images within a three-dimensional space, where image distortion is more pronounced at the North and South Poles. By employing the reprojection angle error method to calculate the error between three-dimensional points and their corresponding feature points, the method avoids the problem of increased error due to image distortion. Therefore, the reprojection angle error can accurately represent the error between three-dimensional points and their corresponding two-dimensional feature points, thus improving the accuracy of the estimated target camera pose.

[0056] In some embodiments, S252 includes: (1) Obtain the pose degrees of freedom of the bundle adjustment, wherein the pose degrees of freedom are determined based on the usage state of the panoramic camera, which is used to acquire panoramic images.

[0057] (2) The initial camera pose of the fixed target panoramic image is adjusted by taking the minimum reprojection angle error as the optimization objective and the pose freedom as the constraint term, and the initial camera pose of other panoramic images in the reprojection angle error is bundled and adjusted to determine the target camera pose of the panoramic image.

[0058] During the assembly and adjustment process, pose degrees of freedom are also used for constraint. Generally, there are 6 pose degrees of freedom, namely the x, y, and z coordinates and the angles in the three directions (pitch, roll, and yaw). These pose degrees of freedom are determined based on the usage state of the panoramic camera, which includes whether the photographer is holding the panoramic camera while acquiring panoramic images, and whether the panoramic camera is fixed using a fixed bracket. If the panoramic camera is held by hand, the pose degrees of freedom are 6; if a fixed bracket is used to fix the panoramic camera, the pose degrees of freedom are 4. In this case, it is assumed that the panoramic camera only rotates at the yaw angle, and the pitch and roll angles are fixed at 0.

[0059] During the bundle adjustment optimization, pose degrees of freedom are introduced as constraints. The optimization objective is to minimize the reprojection angle error. The initial camera pose of the target panoramic image is fixed, and the initial camera poses of other panoramic images in the reprojection angle error are bundled and adjusted to determine the target camera pose of the panoramic image.

[0060] When bundling and adjusting the initial camera poses of other panoramic images, constraints on pose degrees of freedom are introduced to increase the constraint terms, which further improves the success rate of camera pose estimation when constraint information is insufficient.

[0061] In some implementations, the initial point cloud is also optimized during bundle adjustment. However, due to errors in the initial point cloud, the optimized target point cloud may contain erroneous and / or duplicate point clouds. Therefore, it is necessary to process the erroneous and / or duplicate point clouds to obtain an adjusted target point cloud, which is then bundled and adjusted again. That is, after each bundle adjustment optimization, the obtained point cloud is filtered and / or merged, and then bundle adjustment optimization is performed again. This process is repeated iteratively multiple times until convergence, resulting in the final optimized target camera pose. Based on this, the above-mentioned camera pose estimation method also includes: (1) Obtain the target camera pose after bundle adjustment and the target point cloud corresponding to the initial point cloud.

[0062] The target camera pose and the target point cloud corresponding to the initial point cloud are both obtained after the initial bundle adjustment.

[0063] (2) Filter and / or merge the target point cloud to obtain the adjusted target point cloud.

[0064] Target point cloud filtering refers to removing erroneous points from the target point cloud, or removing 3D points with significant errors. For example, this can be done by combining calculations with the reprojection angle error mentioned above, and so on.

[0065] Merging target point clouds involves combining 3D points that belong to the same feature point within the target point cloud. For example, retaining one of multiple 3D points, and so on.

[0066] As an optional implementation of this disclosure, filtering out the target point cloud includes: 2.1) Calculate the reprojection error between each 3D point in the target point cloud and its corresponding feature point.

[0067] 2.2) Based on the relationship between the reprojection error and the error threshold, the three-dimensional points in the target point cloud are filtered out to obtain the adjusted target point cloud.

[0068] Calculate the reprojection error between each 3D point in the target point cloud and its corresponding feature point, such as the reprojection angle error. Compare the calculated reprojection error with an error threshold. Points with errors greater than the threshold are considered erroneous and are removed from the target point cloud to obtain the adjusted target point cloud.

[0069] During feature point matching, there may be some mismatches, resulting in some incorrect point clouds. Incorrect point clouds can lead to errors in the estimated target camera pose. Therefore, the incorrect point clouds are filtered out by combining the reprojection error, which further improves the accuracy of the estimated target camera pose.

[0070] As an optional implementation of this disclosure, merging the target point cloud includes: 2.1) Select multiple target 3D points corresponding to the same feature point from the target point cloud.

[0071] 2.2) Based on the average of the coordinates of multiple target 3D points, determine the coordinates of the merged 3D point after merging the multiple target 3D points.

[0072] When calculating 3D points using feature points and camera pose, the correspondence between feature points and 3D points is recorded. This correspondence can then be used to identify multiple target 3D points corresponding to the same feature point. The average coordinates of these multiple target 3D points are then taken as the coordinates of the merged 3D point.

[0073] Since some feature points appear in multiple panoramic images, they are reconstructed into multiple 3D points when constructing the point cloud. Based on this, multiple target 3D points corresponding to the same feature point are merged, and the coordinates of the merged 3D point are determined based on the average coordinates of the multiple target 3D points. This method not only realizes the merging of repeated 3D points, but also integrates the coordinate information of each repeated 3D point, ensuring the accuracy of the obtained merged 3D.

[0074] (3) Based on the target camera pose of the panoramic image and the adjusted target point cloud, the target camera pose of other panoramic images is bundled and adjusted to determine the adjusted target camera pose.

[0075] After obtaining the adjusted target point cloud, the target camera poses of other panoramic images are bundled and adjusted again based on the target camera pose of the panoramic image to determine the adjusted target camera pose. The above-mentioned filtering and / or merging, and bundling adjustment of the target point cloud can be iterated multiple times to finally determine the adjusted target camera pose.

[0076] After bundle adjustment, the obtained target point cloud is filtered and / or merged to remove erroneous point clouds and / or merge duplicate point clouds, resulting in an adjusted target point cloud. Based on this, bundle adjustment is performed again to further improve the accuracy of the adjusted target camera pose.

[0077] This embodiment provides a camera pose estimation method that can be used in electronic devices, such as computers and mobile terminals. Figure 3This is a flowchart of a machine pose estimation method according to an embodiment of the present disclosure, as follows: Figure 3 As shown, the process includes the following steps: S31, acquire at least two panoramic images and the initial camera pose of the panoramic images.

[0078] At least two of the panoramic images are panoramic images taken of the same object from different points. See details below. Figure 1 S11 of the illustrated embodiment will not be described again here.

[0079] S32: Extract and match feature points from any two panoramic images to obtain matching feature points in the panoramic images.

[0080] Specifically, S32 includes: S321 converts any two panoramic images into target images in hexahedral format.

[0081] As mentioned above, panoramic images exhibit significant distortion, especially near the North and South Poles. Therefore, the panoramic image is first converted into a hexahedral target image for subsequent feature extraction. The hexahedral target image consists of six images obtained by processing the panoramic image.

[0082] S322, perform feature extraction on the target image to obtain the coordinates of the feature points in the target image.

[0083] Feature extraction is performed on the obtained target image to obtain the coordinates of feature points and feature descriptors. The feature point coordinates here are the coordinates within the target image, and they need to be mapped back onto the panoramic image.

[0084] Feature points are representative points in the target image, such as points whose color differs significantly from their surroundings. Feature points do not necessarily lie on a single pixel; that is, the coordinates of a feature point can be decimals. The specific representation of a feature descriptor is a high-dimensional vector, such as a 128-dimensional or 256-dimensional vector, where each element is in the range of 0.0 to 1.0. A feature point represents the coordinates of the center point of its predetermined region, and the feature descriptor represents the regional information and orientation of the predetermined region containing the feature point.

[0085] S323 maps the feature point coordinates of the target image to the corresponding panoramic image to obtain the feature points of the panoramic image.

[0086] The coordinate mapping is the inverse operation of the panoramic image mapping described above, mapping the feature points of the target image back to the corresponding panoramic image to obtain the feature points of the panoramic image.

[0087] S324: Perform feature point matching on any two panoramic images to determine selectable matching feature points.

[0088] For feature point matching, one method is to traverse all feature points in two images and calculate the dot product of the feature descriptors pairwise. If the dot product is greater than a certain threshold, the feature points are considered to be matched and these two feature points are considered to be optional matching feature points.

[0089] S325 filters available matching feature points based on the initial camera pose of any two panoramic images to determine the matching feature points.

[0090] Because mismatches are prone to occur in similar scenes (such as similar stickers, doors, windows, floor tiles, etc.), affecting subsequent pose estimation, a matching method guided by the initial camera pose is adopted. That is, the initial camera pose is used to filter the available matching feature points, removing mismatches, thereby obtaining the matching feature points.

[0091] In some embodiments, S325 includes: (1) Calculate the relative initial pose using the initial camera pose of any two panoramic images to determine the essential matrix between any two panoramic images.

[0092] (2) Based on the essential matrix between any two panoramic images, the selectable matching feature points are screened by epipolar geometric constraints to determine the matching feature points.

[0093] For any two panoramic images with initial camera poses, calculate the relative initial poses, and then use the relative initial poses to calculate the essential matrix between the two panoramic images. Correctly matched feature points should satisfy the epipolar geometry constraint, i.e., optional feature points whose ||x2^tEx1|| < the threshold are considered matched feature points. Here, x1 and x2 are the normalized coordinates of optional feature points on the two panoramic images in the camera coordinate system, respectively.

[0094] By combining the initial pose of the camera to obtain the essential matrix between panoramic images, the optional matching feature points are then filtered by polar geometric constraints, which improves the accuracy of the obtained matching feature points.

[0095] S33, based on matching feature points and initial camera pose, constructs the initial point cloud of the object.

[0096] Please see details Figure 1 S13 of the illustrated embodiment will not be described again here.

[0097] S34, based on matching feature points, select target panoramic images from at least two panoramic images whose constraint information is below a threshold.

[0098] The constraint information includes the number of matching feature points and / or the distribution of matching feature points.

[0099] Please see details Figure 1 S14 of the illustrated embodiment will not be described again here.

[0100] S35: Fix the initial camera pose of the target panoramic image, and adjust the initial camera poses of other panoramic images based on the initial point cloud and feature points to determine the target camera pose of the panoramic image.

[0101] Please see details Figure 2 S25 of the illustrated embodiment will not be described again here.

[0102] The camera pose estimation method provided in this embodiment addresses the significant distortion and warping in panoramic images, especially near the North and South Poles. Therefore, it converts the panoramic image into a hexahedral target image, which is a planar image for each target image. Feature extraction is performed on this target image, ensuring the accuracy of the extracted feature points. These extracted feature points are then mapped back to the panoramic image, resulting in more accurate feature points. Directly using feature point matching to obtain selectable matching feature points can easily lead to mismatches in similar scenes, such as similar stickers, doors, windows, and floor tiles, which can affect subsequent pose estimation. Therefore, filtering selectable matching feature points based on the initial camera pose eliminates a large number of mismatched selectable matching feature points, improving the accuracy of the obtained matching feature points.

[0103] In some implementations, after determining the target camera pose of the panoramic image, the method further includes: performing three-dimensional reconstruction based on at least two panoramic images and their target camera poses to determine a target three-dimensional model.

[0104] For acquiring panoramic images within the target space, after obtaining the target camera pose from the panoramic images, 3D reconstruction is performed on all panoramic images using the target camera pose to obtain a 3D model of the target space. Based on the relatively accurate target camera pose obtained, 3D reconstruction is performed on at least two panoramic images to ensure the accuracy of the obtained 3D model of the target.

[0105] As a specific application example of this disclosure, a stadium needs to be reconstructed in 3D. Based on this, a photographer uses a panoramic camera to capture multiple panoramic images from different locations within the stadium and synchronizes these images to a mobile terminal. The photographer interacts with the mobile terminal to drag the multiple panoramic images, thereby determining the initial camera pose of the panoramic images. Feature point extraction and matching are then performed on any two panoramic images to obtain matching feature points in the panoramic images, which are then combined with the initial camera pose to construct an initial point cloud. After obtaining the matching feature points, target panoramic images with insufficient constraint information are selected from the panoramic images, and their initial camera poses are fixed. Based on this, the initial camera poses of other panoramic images are bundled and adjusted according to the initial point cloud and the aforementioned feature points to determine the target camera pose of the panoramic images. After obtaining the target camera poses of each panoramic image, 3D reconstruction is performed on the panoramic images to obtain a 3D model of the stadium.

[0106] This embodiment also provides a camera pose estimation device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0107] This embodiment provides a camera pose estimation device, such as... Figure 4 As shown, it includes: The acquisition module 41 is used to acquire at least two panoramic images and the initial camera pose of the panoramic images, wherein the at least two panoramic images are panoramic images obtained by taking pictures of the same object at different points. Matching module 42 is used to extract and match feature points from any two panoramic images to obtain matching feature points in the panoramic images. Construction module 43 is used to construct an initial point cloud of the object based on the matched feature points and the initial camera pose; The filtering module 44 is used to filter out target panoramic images with constraint information below a threshold from at least two panoramic images based on the matching feature points, wherein the constraint information includes the number of matching feature points and / or the distribution of the matching feature points; The adjustment module 45 is used to fix the initial camera pose of the target panoramic image, and to bundle and adjust the initial camera poses of other panoramic images based on the initial point cloud and the feature points, so as to determine the target camera pose of the panoramic image.

[0108] In some implementations, the adjustment module 45 includes: The first determining unit is used to determine the reprojection angle error between the three-dimensional point and the corresponding feature point based on the position of each three-dimensional point in the initial point cloud and the corresponding feature point in the camera coordinate system. The bundle adjustment unit is used to fix the initial camera pose of the target panoramic image, and with the minimum reprojection angle error as the optimization target, bundle and adjust the initial camera poses of other panoramic images in the reprojection angle error to determine the target camera pose of the panoramic image.

[0109] In some implementations, the bundle adjustment unit includes: The first acquisition subunit is used to acquire the pose degrees of freedom of the bundle adjustment, the pose degrees of freedom being determined based on the usage state of the panoramic camera, the panoramic camera being used to acquire the panoramic image; The bundle adjustment subunit is used to fix the initial camera pose of the target panoramic image, take the minimum reprojection angle error as the optimization objective and the pose degree of freedom as the constraint, and bundle and adjust the initial camera pose of other panoramic images in the reprojection angle error to determine the target camera pose of the panoramic image.

[0110] In some embodiments, the apparatus further includes: The pose and point cloud acquisition module is used to acquire the target camera pose obtained after bundle adjustment and the target point cloud corresponding to the initial point cloud. The point cloud adjustment module is used to filter and / or merge the target point cloud to obtain the adjusted target point cloud; The bundle adjustment module is used to bundle and adjust the target camera poses of other panoramic images based on the target camera pose of the panoramic image and the adjusted target point cloud, and determine the adjusted target camera pose.

[0111] In some implementations, the point cloud adjustment module includes: The calculation unit is used to calculate the reprojection error between each three-dimensional point in the target point cloud and the corresponding feature point; The filtering unit is used to filter out three-dimensional points in the target point cloud based on the relationship between the reprojection error and the error threshold, so as to obtain an adjusted target point cloud.

[0112] In some implementations, the point cloud adjustment module includes: The first filtering unit is used to filter out multiple target three-dimensional points corresponding to the same feature point in the target point cloud; The merging unit is used to determine the coordinates of the merged 3D point after merging the multiple target 3D points based on the average of the coordinates of the multiple target 3D points.

[0113] In some implementations, the matching module 42 includes: A conversion unit is used to convert any two of the panoramic images into target images in hexahedral format; The feature extraction unit is used to extract features from the target image to obtain the coordinates of the feature points in the target image; The mapping unit is used to map the feature point coordinates of the target image to the corresponding panoramic image to obtain the feature points of the panoramic image.

[0114] In some implementations, the matching module 42 includes: A matching unit is used to perform feature point matching on any two panoramic images to determine selectable matching feature points; The second filtering unit is used to filter the selectable matching feature points based on the initial camera poses of any two panoramic images, and to determine the matching feature points.

[0115] In some implementations, the second screening unit includes: The computational subunit is used to calculate the relative initial pose using the initial camera poses of any two of the panoramic images, so as to determine the essential matrix between any two of the panoramic images. The filtering subunit is used to filter the optional matching feature points based on the essential matrix between any two of the panoramic images, and to determine the matching feature points.

[0116] In some embodiments, the apparatus further includes: The 3D reconstruction module is used to perform 3D reconstruction based on the at least two panoramic images and the target camera pose to determine the target 3D model.

[0117] In this embodiment, the camera pose estimation device is presented in the form of a functional unit. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0118] Further functional descriptions of the above modules are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0119] This disclosure also provides an electronic device having the above-described features. Figure 4 The camera pose estimation device shown.

[0120] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an optional embodiment of this disclosure, such as... Figure 5As shown, the electronic device may include: at least one processor 51, such as a CPU (Central Processing Unit), at least one communication interface 53, memory 54, and at least one communication bus 52. The communication bus 52 is used to enable communication between these components. The communication interface 53 may include a display screen or a keyboard; optionally, the communication interface 53 may also include a standard wired interface or a wireless interface. The memory 54 may be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 54 may also be at least one storage device located remotely from the aforementioned processor 51. The processor 51 may be combined with... Figure 5 The described apparatus has an application program stored in memory 54, and the processor 51 calls the program code stored in memory 54 to perform any of the above method steps.

[0121] The communication bus 52 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 52 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0122] The memory 54 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 54 may also include a combination of the above types of memory.

[0123] The processor 51 can be a central processing unit (CPU), a network processor (NP), or a combination of CPU and NP.

[0124] The processor 51 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0125] Optionally, memory 54 is also used to store program instructions. Processor 51 can invoke program instructions to implement the camera pose estimation method as shown in any embodiment of this application.

[0126] This disclosure also provides a non-transitory computer storage medium storing computer-executable instructions that can execute the camera pose estimation method in any of the above method embodiments. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0127] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the method embodiments are basically similar to the device and system embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the device and system embodiments.

[0128] It is understood that in the specific embodiments of this disclosure, data such as panoramic images are involved. When the above embodiments of this disclosure are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0129] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A camera pose estimation method, characterized in that, include: Acquire at least two panoramic images and the initial camera pose of the panoramic images, wherein the at least two panoramic images are panoramic images obtained by taking pictures of the same object from different points; Feature point extraction and feature point matching are performed on any two of the panoramic images to obtain the matching feature points in the panoramic images; Based on the matched feature points and the initial camera pose, an initial point cloud of the object is constructed; Based on the matching feature points, target panoramic images with constraint information below a threshold are selected from at least two panoramic images. The constraint information includes the number of matching feature points and / or the distribution of the matching feature points. The initial camera pose of the target panoramic image is fixed, and the initial camera poses of other panoramic images are bundled and adjusted based on the initial point cloud and the feature points to determine the target camera pose of the panoramic image.

2. The method of claim 1, wherein, The process of fixing the initial camera pose of the target panoramic image and, based on the initial point cloud and the feature points, bundling and adjusting the initial camera poses of other panoramic images to determine the target camera pose of the panoramic image includes: Based on the positions of each 3D point in the initial point cloud and the corresponding feature points of the 3D points in the camera coordinate system, the reprojection angle error between the 3D points and the corresponding feature points is determined. The initial camera pose of the target panoramic image is fixed, and the initial camera poses of other panoramic images with the minimum reprojection angle error are bundled and adjusted to determine the target camera pose of the panoramic image, with the minimum reprojection angle error as the optimization objective.

3. The method of claim 2, wherein, The process of fixing the initial camera pose of the target panoramic image, optimizing for minimizing the reprojection angle error, and then bundling and adjusting the initial camera poses of other panoramic images within the reprojection angle error range to determine the target camera pose of the panoramic image includes: The pose degrees of freedom of the bundle adjustment are obtained, and the pose degrees of freedom are determined based on the usage state of the panoramic camera, which is used to acquire the panoramic image. The initial camera pose of the target panoramic image is fixed. With the minimum reprojection angle error as the optimization objective and the pose degrees of freedom as constraints, the initial camera poses of other panoramic images in the reprojection angle error are bundled and adjusted to determine the target camera pose of the panoramic image.

4. The method of claim 1, wherein, The method further includes: Obtain the target camera pose after bundle adjustment and the target point cloud corresponding to the initial point cloud; The target point cloud is filtered and / or merged to obtain an adjusted target point cloud; Based on the target camera pose of the panoramic image and the adjusted target point cloud, the target camera poses of other panoramic images are bundled and adjusted to determine the adjusted target camera pose.

5. The method of claim 4, wherein, The process of filtering out the target point cloud includes: Calculate the reprojection error between each 3D point in the target point cloud and its corresponding feature point; Based on the relationship between the reprojection error and the error threshold, the three-dimensional points in the target point cloud are filtered out to obtain an adjusted target point cloud.

6. The method according to claim 4, characterized in that, Merging the target point cloud includes: Multiple target 3D points corresponding to the same feature point are selected from the target point cloud; Based on the average of the coordinates of the multiple target 3D points, the coordinates of the merged 3D point are determined.

7. The method according to claim 1, characterized in that, Feature point extraction is performed on any two of the panoramic images, including: Convert any two of the panoramic images into target images in hexahedral format; Feature extraction is performed on the target image to obtain the coordinates of the feature points in the target image; The feature point coordinates of the target image are mapped onto the corresponding panoramic image to obtain the feature points of the panoramic image.

8. The method according to claim 1, characterized in that, Feature point matching is performed on any two of the panoramic images, including: Feature point matching is performed on any two panoramic images to determine selectable matching feature points; The optional matching feature points are filtered based on the initial camera pose of any two of the panoramic images to determine the matching feature points.

9. The method according to claim 8, characterized in that, The step of filtering the selectable matching feature points based on the initial camera pose of any two of the panoramic images to determine the matching feature points includes: Using the initial camera poses of any two panoramic images, calculate the relative initial poses to determine the essential matrix between any two panoramic images; Based on the essential matrix between any two panoramic images, the optional matching feature points are filtered by epipolar geometric constraints to determine the matching feature points.

10. The method according to any one of claims 1-9, characterized in that, The method further includes: Based on the at least two panoramic images and the target camera pose, a 3D reconstruction is performed to determine the target 3D model.

11. A camera pose estimation device, characterized in that, include: The acquisition module is used to acquire at least two panoramic images and the initial camera pose of the panoramic images, wherein the at least two panoramic images are panoramic images obtained by taking pictures of the same object from different points. The matching module is used to extract and match feature points from any two panoramic images to obtain matching feature points in the panoramic images. A construction module is used to construct an initial point cloud of the object based on the matched feature points and the initial camera pose; A filtering module is used to filter out target panoramic images with constraint information below a threshold from at least two panoramic images based on the matching feature points, wherein the constraint information includes the number of matching feature points and / or the distribution of the matching feature points; An adjustment module is used to fix the initial camera pose of the target panoramic image, and to bundle and adjust the initial camera poses of other panoramic images based on the initial point cloud and the feature points, so as to determine the target camera pose of the panoramic image.

12. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the camera pose estimation method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the camera pose estimation method according to any one of claims 1-10.

Citation Information

Patent Citations

  • A monocular SLAM initialization method and system based on a wheel type encoder

    CN109671120A

  • Panoramic image-based camera pose estimation method

    CN110580720A