A method, apparatus and electronic device for determining a camera pose

Abstract

The present disclosure relates to a method and device for determining camera pose, and an electronic device, and particularly relates to the technical field of visual positioning and tracking. The method comprises: acquiring a target image captured by a camera; determining a plurality of first feature points in voxels; when there is no other feature point in the voxels between a first target feature point and a center point of the camera, performing feature point matching between the first target feature point and a pixel point in the target image, the first target feature point being any one of the plurality of first feature points; and if the first target feature point matches the pixel point in the target image, calculating an actual pose of the camera based on the first target feature point.

Description

Technical Field

[0001] This disclosure relates to the field of visual positioning and tracking technology, and in particular to a method, apparatus and electronic device for determining camera pose. Background Technology

[0002] During camera localization and tracking, a subset of images from the camera's historical imagery is selected as historical keyframes. Candidate map points from these keyframes are then matched with features in the currently captured camera image. The camera pose is calculated using the matched points from the candidate map points, thus achieving camera localization and tracking. Currently, when performing feature matching, candidate map points are selected based on the historical matching success rate of the feature points. This results in most candidate map points being from historical maps, while new map points are not used. Furthermore, when feature points from historical maps are occluded or influenced by feature points from new maps, the feature matching accuracy becomes low, leading to inaccurate determination of the camera pose. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a method, apparatus, and electronic device for determining camera pose, which can select feature points based on voxels for feature matching, thereby improving the accuracy of feature matching and the accuracy of the determined camera pose.

[0004] To achieve the above objectives, the technical solutions provided by the embodiments of this disclosure are as follows:

[0005] Firstly, a method for determining camera pose is provided, including:

[0006] Acquire the target image captured by the camera;

[0007] Identify multiple first feature points in a voxel;

[0008] When there are no other feature points in the voxel on the line connecting the first target feature point and the center point of the camera, feature point matching is performed between the first target feature point and the pixel in the target image; the first target feature point is any one of the plurality of first feature points;

[0009] If the first target feature point matches a pixel in the target image, the actual pose of the camera is calculated based on the first target feature point.

[0010] As an optional implementation of this disclosure, the method further includes:

[0011] If other feature points in the voxel exist on the line connecting the first target feature point and the center point of the camera, the first target feature point is discarded.

[0012] As an optional implementation of this disclosure, determining a plurality of first feature points in a voxel includes:

[0013] Based on the historical images captured by the camera, multiple second feature points in the voxel are determined;

[0014] Obtain the estimated pose of the camera;

[0015] From the plurality of second feature points, determine the plurality of first feature points that match the estimated pose of the camera.

[0016] As an optional implementation of this disclosure, determining the plurality of first feature points that match the estimated pose of the camera from the plurality of second feature points includes:

[0017] Determine the transformation matrix of the camera's estimated pose;

[0018] The products of the three-dimensional coordinates of the plurality of second feature points and the transformation matrix are calculated respectively to obtain the plurality of three-dimensional coordinates in the camera coordinate system;

[0019] Based on the plurality of three-dimensional coordinates and the intrinsic parameters of the camera, calculate the two-dimensional pixel coordinates of the plurality of second feature points;

[0020] Based on the two-dimensional pixel coordinates of the plurality of second feature points, the plurality of first feature points are determined from the plurality of second feature points, wherein the two-dimensional pixel coordinates corresponding to the plurality of first feature points are within the two-dimensional pixel range of the camera.

[0021] As an optional implementation of this disclosure, the intrinsic parameters of the camera include: focal length and image origin offset;

[0022] The step of calculating the two-dimensional pixel coordinates of the plurality of second feature points based on the plurality of three-dimensional coordinates and the intrinsic parameters of the camera includes:

[0023] The two-dimensional pixel coordinates of the multiple second feature points are calculated based on the multiple three-dimensional coordinates, the focal length, the image origin offset, and Formula 1 and Formula 2.

[0024] Formula 1 is:

[0025] Formula 2 is as follows:

[0026] Where u and v represent the two-dimensional pixel coordinates of the second feature point, x c y c z c The f represents the three-dimensional coordinates in the camera coordinate system.x and f y c represents focal length. x and c y c is the offset of the image origin. x c represents the offset of the camera's image origin relative to the optical center imaging point on the x-axis. y This represents the offset of the camera's image origin relative to the optical center imaging point on the y-axis.

[0027] As an optional implementation of this disclosure, determining the plurality of first feature points that match the estimated pose of the camera from the plurality of second feature points includes:

[0028] From the plurality of second feature points, determine a plurality of third feature points that match the estimated pose of the camera;

[0029] From the plurality of third feature points, determine the plurality of first feature points whose depth values ​​are within a preset depth range;

[0030] The depth value is the distance between the third feature point and the center point of the camera.

[0031] As an optional implementation of this disclosure, determining multiple second feature points in the voxels based on historical images acquired by the camera includes:

[0032] Extract multiple fourth feature points from the historical images;

[0033] For each of the plurality of fourth feature points, the following steps are performed to determine the plurality of second feature points in the voxel:

[0034] Based on the three-dimensional information of the fourth feature point, the corresponding target three-dimensional spatial unit in the voxel is determined;

[0035] If no other feature points are stored in the target three-dimensional spatial unit;

[0036] The fourth feature point is then saved to the target three-dimensional space unit as a second feature point in the voxel.

[0037] As an optional implementation of this disclosure, the plurality of fourth feature points are corner points in the historical image;

[0038] The corner points in the historical images are determined according to any of the following algorithms:

[0039] Harris corner detection algorithm, S. Thomas corner detection algorithm, and corner detection algorithm for accelerating segmented testing features.

[0040] Secondly, a device for determining camera pose is provided, comprising:

[0041] The acquisition module is used to acquire the target image captured by the camera;

[0042] A determination module is used to determine multiple first feature points in a voxel; when there are no other feature points in the voxel on the line connecting the first target feature point and the center point of the camera, the first target feature point is matched with a pixel in the target image; if the first target feature point matches a pixel in the target image, the actual pose of the camera is calculated based on the first target feature point, where the first target feature point is any one of the multiple first feature points.

[0043] As an optional implementation of this disclosure, the determining module is further configured to:

[0044] If other feature points in the voxel exist on the line connecting the first target feature point and the center point of the camera, the first target feature point is discarded.

[0045] As an optional implementation of this disclosure, the determining module is specifically used for:

[0046] Based on the historical images captured by the camera, multiple second feature points in the voxel are determined;

[0047] Obtain the estimated pose of the camera;

[0048] From the plurality of second feature points, determine the plurality of first feature points that match the estimated pose of the camera.

[0049] As an optional implementation of this disclosure, the determining module is specifically used for:

[0050] Determine the transformation matrix of the camera's estimated pose;

[0051] The products of the three-dimensional coordinates of the plurality of second feature points and the transformation matrix are calculated respectively to obtain the plurality of three-dimensional coordinates in the camera coordinate system;

[0052] Based on the plurality of three-dimensional coordinates and the intrinsic parameters of the camera, calculate the two-dimensional pixel coordinates of the plurality of second feature points;

[0053] Based on the two-dimensional pixel coordinates of the plurality of second feature points, the plurality of first feature points are determined from the plurality of second feature points, wherein the two-dimensional pixel coordinates corresponding to the plurality of first feature points are within the two-dimensional pixel range of the camera.

[0054] As an optional implementation of this disclosure, the intrinsic parameters of the camera include: focal length and image origin offset;

[0055] The determining module is specifically used for:

[0056] The two-dimensional pixel coordinates of the multiple second feature points are calculated based on the multiple three-dimensional coordinates, the focal length, the image origin offset, and Formula 1 and Formula 2.

[0057] Formula 1 is:

[0058] Formula 2 is as follows:

[0059] Where u and v represent the two-dimensional pixel coordinates of the second feature point, x c y c z c The f represents the three-dimensional coordinates in the camera coordinate system. x and f y c represents focal length. x and c y c is the offset of the image origin. x c represents the offset of the camera's image origin relative to the optical center imaging point on the x-axis. y This represents the offset of the camera's image origin relative to the optical center imaging point on the y-axis.

[0060] As an optional implementation of this disclosure, the determining module is specifically used for:

[0061] From the plurality of second feature points, determine a plurality of third feature points that match the estimated pose of the camera;

[0062] From the plurality of third feature points, determine the plurality of first feature points whose depth values ​​are within a preset depth range;

[0063] The depth value is the distance between the third feature point and the center point of the camera.

[0064] As an optional implementation of this disclosure, the determining module is specifically used for:

[0065] Extract multiple fourth feature points from the historical images;

[0066] For each of the plurality of fourth feature points, the following steps are performed to determine the plurality of second feature points in the voxel:

[0067] Based on the three-dimensional information of the fourth feature point, the corresponding target three-dimensional spatial unit in the voxel is determined;

[0068] If no other feature points are stored in the target three-dimensional spatial unit;

[0069] The fourth feature point is then saved to the target three-dimensional space unit as a second feature point in the voxel.

[0070] As an optional implementation of this disclosure, the plurality of fourth feature points are corner points in the historical image;

[0071] The corner points in the historical images are determined according to any of the following algorithms:

[0072] Harris corner detection algorithm, S. Thomas corner detection algorithm, and corner detection algorithm for accelerating segmented testing features.

[0073] Thirdly, an electronic device is provided, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method for determining camera pose as described in the first aspect or any alternative embodiment thereof.

[0074] Fourthly, a computer-readable storage medium is provided, comprising: storing a computer program on the computer-readable storage medium, wherein the computer program, when executed by a processor, implements the method for determining camera pose as described in the first aspect or any alternative embodiment thereof.

[0075] Fifthly, a computer program product is provided, characterized in that it includes: when the computer program product is run on a computer, causing the computer to implement the method for determining camera pose as described in the first aspect or any optional embodiment thereof.

[0076] Compared with the prior art, the technical solution provided in this disclosure has the following advantages: multiple first feature points in a voxel can be obtained first, and then it can be determined whether there are other feature points in the voxel on the line connecting the first target feature point (any one of the multiple first feature points) and the center point of the camera. If there are no other feature points in the voxel, it means that the first target feature point is not occluded or affected by other feature points. At this time, when the first target feature point is matched with the pixel in the target image, the actual pose of the camera is calculated based on the first target feature point without being affected by other feature points. In this way, the feature matching accuracy is higher and the determined camera pose is more accurate. Attached Figure Description

[0077] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0078] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0079] Figure 1 This is a schematic diagram of voxel 11 in one embodiment of the present disclosure;

[0080] Figure 2 This is a flowchart illustrating a method for determining camera pose in one embodiment of the present disclosure. Figure One ;

[0081] Figure 3 A schematic diagram of candidate feature point selection provided in an embodiment of this disclosure. Figure One ;

[0082] Figure 4 A schematic diagram of candidate feature point selection provided in an embodiment of this disclosure. Figure Two ;

[0083] Figure 5 This is a flowchart illustrating a method for determining camera pose in one embodiment of the present disclosure. Figure Two ;

[0084] Figure 6 This is a schematic diagram illustrating feature point selection based on the estimated pose of a camera, according to an embodiment of this disclosure.

[0085] Figure 7 This is a structural block diagram of a device for determining camera pose according to an embodiment of the present disclosure;

[0086] Figure 8 This is a schematic diagram of a hardware structure in one embodiment of the present disclosure. Detailed Implementation

[0087] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0088] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0089] During camera localization and tracking, a portion of the camera's historical images are selected as historical keyframes. Candidate map points are then selected from these historical keyframes and matched with the currently captured camera images. The camera pose is calculated using the matched points from the candidate map points, thereby achieving camera localization and tracking.

[0090] Currently, when performing feature matching, candidate map points are selected based on the historical matching success rate of the feature points. This results in most of the candidate map points used in feature matching being feature points from historical maps, while new map points are not used. Furthermore, when feature points in historical maps are occluded or affected by new map points, the feature matching accuracy is low, and the determined camera pose is inaccurate.

[0091] To address the aforementioned issues, this disclosure provides a method, apparatus, and electronic device for determining camera pose. By utilizing a voxel-based approach, feature points that are not affected by other feature points can be selected for feature matching, and pose can be calculated. This results in higher feature matching accuracy and more accurate determined camera pose.

[0092] To better understand the solutions in the embodiments of this disclosure, voxels will be introduced first:

[0093] A voxel is a data structure that can represent three-dimensional data. A voxel can be divided into multiple three-dimensional spatial units, each of which can be a lattice (i.e., a cube). Each three-dimensional spatial unit is conceptually similar to a pixel in a two-dimensional image. Pixels are used in the image data of two-dimensional computer images, and some three-dimensional displays use voxels to describe their resolution. Each three-dimensional spatial unit can store the data of a feature point.

[0094] The size of the three-dimensional spatial units within a voxel can be flexibly divided. The smaller the three-dimensional spatial units within a voxel, the higher the voxel's precision; conversely, the larger the three-dimensional spatial units within a voxel, the lower the voxel's precision. For example... Figure 1 The diagram shown is a schematic diagram of a voxel 11 in one embodiment of the present disclosure, wherein each small cube in the voxel is a three-dimensional spatial unit 111, and each three-dimensional spatial unit 111 can store the data of a feature point.

[0095] The method for determining camera pose provided in this embodiment can be implemented by a device or electronic device for determining camera pose. The device for determining camera pose can be a functional module or functional entity in the electronic device used to implement the method for determining camera pose.

[0096] The aforementioned electronic devices may be mobile phones, tablet computers, laptops, handheld computers, vehicle terminals, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), personal computers (PCs), etc., and this disclosure does not specifically limit them.

[0097] like Figure 2 The diagram shown is a flowchart illustrating a method for determining camera pose in an embodiment of this disclosure. Figure One The method includes the following steps:

[0098] 201. Acquire the target image captured by the camera.

[0099] In this context, acquiring the target image captured by the camera can refer to acquiring the real-time image captured by the camera, where the target image is the currently acquired image.

[0100] 202. Identify multiple first feature points in a voxel.

[0101] In some embodiments of this disclosure, a plurality of first feature points in voxels can be determined based on historical images captured by a camera. These plurality of first feature points can be feature points extracted from historical images, and these feature points in the historical images can be corner points in the historical images.

[0102] In some embodiments, the plurality of first feature points may be all corner points determined from historical images; in other embodiments, the plurality of first feature points may be some corner points determined from historical images.

[0103] In this embodiment of the disclosure, corner points in the historical image are determined according to any of the following algorithms:

[0104] Harris corner detection algorithm, Shi-Tomasi corner detection algorithm, and FAST (Fast-Segmented Test Feature) corner detection algorithm.

[0105] 203. Determine whether there are other feature points in the voxel on the line connecting the first target feature point and the center point of the camera.

[0106] Wherein, the first target feature point is any one of a plurality of first feature points, and other feature points in the voxel refer to any feature point in the voxel other than the first target feature point.

[0107] In this embodiment of the disclosure, it can be understood that step 203 is performed for each of the multiple first feature points to determine multiple candidate feature points, and the multiple candidate feature points are matched with the pixels in the target image currently acquired by the camera. Based on the feature points among the multiple candidate feature points that match the pixels in the target image currently acquired by the camera, the actual pose of the camera is calculated.

[0108] If there are other feature points in the voxel on the line connecting the first target feature point and the center point of the camera, it means that the first target feature point is occluded or affected by other feature points. In this case, the first target feature point will not be used as a candidate feature point and will be discarded, i.e., the following 204 will be executed.

[0109] The center point of the camera is the center point of the camera lens.

[0110] For example, such as Figure 3 The image shown is a schematic diagram of candidate feature point selection provided in an embodiment of this disclosure. Figure One , Figure 3 Point 31 is the center point of the camera, and point 32 is the first target feature point mentioned above. As can be seen from the figure, there is also point 33 on the line connecting point 31 and point 32. Point 33 is a feature point other than the first target feature point in the voxel. From the camera's field of view, point 33 will occlude point 32. Therefore, point 32 will not be used as a candidate feature point for subsequent feature matching and actual pose calculation of the camera.

[0111] Correspondingly, if there are no other feature points in the voxel on the line connecting the first target feature point and the center point of the camera, it means that the first target feature point has not been occluded or affected by other points. The first target feature point can be used as a candidate feature point. The first target feature point is matched with the pixel points in the target image, and the actual pose of the camera is calculated, i.e., the following 205 is executed.

[0112] For example, such as Figure 4 The image shown is a schematic diagram of candidate feature point selection provided in an embodiment of this disclosure. Figure Two , Figure 4 Point 41 is the center point of the camera, and point 42 is the first target feature point mentioned above. As can be seen from the figure, there are no feature points in any voxel on the line connecting points 41 and 42. Point 42 will not be occluded from the camera's field of view. Therefore, point 42 can be used as a candidate feature point for subsequent feature matching and actual pose calculation of the camera.

[0113] 204. Discard the first target feature point.

[0114] 205. If the first target feature point matches a pixel in the target image, then calculate the actual pose of the camera based on the first target feature point.

[0115] Once the first target feature point is determined as a candidate feature point, feature matching can be performed one by one between the first target feature point and the pixels in the target image. After feature matching, the actual pose of the camera can be calculated based on the first target feature point.

[0116] In this embodiment of the disclosure, pose refers to position and attitude. The actual pose of the camera is the current position coordinate information of the camera in the actual space, as well as the attitude information.

[0117] In some embodiments, when performing feature matching between the first target feature point and pixels in the target image, feature information of each pixel in the target image is first extracted, then the feature information of each pixel is traversed and compared with the feature information of the first target feature point to determine the similarity parameter for each pixel, and then it is determined whether the maximum similarity parameter is greater than a preset similarity threshold. If the maximum similarity parameter is greater than the preset similarity threshold, then it is determined that the pixel corresponding to the maximum similarity parameter matches the first target feature point.

[0118] In implementing the above method, after the judgment step based on step 203, all unoccluded first feature points among the multiple first feature points can be added to a candidate point set. Based on the candidate point set obtained after traversing multiple first feature points, subsequent feature matching and actual camera pose calculation can be performed. Correspondingly, during feature matching, the candidate feature points in this candidate point set can be traversed to determine multiple feature points that match pixels in the target image, and the actual camera pose can be calculated based on these matched feature points. The method for determining camera pose provided in this embodiment can first acquire multiple first feature points in voxels determined from historical images, and then determine whether there are other feature points in voxels on the line connecting the first target feature point (any one of the multiple first feature points) and the center point of the camera. If there are no other feature points in voxels, it means that the first target feature point has not been occluded or affected by other feature points. At this time, when the first target feature point is matched with a pixel in the target image, the actual pose of the camera is calculated based on the first target feature point without being affected by other feature points. This results in higher feature matching accuracy and more accurate determined camera pose.

[0119] Compared to the prior art, which selects candidate map points based on the historical matching success rate of feature points, the method described in this embodiment uses voxels to filter feature points when feature points in the historical map are occluded or affected by feature points in the new map. This removes occluded or affected feature points and improves the accuracy of feature matching and camera pose calculation.

[0120] like Figure 5 The diagram shown is a flowchart illustrating a method for determining camera pose in an embodiment of this disclosure. Figure Two The method includes the following steps:

[0121] 501. Acquire the target image captured by the camera.

[0122] 502. Based on historical images captured by the camera, determine multiple second feature points in the voxels.

[0123] Among them, the multiple second feature points can be feature points extracted from historical images, and these feature points in historical images can be corner points in historical images.

[0124] In some embodiments, the plurality of first feature points may be all corner points determined from historical images; in other embodiments, the plurality of first feature points may be some corner points determined from historical images.

[0125] In some embodiments, determining multiple second feature points in a voxel based on historical images captured by the camera can be achieved through the following steps:

[0126] 502a. Extract multiple fourth feature points from historical images. Perform steps 502b to 502e for each of the multiple fourth feature points to determine multiple second feature points in the voxels.

[0127] Here, the plurality of fourth feature points are corner points in the historical image. For example, they could be all corner points determined from the historical image. In this case, the plurality of second feature points are a subset of corner points selected from all corner points determined from the historical image according to the steps 502b to 502e.

[0128] 502b. Based on the three-dimensional information of the fourth feature point, determine the corresponding target three-dimensional spatial unit in the voxel.

[0129] The three-dimensional information can refer to three-dimensional coordinate information.

[0130] 502c. Determine whether other feature points are preserved in the target three-dimensional spatial unit.

[0131] If other feature points are saved in the target 3D spatial unit, execute 502d to discard the fourth feature point; if no other feature points are saved in the target 3D spatial unit, execute 502e to save the target feature point to the target 3D spatial unit as a second feature point in a voxel.

[0132] 502d, Discard the fourth feature point.

[0133] 502e. Save the fourth feature point to the target three-dimensional space unit as a second feature point in a voxel.

[0134] In the above embodiments, when storing multiple fourth feature points extracted from historical images into voxels, the fourth feature points need to be stored in three-dimensional spatial units that match the three-dimensional information. Before this, it is necessary to determine whether other feature points have already been stored in the three-dimensional spatial unit. If other feature points have already been stored, they will not be stored again. In this way, multiple second feature points in the voxel can be determined. Each second feature point corresponds to a three-dimensional spatial unit in the voxel.

[0135] 503. Obtain the estimated pose of the camera.

[0136] In this embodiment of the disclosure, the estimated pose of the camera can be the pose estimated based on the image previously captured by the camera.

[0137] 504. Determine multiple third feature points from multiple second feature points that match the camera's predicted pose.

[0138] In some embodiments, multiple second feature points in voxels can be determined from historical images, and then multiple third feature points that match the estimated pose of the current camera can be selected to determine multiple third feature points in voxels.

[0139] When determining multiple third feature points that match the camera's estimated pose from multiple second feature points, the field of view of the camera under the estimated pose can be determined first, and multiple third feature points within the field of view can be determined from multiple second feature points in the voxel.

[0140] In some embodiments, determining a plurality of third feature points that match the estimated pose of the camera from a plurality of second feature points may include the following steps 504a, 504b, 504c, and 504d:

[0141] 504a. Determine the transformation matrix for the camera's estimated pose.

[0142] The camera's estimated pose information is represented in matrix form, from which the transformation matrix of the camera's estimated pose can be obtained.

[0143] 504b. Calculate the product of the three-dimensional coordinates of multiple second feature points and the transformation matrix to obtain multiple three-dimensional coordinates in the camera coordinate system.

[0144] The camera coordinate system described above is a three-dimensional rectangular coordinate system with the center of the camera as the origin and the optical axis as the z-axis. This rectangular coordinate system includes the x-axis, y-axis, and z-axis.

[0145] 504c. Calculate the two-dimensional pixel coordinates of multiple second feature points based on multiple three-dimensional coordinates and camera intrinsic parameters.

[0146] The camera's intrinsic parameters may include: focal length and image origin offset;

[0147] The above calculation of the two-dimensional pixel coordinates of multiple second feature points based on multiple three-dimensional coordinates and camera intrinsic parameters may include: calculating the two-dimensional pixel coordinates of multiple second feature points based on multiple three-dimensional coordinates, focal length, image origin offset, and Formula 1 and Formula 2.

[0148] Formula 1 is:

[0149] Formula 2 is as follows:

[0150] Where u and v represent the two-dimensional pixel coordinates of the second feature point, x c y c z c f represents the three-dimensional coordinates in the camera coordinate system. x and f y c represents focal length. x and c y c is the offset of the image origin. x c represents the offset of the camera's image origin relative to the optical center imaging point on the x-axis. y This represents the offset of the camera's image origin relative to the optical center imaging point along the y-axis. The aforementioned x... c The parameters representing the x-axis and y-axis in the 3D coordinate system of the camera coordinate system. c The z-axis represents the parameter of the y-axis in the 3D coordinate system of the camera coordinate system. c This represents the parameter of the z-axis in the three-dimensional coordinate system of the camera. x f represents the focal length corresponding to the x-axis. y This represents the focal length corresponding to the y-axis.

[0151] In the process of calculating the two-dimensional pixel coordinates of multiple second feature points based on multiple three-dimensional coordinates, focal length, image origin offset, and Formula 1 and Formula 2, for each three-dimensional coordinate in the camera coordinate system, calculations can be performed with focal length, image origin offset, and Formula 1 and Formula 2 to obtain the two-dimensional pixel coordinates corresponding to each second feature point for each three-dimensional coordinate.

[0152] 504d. Based on the two-dimensional pixel coordinates of multiple second feature points, determine multiple third feature points from the multiple second feature points.

[0153] In this context, the two-dimensional pixel coordinates corresponding to multiple third feature points lie within the two-dimensional pixel range of the camera. This two-dimensional pixel range can be determined based on the camera's resolution.

[0154] like Figure 6 The diagram shown is a schematic representation of a feature point selection method based on the estimated pose of a camera, according to an embodiment of this disclosure. Figure 6 Points 63, 64, 65, and 66 are all feature points in a voxel. Figure 6 The field of view of the camera 61 is shown in the three-dimensional space 62. It can be seen that... Figure 6 Midpoints 63 and 64 lie within the three-dimensional space 62; therefore, points 63 and 64 are feature points that match the camera's estimated pose (i.e., the aforementioned third feature points). Figure 6 Midpoints 65 and 66 are outside the 3D space 62, therefore points 65 and 66 do not match the camera's predicted pose.

[0155] In the above embodiments, multiple second feature points in the voxels are first screened based on the camera's estimated pose, selecting feature points within the camera's field of view under the estimated pose. Since feature points outside the camera's field of view typically have a low matching probability during actual matching, and calculating the camera's actual pose based on these points is inaccurate, a preliminary screening of the multiple second feature points based on the camera's estimated pose is performed to select multiple third feature points with a higher matching probability. This improves the probability of subsequent feature matching and increases the accuracy of calculating the actual pose.

[0156] 505. From multiple third feature points, determine multiple first feature points whose depth values ​​are within a preset depth range.

[0157] In some embodiments, multiple second feature points in voxels can be determined from historical images, and then multiple third feature points that match the estimated pose of the camera can be selected from the multiple second feature points based on the estimated pose of the current camera. Further, multiple first feature points whose depth values ​​are within a preset depth range can be determined from the multiple third feature points.

[0158] The depth value refers to the distance between the third feature point and the center point of the camera.

[0159] In one optional implementation, the depth value being within a preset depth range means that the depth value is less than or equal to a first preset value.

[0160] In one optional implementation, the depth value being within a preset depth range means that the depth value is less than or equal to a first preset value and greater than or equal to a second preset value.

[0161] Optionally, the first preset value can be 50 meters, and the second preset value can be 0.1 meters. The first and second preset values ​​can also be set to other values. The specific first and second preset values ​​can be determined according to the camera's field of view and the requirements of calculation accuracy. This embodiment does not limit the specific values.

[0162] In the above embodiments, by setting a preset depth range, feature points that are too close to or too far from the camera are removed from a plurality of third feature points, and feature points within a certain distance range are retained as the plurality of first feature points. This can improve the probability of matching during subsequent feature matching.

[0163] 506. Determine whether there are other feature points in the voxel on the line connecting the first target feature point and the center point of the camera.

[0164] If there are no other feature points in the voxel on the line connecting the first target feature point and the center point of the camera, execute step 508 below to perform feature point matching between the first target feature point and the pixel in the target image. The first target feature point is any one of a plurality of first feature points. If there are other feature points in the voxel on the line connecting the first target feature point and the center point of the camera, execute step 507 below to discard the first target feature point.

[0165] 507. Discard the first target feature point.

[0166] 508. If the first target feature point matches a pixel in the target image, then calculate the actual pose of the camera based on the first target feature point.

[0167] Based on the descriptions of 506 to 508 above, please refer to the relevant descriptions of 203 to 205 above, which will not be repeated here.

[0168] The method for determining camera pose provided in this embodiment can first acquire multiple first feature points in voxels determined from historical images, and then determine whether there are other feature points in voxels on the line connecting the first target feature point (any one of the multiple first feature points) and the center point of the camera. If there are no other feature points in voxels, it means that the first target feature point has not been occluded or affected by other feature points. At this time, when the first target feature point is matched with a pixel in the target image, the actual pose of the camera is calculated based on the first target feature point without being affected by other feature points. This results in higher feature matching accuracy and more accurate determined camera pose.

[0169] like Figure 7 The diagram shown is a structural block diagram of a device for determining camera pose according to an embodiment of this disclosure. The method includes:

[0170] The acquisition module 701 is used to acquire the target image captured by the camera;

[0171] The determining module 702 is used to determine a plurality of first feature points in a voxel; when there are no other feature points in the voxel on the line connecting the first target feature point and the center point of the camera, the first target feature point is matched with the pixel in the target image; if the first target feature point matches the pixel in the target image, the actual pose of the camera is calculated based on the first target feature point, wherein the first target feature point is any one of the plurality of first feature points.

[0172] As an optional implementation of this disclosure, the determining module 702 is further configured to:

[0173] If other feature points in the voxel exist on the line connecting the first target feature point and the center point of the camera, the first target feature point is discarded.

[0174] As an optional implementation of this disclosure, the determining module 702 is specifically used for:

[0175] Based on the historical images captured by the camera, multiple second feature points in the voxel are determined;

[0176] Obtain the estimated pose of the camera;

[0177] From the plurality of second feature points, determine the plurality of first feature points that match the estimated pose of the camera.

[0178] As an optional implementation of this disclosure, the determining module 702 is specifically used for:

[0179] Determine the transformation matrix of the camera's estimated pose;

[0180] The products of the three-dimensional coordinates of the plurality of second feature points and the transformation matrix are calculated respectively to obtain the plurality of three-dimensional coordinates in the camera coordinate system;

[0181] Based on the plurality of three-dimensional coordinates and the intrinsic parameters of the camera, calculate the two-dimensional pixel coordinates of the plurality of second feature points;

[0182] Based on the two-dimensional pixel coordinates of the plurality of second feature points, the plurality of first feature points are determined from the plurality of second feature points, wherein the two-dimensional pixel coordinates corresponding to the plurality of first feature points are within the two-dimensional pixel range of the camera.

[0183] As an optional implementation of this disclosure, the intrinsic parameters of the camera include: focal length and image origin offset;

[0184] The determining module 702 is specifically used for:

[0185] The two-dimensional pixel coordinates of the multiple second feature points are calculated based on the multiple three-dimensional coordinates, the focal length, the image origin offset, and Formula 1 and Formula 2.

[0186] Formula 1 is:

[0187] Formula 2 is as follows:

[0188] Where u and v represent the two-dimensional pixel coordinates of the second feature point, x c y c z c The f represents the three-dimensional coordinates in the camera coordinate system. x and f y c represents focal length. x and c y c is the offset of the image origin. x c represents the offset of the camera's image origin relative to the optical center imaging point on the x-axis. y This represents the offset of the camera's image origin relative to the optical center imaging point on the y-axis.

[0189] As an optional implementation of this disclosure, the determining module 702 is specifically used for:

[0190] From the plurality of second feature points, determine a plurality of third feature points that match the estimated pose of the camera;

[0191] From the plurality of third feature points, determine the plurality of first feature points whose depth values ​​are within a preset depth range;

[0192] The depth value is the distance between the third feature point and the center point of the camera.

[0193] As an optional implementation of this disclosure, the determining module 702 is specifically used for:

[0194] Extract multiple fourth feature points from the historical images;

[0195] For each of the plurality of fourth feature points, the following steps are performed to determine the plurality of second feature points in the voxel:

[0196] Based on the three-dimensional information of the fourth feature point, the corresponding target three-dimensional spatial unit in the voxel is determined;

[0197] If no other feature points are stored in the target three-dimensional spatial unit;

[0198] The fourth feature point is then saved to the target three-dimensional space unit as a second feature point in the voxel.

[0199] As an optional implementation of this disclosure, the plurality of fourth feature points are corner points in the historical image;

[0200] The corner points in the historical images are determined according to any of the following algorithms:

[0201] Harris corner detection algorithm, S. Thomas corner detection algorithm, and corner detection algorithm for accelerating segmented testing features.

[0202] like Figure 8 The diagram shows a hardware structure of an electronic device according to an embodiment of this disclosure. The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the various processes of the method for determining camera pose in the above-described method embodiments. Since the same technical effects can be achieved, further details are omitted here to avoid repetition.

[0203] This invention provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, it implements the various processes of the method for determining camera pose in the above-described method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0204] The computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0205] This invention provides a computer program product that stores a computer program. When the computer program is executed by a processor, it implements the various processes of the method for determining camera pose described in the above method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0206] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0207] In this disclosure, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0208] In this disclosure, memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0209] In this disclosure, computer-readable media includes both permanent and non-permanent, removable and non-removable storage media. Storage media can store information using any method or technology; the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0210] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0211] The above are merely specific embodiments of this disclosure, enabling those skilled in the art to understand or implement this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for determining camera pose, characterized in that, include: Acquire the target image captured by the camera; Identify multiple first feature points in a voxel; When there are no other feature points in the voxel on the line connecting the first target feature point and the center point of the camera, the first target feature point is matched with the pixel in the target image. The first target feature point is any one of the plurality of first feature points. If the first target feature point matches a pixel in the target image, then the actual pose of the camera is calculated based on the first target feature point; The determination of multiple first feature points in the voxel includes: Based on the historical images captured by the camera, multiple second feature points in the voxel are determined; Obtain the estimated pose of the camera; From the plurality of second feature points, determine the plurality of first feature points that match the estimated pose of the camera.

2. The method according to claim 1, characterized in that, The method further includes: If other feature points in the voxel exist on the line connecting the first target feature point and the center point of the camera, the first target feature point is discarded.

3. The method according to claim 1, characterized in that, The step of determining the plurality of first feature points that match the estimated pose of the camera from the plurality of second feature points includes: Determine the transformation matrix of the camera's estimated pose; The products of the three-dimensional coordinates of the plurality of second feature points and the transformation matrix are calculated respectively to obtain the plurality of three-dimensional coordinates in the camera coordinate system; Based on the plurality of three-dimensional coordinates and the intrinsic parameters of the camera, calculate the two-dimensional pixel coordinates of the plurality of second feature points; Based on the two-dimensional pixel coordinates of the plurality of second feature points, the plurality of first feature points are determined from the plurality of second feature points, wherein the two-dimensional pixel coordinates corresponding to the plurality of first feature points are within the two-dimensional pixel range of the camera.

4. The method according to claim 3, characterized in that, The camera's intrinsic parameters include: focal length and image origin offset; The step of calculating the two-dimensional pixel coordinates of the plurality of second feature points based on the plurality of three-dimensional coordinates and the intrinsic parameters of the camera includes: The two-dimensional pixel coordinates of the multiple second feature points are calculated based on the multiple three-dimensional coordinates, the focal length, the image origin offset, and Formula 1 and Formula 2. Formula 1 is: ; Formula 2 is as follows: ; Where u and v represent the two-dimensional pixel coordinates of the second feature point, x c y c z c The f represents the three-dimensional coordinates in the camera coordinate system. x and f y c represents focal length. x and c y c is the offset of the image origin. x c represents the offset of the camera's image origin relative to the optical center imaging point on the x-axis. y This represents the offset of the camera's image origin relative to the optical center imaging point on the y-axis.

5. The method according to claim 1, characterized in that, The step of determining the plurality of first feature points that match the camera's estimated pose from the plurality of second feature points includes: From the plurality of second feature points, determine a plurality of third feature points that match the estimated pose of the camera; From the plurality of third feature points, determine the plurality of first feature points whose depth values ​​are within a preset depth range; The depth value is the distance between the third feature point and the center point of the camera.

6. The method according to claim 1, characterized in that, The step of determining multiple second feature points in the voxel based on historical images acquired by the camera includes: Extract multiple fourth feature points from the historical images; For each of the plurality of fourth feature points, the following steps are performed to determine the plurality of second feature points in the voxel: Based on the three-dimensional information of the fourth feature point, the corresponding target three-dimensional spatial unit in the voxel is determined; If no other feature points are saved in the target three-dimensional space unit, then the fourth feature point is saved to the target three-dimensional space unit as a second feature point in the voxel.

7. The method according to claim 6, characterized in that, The plurality of fourth feature points are corner points in the historical image; The corner points in the historical images are determined according to any of the following algorithms: Harris corner detection algorithm, S. Thomas corner detection algorithm, and corner detection algorithm for accelerating segmented testing features.

8. A device for determining camera pose, characterized in that, include: The acquisition module is used to acquire the target image captured by the camera; The determination module is used to determine multiple first feature points in a voxel; If there are no other feature points in the voxel on the line connecting the first target feature point and the center point of the camera, the first target feature point is matched with the pixel in the target image; if the first target feature point matches the pixel in the target image, the actual pose of the camera is calculated based on the first target feature point, where the first target feature point is any one of the plurality of first feature points. The determination of multiple first feature points in the voxel includes: Based on the historical images captured by the camera, multiple second feature points in the voxel are determined; Obtain the estimated pose of the camera; From the plurality of second feature points, determine the plurality of first feature points that match the estimated pose of the camera.

9. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method for determining camera pose as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, include: A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method for determining camera pose as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, include: When the computer program product is run on a computer, the computer enables the computer to implement the method for determining camera pose as described in any one of claims 1 to 7.

Images