Pose determination method and device, electronic equipment and storage medium

Abstract

The invention relates to a pose determination method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a target image shot by a camera, the target image comprising a plurality of 2D feature points; obtaining 3D feature points matched with the plurality of 2D feature points from a target map to obtain a plurality of groups of 2D-3D point pairs; establishing a target geometric model between the 2D-3D point pair and the pose of the camera; a solution space of the pose in the target geometric model is a high-dimensional linear space, the 2D-3D point pair is a hyperplane in the solution space, and the solution space comprises a plurality of nodes; obtaining a target node in the solution space; wherein a target node is determined based on the number of hyperplanes intersected with the nodes in the solution space; and taking the hyperplane intersected with the target node as a target hyperplane, taking the 2D-3D point pair corresponding to the target hyperplane as an inner point, and determining the pose of the camera based on the inner point. In this way, the inner points are determined from the obtained multiple groups of 2D-3D point pairs, and the pose of the camera can be determined more accurately through the determined inner points.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a method, device, electronic device and storage medium for determining a posture. Background Art

[0002] In scenarios such as SLAM (Simultaneous Localization And Mapping) and visual navigation, the camera pose is usually tracked when capturing scene images to facilitate matching of scene images with three-dimensional maps and achieve further mapping or navigation functions.

[0003] Related visual positioning solutions usually adopt a positioning method based on 2D-3D feature point matching. In the part of finding matches and estimating poses, the most common solution is to perform nearest neighbor matching of feature points between the query frame and the reference frame retrieved in the coarse positioning, and use a threshold-based similarity point filtering strategy to perform inlier screening of 2D-3D matching point pairs to realize pose calculation.

[0004] However, in scenes with little texture or repeated texture, the 2D-3D feature point matching obtained by nearest neighbor search usually contains a large number of false matches, resulting in a low ratio of inliers, which can easily cause the camera's pose positioning to fail. Summary of the invention

[0005] The present disclosure provides a method, device, electronic device and storage medium for determining a posture.

[0006] According to a first aspect of the present disclosure, a method for determining a position and posture is provided, the method comprising:

[0007] Acquire a target image captured by a camera, wherein the target image includes a plurality of 2D feature points;

[0008] Acquire 3D feature points matching the plurality of 2D feature points from the target map to obtain a plurality of groups of 2D-3D point pairs;

[0009] Establishing a target geometric model between a 2D-3D point pair and the camera's posture; a solution space of the posture in the target geometric model is a high-dimensional linear space, the 2D-3D point pair is a hyperplane in the solution space, and the solution space includes a plurality of nodes;

[0010] Acquire a target node in the solution space; wherein the target node is determined based on the number of hyperplanes intersecting with the nodes in the solution space;

[0011] A hyperplane intersecting the target node is used as a target hyperplane, and a 2D-3D point pair corresponding to the target hyperplane is used as an inner point, and the position and posture of the camera is determined based on the inner point.

[0012] Optionally, obtaining the target node in the solution space includes:

[0013] Dividing the nodes in the solution space into multiple levels of sub-nodes; the smallest node in the sub-nodes meets the target granularity index;

[0014] The minimum node with the largest number of intersecting hyperplanes among the minimum nodes is obtained, and the minimum node with the largest number of intersecting hyperplanes is determined as the target node.

[0015] Optionally, the method further comprises:

[0016] Obtain the maximum number of hyperplanes intersecting the minimum node among the first N nodes, and save the maximum number of hyperplanes as a saved value; wherein N is a positive integer;

[0017] Obtain the number of target hyperplanes intersecting with the N+1th node in the solution space;

[0018] When the number of the target hyperplanes is less than the stored value, the N+1th node is excluded.

[0019] Optionally, the method further comprises:

[0020] When the target number of hyperplanes is greater than the stored value, the stored value is updated to the target number of hyperplanes, and the number of hyperplanes intersecting with the smallest node in the N+1th node is obtained.

[0021] Optionally, the method further comprises:

[0022] Obtaining parameters of a first hyperplane and parameters of a second hyperplane in the solution space;

[0023] When the difference between the parameters of the first hyperplane and the parameters of the second hyperplane is less than a threshold, the first hyperplane and the second hyperplane are fused into a target hyperplane.

[0024] Optionally, the method further comprises:

[0025] Acquiring gravity direction information of the target image;

[0026] The target image is aligned with the target map in a gravity direction based on the gravity direction information.

[0027] Optionally, the method further comprises:

[0028] Acquiring positioning information of the camera when taking the target image;

[0029] A target area range is determined based on the positioning information, and nodes outside the target area range are excluded.

[0030] According to a second aspect of the present disclosure, a posture determination device is provided, characterized in that the device comprises:

[0031] An image acquisition module is used to acquire a target image captured by a camera, wherein the target image includes a plurality of 2D feature points;

[0032] A feature point acquisition module, used to acquire 3D feature points matching the multiple 2D feature points from the target map to obtain multiple groups of 2D-3D point pairs;

[0033] A model building module, used to build a target geometric model between a 2D-3D point pair and the camera's posture; the solution space of the posture in the target geometric model is a high-dimensional linear space, the 2D-3D point pair is a hyperplane in the solution space, and the solution space includes a plurality of nodes;

[0034] A target node acquisition module, used to acquire a target node in the solution space; wherein the target node is determined based on the number of hyperplanes intersecting with the nodes in the solution space;

[0035] The pose determination module is used to use the hyperplane intersecting with the target node as the target hyperplane, and the 2D-3D point pair corresponding to the target hyperplane as the inner point, and determine the pose of the camera based on the inner point.

[0036] Optionally, the target node acquisition module is further configured to:

[0037] Dividing the nodes in the solution space into multiple levels of sub-nodes; the smallest node in the sub-nodes meets the target granularity index;

[0038] The minimum node with the largest number of intersecting hyperplanes among the minimum nodes is obtained, and the minimum node with the largest number of intersecting hyperplanes is determined as the target node.

[0039] Optionally, the device further comprises:

[0040] A saved value determination module, used to obtain the maximum number of hyperplanes intersecting with the minimum node among the first N nodes, and use the maximum number of hyperplanes as a saved value; wherein N is a positive integer;

[0041] A target hyperplane quantity acquisition module is used to acquire the number of target hyperplanes intersecting with the N+1th node in the solution space;

[0042] The node processing module is used to exclude the N+1th node when the number of the target hyperplanes is less than the saved value.

[0043] Optionally, the device further comprises:

[0044] A saved value updating module is used to update the saved value to the target number of hyperplanes when the target number of hyperplanes is greater than the saved value, and obtain the number of hyperplanes intersecting with the smallest node in the N+1th node.

[0045] Optionally, the device further comprises:

[0046] A parameter acquisition module, used to acquire parameters of the first hyperplane and the second hyperplane in the solution space;

[0047] The plane fusion module is used to fuse the first hyperplane and the second hyperplane into a target hyperplane when the difference between the parameters of the first hyperplane and the parameters of the second hyperplane is less than a threshold.

[0048] Optionally, the device further comprises:

[0049] A gravity direction information acquisition module, used to acquire the gravity direction information of the target image;

[0050] An alignment module is used to align the target image with the target map in the direction of gravity based on the gravity direction information.

[0051] Optionally, the device further comprises:

[0052] A positioning information acquisition module, used to acquire the positioning information of the camera when shooting the target image;

[0053] The area determination module is used to determine the target area range based on the positioning information and exclude nodes outside the target area range.

[0054] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0055] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above method of the present disclosure is implemented.

[0056] The posture determination method, device, electronic device and storage medium provided by the embodiments of the present disclosure obtain a target image captured by a camera, obtain 3D feature points matching multiple 2D feature points from a target map, obtain multiple groups of 2D-3D point pairs, obtain target nodes in the solution space by establishing a target geometric model between the 2D-3D point pairs and the posture of the camera, and use the hyperplane intersecting with the target node as the target hyperplane, and use the 2D-3D point pairs corresponding to the target hyperplane as the inner points, and determine the posture of the camera based on the inner points. In this way, by determining the inner points from the multiple groups of 2D-3D point pairs obtained, the posture of the camera can be determined more accurately through these determined inner points. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0058] Figure 1 A flowchart of a method for determining a posture according to an exemplary embodiment of the present disclosure;

[0059] Figure 2 A schematic block diagram of functional modules of a posture determination device provided for an exemplary embodiment of the present disclosure;

[0060] Figure 3 A structural block diagram of an electronic device provided for an exemplary embodiment of the present disclosure;

[0061] Figure 4 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0062] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0063] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0064] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0065] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0066] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0067] In scenarios such as visual relocalization, SLAM or visual navigation, AR (Augmented Reality) and other related applications are often involved. For example, if a user wants to display some special effects in a scenic spot, such as stickers, etc., if the user wants to get such a special effect directly through the mobile phone, it is necessary to be able to locate the global posture of the user's mobile phone, which is the overall goal of the visual relocalization service.

[0068] If you need to locate the global position of the user's mobile phone, you first need to build a sparse point cloud map for the scene involved. When locating the user's mobile phone, the image taken by the camera on the user's mobile phone will be extracted for local feature points.

[0069] In the process of feature point extraction, relevant feature detection algorithms need to be involved. Traditional feature detection algorithms are generally based on the grayscale information of pixels, such as SIFT (Scale-invariant feature transform), SURF (Speeded Up Robust Features), ORB (Oriented Fast and Rotated Brief) algorithms, etc. The extracted feature points are usually in the edges, corners, textures and other places where the pixel grayscale values ​​change significantly. However, in the face of complex environmental scenes, traditional algorithms usually have unstable feature extraction. With the development and application of computer vision, methods based on deep learning have been applied to the field of feature extraction and have achieved good results. For example, deep learning convolutional neural networks.

[0070] After extracting some local feature points on the captured image, they can be matched with the three-dimensional map points in the pre-built point cloud map. After the feature points on the captured image are extracted, for example, there will be a 128-dimensional descriptor. At the same time, when the map points in the point cloud map are constructed, there will also be a corresponding descriptor. This requires the two-dimensional (2D) feature points on the image to find the most similar three-dimensional (3D) feature points in the point cloud map that have a corresponding relationship with similar descriptors. For example, the nearest neighbor method is used to find 3D feature points in the point cloud map that match the 2D feature points in the image. After the above feature points are matched, there will often be some errors or some wrong matches. For example, if the scene is large, there will be many similar 3D feature points in the point cloud map, so many times the 3D feature points found are not real matching points, that is, there is a problem of mismatching. For example, in large scenes such as cities and large shopping malls, or scenes with few or repeated textures, or scenes with weather, seasons, and lighting changes, the 2D-3D point pairs found by the nearest neighbor will contain a large number of mismatches, resulting in a low ratio of internal points in the 2D-3D point pairs with correct matches, and the RANSAC algorithm is also prone to failure, and thus the image cannot be effectively positioned, that is, the user's mobile phone cannot be effectively positioned. It should be noted that the embodiments are described by taking the image taken by the camera on the mobile phone as an example, and the image taken can also be obtained directly by the camera, etc.

[0071] In order to effectively locate the posture of the user's mobile phone, it is necessary to detect and eliminate the mismatches in the matched 2D-3D point pairs, so that a more accurate posture can be obtained by calculating the posture through the remaining more accurate 2D-3D points. That is, a global posture of the current 6dof (degree of freedom) of the device is obtained, including the freedom of rotation in three directions and the freedom of translation in three directions in the three-dimensional coordinate system. In implementation, the device can be a terminal including a mobile phone or a camera, etc.

[0072] Therefore, in the embodiments provided in the present disclosure, by acquiring the image taken by the camera and extracting the 2D feature points on the image, the 3D feature points corresponding to the 2D feature points in the point cloud map can be acquired by using the nearest neighbor search algorithm and the like, thereby obtaining multiple groups of 2D-3D point pairs.

[0073] In an embodiment, for the multiple groups of 2D-3D point pairs obtained above, a geometric relationship model can be performed for each 2D-3D point pair to obtain a geometric model between the 2D-3D point pair and the camera's posture. The solution space of the camera's posture in the geometric model is a high-dimensional linear space, and the 2D-3D point pair is a hyperplane in the solution space. The geometric model is as follows:

[0074]

[0075] Among them, ξ and η are the coordinates of the 2D feature points on the above image, P is the matrix of the camera pose, w is the coordinate of the 3D feature point in the map, x, y and z are the translation values ​​of the camera, r1, r2 and r3 are a column vector of the camera rotation matrix, and f is the focal length of the camera.

[0076] In the embodiment, if ξ and η, i.e., the coordinates of the 2D feature points on the image, and the coordinates W of the 3D feature points are known, and the camera translation value z and the camera rotation matrix r1, r2 and r3 are known, then the camera translation values ​​x and y can be directly obtained by the following equations (2) and (3):

[0077] x=ξ(r3w+z) / f-r1w (2)

[0078] y=η(r3w+z) / f-r2w (3)

[0079] In the embodiment, the gravity information of the image captured by the camera can be obtained, and the gravity information can be used to align the image and the point cloud map in the direction of gravity. For example, the gravity direction of the image can be provided by the gravimeter on the camera, and the point cloud map will also include the gravity direction, so that the gravity direction of the image and the gravity direction of the map are aligned to obtain the aligned image. After aligning the captured image with the point cloud map in the gravity direction, the 6dof pose estimation problem can be converted into a 4-degree-of-freedom estimation. Before the gravity direction is aligned, it is necessary to solve the 6dof pose of the image in the point cloud map, that is, including 3 degrees of freedom translation (3D coordinates of the camera optical center) and 3 degrees of freedom rotation. After aligning the gravity direction, the pose of the camera to be solved can be converted into 3 degrees of freedom translation and 1 degree of freedom rotation (i.e., the rotation angle of the camera around the gravity direction as the axis) in the embodiment.

[0080] In this way, after the gravity direction is aligned, we can change the solution space in the geometric model from a 6-dimensional high-dimensional linear space to a 4-dimensional space, and each group of 2D-3D point pairs is a hyperplane in the 4-dimensional space. In this way, two quantities r2 and r3 are actually known in the above r1, r2 and r3. That is, it is necessary to solve the three degrees of freedom of the camera's translation x, y and z, and the camera's one degree of freedom rotation r1. In the embodiment, r1 can be between 0° and 360°, and z can be the height interval within 10m of the camera, that is, (-5m, 5m). In this way, the values ​​of x and y can be obtained by searching the camera in the hypercube of the 4-dimensional space between 0° and 360°, and between -5m and 5m.

[0081] In the embodiment, in the process of solving the camera's posture, it is necessary to do it within a certain error range. The minimum granularity allowed can be pre-set, for example, the camera's translation error is within 1 cm, and the error is selected within 1°, which is equivalent to finally searching for a hypercube with a side length of 1 cm in the solution space as the minimum granularity. Therefore, by searching in the hypercube of the above-mentioned 4-dimensional space, a hypercube with the minimum granularity that meets the conditions is searched, then the posture of the camera corresponding to the hypercube of the minimum granularity is the optimal posture of the camera, that is, the hyperplane intersecting with the hypercube of the minimum granularity is the inner point of the camera posture, and the posture of the camera can be more accurately calculated through the inner point. Specifically, the accurate posture of the camera can be obtained by using the RANSAC solution PnP (Perspective-n-Points) and other methods in the related technology, which will not be repeated here.

[0082] In the embodiments provided by the present disclosure, the above solution space can be hierarchically searched by means of an octree, and the solution space is searched from larger nodes downward to smaller and smaller nodes by means of the octree structure until the node range reaches the minimum granularity of the number of rows. In the embodiments, by determining whether each node of the octree intersects with each hyperplane, the number of hyperplanes intersecting each node with the hyperplane is obtained, and the number of intersecting hyperplanes corresponds to the number of 2D-3D point pairs. In the embodiments, based on a certain hyperplane, by obtaining z and the rotation matrices r1, r2 and r3 of the camera, x and y are calculated in combination with the above formulas (2) and (3) to determine whether it is located at the current node. If it is located at the current node, it is deemed that the hyperplane intersects with the current node.

[0083] The minimum node with the largest number of intersecting hyperplanes can be obtained in the above manner. The minimum node is a hypercube with the smallest granularity in the solution space. The number of hyperplanes intersecting the minimum node is the number of corresponding 2D-3D point pairs. The 2D-3D point pairs are used as the internal points of the camera. In an embodiment, the minimum node with the largest number of intersecting hyperplanes can be used as the target node.

[0084] In order to improve the search speed of determining the search target node in the solution space, pruning can be performed. First, by obtaining the number of hyperplanes intersecting with each maximum node in the solution space, it is determined whether the child nodes of the current maximum node need to be further searched. Exemplary, the solution space in the embodiment includes multiple nodes, and each node includes multiple child nodes. The maximum number of hyperplanes intersecting the minimum node in the first N nodes in the solution space can be obtained, and the maximum number of hyperplanes can be used as a saved value. By obtaining the target hyperplane number intersecting the N+1th node, if the target hyperplane number is less than the saved value, then its child nodes will not have more intersecting hyperplanes, and the N+1th node can be directly excluded, thereby improving the search speed. If the target hyperplane number is greater than the saved value, then the saved value is updated to the target hyperplane number.

[0085] In the embodiment, the above-mentioned saved value is a dynamically changing value, which is a maximum number of inliers obtained during the search. For example, at the very beginning, the maximum number of inliers is 0. If in a certain node, there are 10 hyperplanes intersecting with the smallest node in the node, then the saved value is updated to 10. In the process of searching other nodes, if the number of inliers with the corresponding node is less than 10, stop searching at the node.

[0086] In the embodiments provided in the present disclosure, the scope of the solution space can be constrained by obtaining relevant device information, such as GPS (Global Positioning System) information, magnetometer information, etc., to accelerate the search. For example, if the GPS information is known, the maximum solution space range of the initial search can be set to the translation position of the GPS as the center, for example, within the range of 50m*50m*5m. In addition, in the process of searching for nodes downward, child nodes closer to the GPS position can be searched preferentially. In addition, the rotation angle of the camera can be obtained by obtaining the information collected by the magnetometer, for example, from 100° to 200°, avoiding direct traversal from 0° to 360°, which is equivalent to a screening and filtering of the camera rotation angle, which can reduce the complexity of the calculation.

[0087] In an embodiment, similar hyperplanes can be approximated as the same hyperplane to reduce the complexity of the algorithm. If the difference in parameters between two planes is less than a preset threshold, the two planes can be determined to be similar. If the two planes are determined to be similar, the parameters of the two planes can be updated and merged into a new plane. Then, to determine whether a node intersects with the two planes, it is only necessary to calculate the intersection determination once with the merged new plane.

[0088] In combination with the above embodiments, the present disclosure provides a method for determining a posture, such as Figure 1 As shown, the method may include the following steps:

[0089] In step S110, a target image captured by a camera is acquired, wherein the target image includes a plurality of 2D feature points.

[0090] In the embodiment, the camera may be a camera on a terminal or an independent camera device, and the terminal may be a mobile phone. The target image may be a two-dimensional image captured by a user through a camera on a mobile phone, and the 2D feature points in the target image may be obtained by using the feature detection algorithm in the above embodiment. For example, a feature detection algorithm such as SIFT, SURF or ORB or a feature detection algorithm based on deep learning may be used to extract the 2D feature points in the target image.

[0091] In step S120, 3D feature points matching the multiple 2D feature points are acquired from the target map to obtain multiple groups of 2D-3D point pairs.

[0092] In an embodiment, the target map may be a pre-built point cloud map, which includes the shooting scene of the camera. 3D feature points matching the 2D feature points of the image in the point cloud map may be searched by the nearest neighbor method to obtain multiple groups of 2D-3D point pairs.

[0093] In step S130, a target geometric model is established between the 2D-3D point pairs and the camera's posture, wherein the solution space of the camera's posture in the target geometric model is a high-dimensional linear space, the 2D-3D point pairs are hyperplanes in the solution space, and the solution space may include multiple nodes.

[0094] In the embodiment, the target geometric model is the geometric model in the above embodiment. For details, please refer to the description of the above embodiment, which will not be repeated here.

[0095] In step S140, a target node in the solution space is obtained, wherein the target node is determined based on the number of hyperplanes intersecting with the nodes in the solution space.

[0096] In the embodiment, the solution space in the embodiment includes multiple nodes, and each node includes multiple child nodes. The solution space can be searched from a larger node to a minimum node through an octree structure. In the embodiment, by determining whether each node of the octree intersects with each hyperplane, the number of hyperplanes intersecting each node and the hyperplane is obtained. The minimum node with the largest number of hyperplanes intersecting the hyperplane is obtained, and the minimum node with the largest number of hyperplanes intersecting the hyperplane is used as the target node.

[0097] In an embodiment, similar hyperplanes can be approximated as the same hyperplane to reduce the complexity of the algorithm. If the difference in parameters between two planes is less than a preset threshold, the two planes can be determined to be similar. If the two planes are determined to be similar, the parameters of the two planes can be updated and fused into a new plane. For example, when the difference between the parameters of the first hyperplane and the parameters of the second hyperplane is less than a threshold, the first hyperplane and the second hyperplane can be fused into a target hyperplane. Then, to subsequently determine whether a node intersects with the two planes, it is only necessary to calculate an intersection determination with the fused new plane once. In this way, when it is detected that a node intersects with the fused plane, the detection result of the intersection of the two hyperplanes with the node can be determined by a single detection, thereby improving detection efficiency.

[0098] In the embodiments provided by the present disclosure, in the embodiments, the gravity information of the image captured by the camera can be obtained, and the gravity information can be used to align the image and the point cloud map in the direction of gravity. For example, the gravity direction of the image can be provided by the gravimeter on the camera, and the point cloud map will also include the gravity direction, so that the gravity direction of the image and the gravity direction of the map are aligned to obtain an aligned image. After aligning the captured image with the point cloud map in the gravity direction, the 6dof pose estimation problem can be converted into a 4-degree-of-freedom estimation. For details, please refer to the description of the above embodiments.

[0099] In step S150, the hyperplane intersecting the target node is used as the target hyperplane, and the 2D-3D point pairs corresponding to the target hyperplane are used as internal points, and the position and posture of the camera are determined based on the internal points.

[0100] In the embodiment, the number of hyperplanes intersecting the hyperplane of the target node is the number of corresponding 2D-3D point pairs, and the 2D-3D point pairs are used as the internal points of the camera. The position and posture of the camera can be more accurately calculated by determining the internal points, and specifically, the accurate position and posture of the camera can be obtained by using the RANSAC PnP solution in the related technology, which will not be repeated here.

[0101] Based on the above embodiment, in the embodiment provided by the present disclosure, the above step S140 may further specifically include the following steps:

[0102] S141, dividing the nodes in the solution space into multiple levels of sub-nodes, wherein the smallest node in the sub-nodes meets the target granularity index.

[0103] The target granularity index may be the minimum granularity in the above embodiment. For details, please refer to the description in the above embodiment, which will not be repeated here.

[0104] S142, obtaining the minimum node with the largest number of intersecting hyperplanes among the minimum nodes, and determining the minimum node with the largest number of intersecting hyperplanes as the target node.

[0105] In an embodiment, in order to determine the smallest node with the largest number of intersecting hyperplanes, the number of hyperplanes intersecting with each node in the solution space can be first obtained to further determine whether the target node exists in the child nodes under the node, which can improve the search speed for searching the target node in the solution space.

[0106] In the embodiment, by obtaining the number of hyperplanes intersecting with each maximum node in the solution space, it is judged whether it is necessary to further search the child nodes of the current maximum node. Exemplary, the solution space in the embodiment includes multiple nodes, and each node includes multiple child nodes. The maximum number of hyperplanes intersecting with the minimum node in the first N nodes in the solution space can be obtained, and the maximum number of hyperplanes is used as a saved value. By obtaining the target number of hyperplanes intersecting with the N+1th node, if the target number of hyperplanes is less than the saved value, then its child nodes will not have more number of intersecting hyperplanes, and the N+1th node can be directly excluded, thereby improving the search speed. If the target number of hyperplanes is greater than the saved value, then the saved value is updated to the target number of hyperplanes.

[0107] In the embodiments provided by the present disclosure, the positioning information of the camera when shooting the target image is obtained, and the target area range is determined based on the positioning information, so that the nodes outside the target area range are excluded. For example, the search can be accelerated by obtaining the positioning information such as the GPS of the device. The maximum solution space range of the initial search can be set to be centered on the translation position of the GPS, for example, within the range of 50m*50m*5m. In addition, in the process of searching for nodes downward, the subnodes closer to the GPS position can be preferentially searched to improve the efficiency of the search.

[0108] In the case of dividing each functional module according to each function, an embodiment of the present disclosure provides a posture determination device, which may be a server or a chip applied to a server. Figure 2 A schematic block diagram of the functional modules of a posture determination device provided by an exemplary embodiment of the present disclosure. Figure 2 As shown, the posture determination device includes:

[0109] An image acquisition module 10 is used to acquire a target image captured by a camera, wherein the target image includes a plurality of 2D feature points;

[0110] A feature point acquisition module 20 is used to acquire 3D feature points matching the plurality of 2D feature points from the target map to obtain a plurality of groups of 2D-3D point pairs;

[0111] A model building module 30 is used to build a target geometric model between a 2D-3D point pair and the camera's posture; the solution space of the posture in the target geometric model is a high-dimensional linear space, the 2D-3D point pair is a hyperplane in the solution space, and the solution space includes a plurality of nodes;

[0112] A target node acquisition module 40 is used to acquire a target node in the solution space; wherein the target node is determined based on the number of hyperplanes intersecting with the nodes in the solution space;

[0113] The pose determination module 50 is used to use the hyperplane intersecting with the target node as the target hyperplane, and the 2D-3D point pairs corresponding to the target hyperplane as the inner points, and determine the pose of the camera based on the inner points.

[0114] In another embodiment provided by the present disclosure, the target node acquisition module is further configured to:

[0115] Dividing the nodes in the solution space into multiple levels of sub-nodes; the smallest node in the sub-nodes meets the target granularity index;

[0116] The minimum node with the largest number of intersecting hyperplanes among the minimum nodes is obtained, and the minimum node with the largest number of intersecting hyperplanes is determined as the target node.

[0117] In another embodiment provided by the present disclosure, the device further includes:

[0118] A saved value determination module, used to obtain the maximum number of hyperplanes intersecting with the minimum node among the first N nodes, and use the maximum number of hyperplanes as a saved value; wherein N is a positive integer;

[0119] A target hyperplane quantity acquisition module is used to acquire the number of target hyperplanes intersecting with the N+1th node in the solution space;

[0120] The node processing module is used to exclude the N+1th node when the number of the target hyperplanes is less than the saved value.

[0121] In another embodiment provided by the present disclosure, the device further includes:

[0122] A saved value updating module is used to update the saved value to the target number of hyperplanes when the target number of hyperplanes is greater than the saved value, and obtain the number of hyperplanes intersecting with the smallest node in the N+1th node.

[0123] In another embodiment provided by the present disclosure, the device further includes:

[0124] A parameter acquisition module, used to acquire parameters of the first hyperplane and the second hyperplane in the solution space;

[0125] The plane fusion module is used to fuse the first hyperplane and the second hyperplane into a target hyperplane when the difference between the parameters of the first hyperplane and the parameters of the second hyperplane is less than a threshold.

[0126] In another embodiment provided by the present disclosure, the device further includes:

[0127] A gravity direction information acquisition module, used to acquire the gravity direction information of the target image;

[0128] An alignment module is used to align the target image with the target map in the direction of gravity based on the gravity direction information.

[0129] In another embodiment provided by the present disclosure, the device further includes:

[0130] A positioning information acquisition module, used to acquire the positioning information of the camera when shooting the target image;

[0131] The area determination module is used to determine the target area range based on the positioning information and exclude nodes outside the target area range.

[0132] Since the device embodiment corresponds to the above method embodiment, please refer to the description of the embodiment corresponding to the above method for details, which will not be repeated here.

[0133] The posture determination device provided by the embodiment of the present disclosure obtains the target image captured by the camera, obtains 3D feature points matching multiple 2D feature points from the target map, obtains multiple groups of 2D-3D point pairs, obtains the target node in the solution space by establishing a target geometric model between the 2D-3D point pairs and the posture of the camera, and uses the hyperplane intersecting with the target node as the target hyperplane, and uses the 2D-3D point pairs corresponding to the target hyperplane as the inner points, and determines the posture of the camera based on the inner points. In this way, by determining the inner points from the obtained multiple groups of 2D-3D point pairs, the posture of the camera can be determined more accurately through these determined inner points.

[0134] An embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory for storing instructions executable by the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the above method disclosed in the embodiment of the present disclosure.

[0135] Figure 3 The structure diagram of an electronic device provided by an exemplary embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801 , and the processor 1801 can execute corresponding steps in the above method disclosed in the embodiment of the present disclosure.

[0136] The processor 1801 may also be referred to as a central processing unit (CPU), which may be an integrated circuit chip having signal processing capabilities. Each step in the method disclosed in the embodiment of the present disclosure may be completed by an integrated logic circuit of hardware in the processor 1801 or by instructions in the form of software. The processor 1801 may be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a memory 1802, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, or other mature storage media in the art. The processor 1801 reads the information in the memory 1802 and completes the steps of the method in combination with its hardware.

[0137] In addition, when various operations / processes according to the present disclosure are implemented by software and / or firmware, they can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, such as Figure 4 The computer system 1900 shown is installed with the programs constituting the software. When the various programs are installed, the computer system can perform various functions, including the functions described above. Figure 4 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure.

[0138] Computer system 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0139] like Figure 4As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 to a random access memory (RAM) 1903. In the RAM 1903, various programs and data required for the operation of the computer system 1900 can also be stored. The computing unit 1901, the ROM 1902, and the RAM 1903 are connected to each other via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0140] A plurality of components in the computer system 1900 are connected to the I / O interface 1905, including: an input unit 1906, an output unit 1907, a storage unit 1908, and a communication unit 1909. The input unit 1906 may be any type of device capable of inputting information to the computer system 1900, and the input unit 1906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 1907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1908 may include, but is not limited to, a disk, an optical disk. The communication unit 1909 allows the computer system 1900 to exchange information / data with other devices over a network such as the Internet, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0141] The computing unit 1901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the above methods disclosed in the embodiments of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1900 via the ROM 1902 and / or the communication unit 1909. In some embodiments, the computing unit 1901 may be configured to perform the above methods disclosed in the embodiments of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0142] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above method disclosed in the embodiment of the present disclosure.

[0143] The computer-readable storage medium in the disclosed embodiments may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specifically, the computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0145] The embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor, the above method disclosed in the embodiments of the present disclosure is implemented.

[0146] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer.

[0147] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.

[0149] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0150] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above features with the technical features with similar functions disclosed in the present disclosure (but not limited to).

[0151] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for determining a posture, characterized in that: The method comprises: Acquire a target image captured by a camera, wherein the target image includes a plurality of 2D feature points; Acquire 3D feature points matching the plurality of 2D feature points from the target map to obtain a plurality of groups of 2D-3D point pairs; Establishing a target geometric model between a 2D-3D point pair and the camera's posture; a solution space of the posture in the target geometric model is a high-dimensional linear space, the 2D-3D point pair is a hyperplane in the solution space, and the solution space includes a plurality of nodes; Acquire a target node in the solution space; wherein the target node is determined based on the number of hyperplanes intersecting with the nodes in the solution space; A hyperplane intersecting the target node is used as a target hyperplane, and a 2D-3D point pair corresponding to the target hyperplane is used as an inner point, and the position and posture of the camera is determined based on the inner point.

2. The method according to claim 1, characterized in that The obtaining of the target node in the solution space comprises: Dividing the nodes in the solution space into multiple levels of sub-nodes; the smallest node in the sub-nodes meets the target granularity index; The minimum node with the largest number of intersecting hyperplanes among the minimum nodes is obtained, and the minimum node with the largest number of intersecting hyperplanes is determined as the target node.

3. The method according to claim 2, characterized in that The method further comprises: Obtain the maximum number of hyperplanes intersecting the minimum node among the first N nodes, and save the maximum number of hyperplanes as a saved value; wherein N is a positive integer; Obtain the number of target hyperplanes intersecting with the N+1th node in the solution space; When the number of the target hyperplanes is less than the stored value, the N+1th node is excluded.

4. The method according to claim 3, characterized in that The method further comprises: When the target number of hyperplanes is greater than the stored value, the stored value is updated to the target number of hyperplanes, and the number of hyperplanes intersecting with the smallest node in the N+1th node is obtained.

5. The method according to claim 1, characterized in that The method further comprises: Obtaining parameters of a first hyperplane and parameters of a second hyperplane in the solution space; When the difference between the parameters of the first hyperplane and the parameters of the second hyperplane is less than a threshold, the first hyperplane and the second hyperplane are fused into a target hyperplane.

6. The method according to claim 1, characterized in that The method further comprises: Acquiring gravity direction information of the target image; The target image is aligned with the target map in a gravity direction based on the gravity direction information.

7. The method according to claim 1, characterized in that The method further comprises: Acquiring positioning information of the camera when taking the target image; A target area range is determined based on the positioning information, and nodes outside the target area range are excluded.

8. A posture determination device, characterized in that: The device comprises: An image acquisition module is used to acquire a target image captured by a camera, wherein the target image includes a plurality of 2D feature points; A feature point acquisition module, used to acquire 3D feature points matching the multiple 2D feature points from the target map to obtain multiple groups of 2D-3D point pairs; A model building module, used to build a target geometric model between a 2D-3D point pair and the camera's posture; the solution space of the posture in the target geometric model is a high-dimensional linear space, the 2D-3D point pair is a hyperplane in the solution space, and the solution space includes a plurality of nodes; A target node acquisition module, used to acquire a target node in the solution space; wherein the target node is determined based on the number of hyperplanes intersecting with the nodes in the solution space; The pose determination module is used to use the hyperplane intersecting with the target node as the target hyperplane, and the 2D-3D point pair corresponding to the target hyperplane as the inner point, and determine the pose of the camera based on the inner point.

9. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 7.

Images