An ar vision visual positioning method and system

Abstract

The application relates to an AR long-range visual positioning method, wherein the method comprises: establishing 2D-3D association between a real-time image and a similar map frame; constructing a virtual camera coordinate system corresponding to a scene map, and determining a first observation vector corresponding to a preset road marker point in the scene map in the virtual camera coordinate system; acquiring a second observation vector corresponding to each preset road marker point in an entity camera coordinate system, converting the 2D-3D association into 3D-3D association according to the first observation vector and the second observation vector; and restoring an absolute pose transformation through real scale information; acquiring a rigid transformation matrix between the virtual camera coordinate system and the scene map coordinate system, and obtaining a positioning result corresponding to the real-time image according to the rigid transformation matrix and the absolute pose transformation and sending the positioning result to a terminal. Through the application, the problem of poor positioning accuracy in the AR long-range positioning scene is solved, and the accuracy of long-range positioning is improved without additional application of auxiliary facilities.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of augmented reality, and in particular to an AR long-range visual positioning method and system. BACKGROUND

[0002] AR real scene positioning navigation, that is, through a terminal device to shoot a offline scene image, matching and processing the scene image in a scene map to obtain real-time positioning information (6dof) of the terminal device, and then assisting navigation according to the positioning information.

[0003] In related technologies, the relatively mature AR navigation technology is generally applied in indoor environments or small-range outdoor environments. For example, "Huawei Hetu", "Didi AR real scene navigation", and "Sogou AR real scene navigation", etc. Such close-range AR navigation technology provides reference information by setting high-precision positioning devices (such as Bluetooth devices) in a small-range environment to assist visual positioning to improve positioning accuracy.

[0004] However, for AR visual positioning in a large scene at a long distance (also known as a long-range positioning problem), since such a scene is generally in a large outdoor space, the actual experience points and observation objects (scenery) are often far apart (up to hundreds of meters or even kilometers). In such a scene, it is undoubtedly costly and difficult to implement to set up additional high-precision auxiliary devices, and since the GPS has poor accuracy (usually tens of meters), it cannot be used as effective auxiliary information for long-range positioning.

[0005] For the long-range positioning problem, since no effective additional auxiliary information can be obtained, only the information of the positioning image itself can be relied on. However, the 3D points in the three-dimensional map at a long distance are generally not well reconstructed, and the features that can be extracted on the 2D image of the long-range are generally of poor accuracy. In the case of only applying the information of the image itself, in order to improve the positioning accuracy as much as possible, the following methods are generally used for improvement in the prior art:

[0006] 1. Using a random sample consensus (RANSAC) algorithm to remove outliers; however, for the long-range positioning problem, removing outliers will still cause positioning jumps, and in the case where all observations are inliers, the above positioning jumps are caused by the fact that the observations cannot provide sufficient constraints;

[0007] 2. Nonlinear optimization through robust kernel functions (such as Huber, Cauchy, Geman-McClure, Turkey). However, for the long-range positioning problem, the Hessian matrix of the nonlinear solution is often not full rank, and the main role of the robust kernel function is to "add icing to the cake" for the full rank Hessian, so it cannot achieve good positioning results.

[0008] Both of the above two ways are carried out on the premise that the input information is complete (i.e., six degrees of freedom are relatively certain), that is, the maximum possible noise is removed in complete information. However, for the long-range positioning problem, due to the limited accuracy of the information provided by the positioning image itself, there are often multiple underdetermined degrees of freedom, and additional auxiliary information cannot be provided by appropriate means. Therefore, in the prior art, there is no effective solution to the problem of poor AR long-range positioning accuracy. SUMMARY

[0009] Embodiments of the present application provide an AR long-range visual positioning method, system, computer device and computer readable storage medium to at least solve the problem of poor AR long-range positioning accuracy in the related art.

[0010] In a first aspect, the embodiments of the present application provide an AR long-range visual positioning method, characterized in that it is applied to a server, and the method comprises:

[0011] establishing a 2D-3D association between a real-time image and a similar map frame, wherein the similar map frame is a map frame in a scene map that matches the real-time image, and the real-time image is sent by a terminal;

[0012] constructing a virtual camera coordinate system corresponding to the scene map, and determining a first observation vector corresponding to a preset landmark point in the scene map in the virtual camera coordinate system;

[0013] obtaining a second observation vector corresponding to each preset landmark point in an entity camera coordinate system, and converting the 2D-3D association into a 3D-3D association according to the first observation vector and the second observation vector;

[0014] calculating a relative pose transformation between the virtual camera coordinate system and a world coordinate system according to the 3D-3D association;

[0015] optimizing the relative pose transformation based on a preset strategy, calculating real scale information with the relative pose transformation after optimization as a constraint condition, and restoring an absolute pose transformation through the real scale information;

[0016] obtaining a rigid transformation matrix between the virtual camera coordinate system and the scene map coordinate system, and obtaining a positioning result corresponding to the real-time image according to the rigid transformation matrix and the absolute pose transformation, and sending the positioning result to the terminal.

[0017] In some embodiments, a virtual camera coordinate system corresponding to the scene map is established based on the geometric structure of the scene map,

[0018] replace the 3D landmark points in the positioning map coordinate system with the orientation vectors in the virtual camera coordinate system, and convert an observation relationship between a world coordinate system and a physical camera coordinate system into a relationship between cameras;

[0019] wherein the virtual camera coordinate system meets a preset condition, including:

[0020] a field of view of the virtual camera is directed toward the landmark points;

[0021] the preset landmark points in the scene map are uniformly distributed on the left and right of the optical axis of the virtual camera;

[0022] a view cone of the virtual camera covers a preset proportion of the preset landmark points, and a field of view angle is within a preset range.

[0023] In some embodiments, obtaining a second observation vector corresponding to each of the preset landmark points in the physical camera coordinate system includes:

[0024] obtaining camera intrinsic parameters sent by a terminal device;

[0025] According to the camera intrinsic parameters, the 2D feature points in the real-time image are back-projected to the physical camera coordinate system to obtain the second observation vector.

[0026] In some embodiments, the 2D-3D correlation is converted into a 3D-3D correlation according to the first observation vector and the second observation vector, including:

[0027] respectively replacing the 2D feature points and the 3D feature points in the 2D-3D correlation with the second observation vector and the first observation vector to obtain the 3D-3D correlation,

[0028] wherein a pair of observation vectors forms a 3D-3D observation pair, and all 3D-3D observation pairs form an epipolar geometry relationship between the virtual camera coordinate system and the physical camera coordinate system.

[0029] In some embodiments, the relative pose transformation between the virtual camera coordinate system and the world coordinate system is calculated according to the 3D-3D correlation, including:

[0030] obtaining an epipolar geometry relationship corresponding to the 3D-3D correlation;

[0031] based on the epipolar geometry relationship, constructing a linear equation system with minimum observations, and calculating the relative pose transformation between the virtual camera coordinate system and the world coordinate system by solving the linear equation system.

[0032] In some embodiments, in the solving process of the relative pose transformation, a RANSAC algorithm is combined to screen an inlier set of the 3D-3D observation pairs.

[0033] In some embodiments, the optimization of the relative pose transformation based on the preset strategy comprises:

[0034] A nonlinear least squares model is established based on the screened inlier set of the 3D-3D observation pairs.

[0035] Based on the nonlinear least squares model, the optimization condition that the orientation vector in the transformed virtual camera coordinate system is equal to the orientation vector in the to-be-solved camera coordinate system is used to calculate the optimized relative pose transformation.

[0036] In some embodiments, the nonlinear optimization equation is shown in the following formula 1,

[0037]

[0038]

[0039] wherein, Rt is the relative pose transformation, voc is the second observation vector, voc' is the second observation vector converted according to the relative pose transformation, the preset road point constraint is on the displacement vector of the relative pose transformation, and the optimization result of the relative pose transformation is solved by iterative updating in the optimization process.

[0040] In some embodiments, the real scale information is calculated based on the optimized relative pose transformation as a constraint condition, and the absolute pose transformation is recovered through the real scale information, comprising:

[0041] Under the constraint of the relative pose transformation, an optimization condition is constructed in which the distance between the optical center of the virtual camera and multiple similar map frames is minimized, wherein the positioning result is on the displacement vector of the relative pose.

[0042] According to the optimization condition, a least squares equation is constructed and iteratively solved to obtain the real scale information.

[0043] In some embodiments, the method further comprises:

[0044] The server collects the pitch angle parameter and the roll angle parameter sent by the terminal.

[0045] In the process of optimizing the relative pose transformation, the pitch angle parameter and the roll angle parameter are used as initial values for optimization, and the optimization of the pitch angle dimension and the roll angle dimension is omitted.

[0046] In some embodiments, before establishing the 2D-3D association between the real-time image and the similar map frame sent by the terminal, the method further comprises:

[0047] receiving the real-time image and the camera intrinsic parameter sent by the terminal;

[0048] performing feature extraction on the real-time image to obtain global features and local features;

[0049] matching the global features in a scene map to obtain the similar map frame, wherein the scene map is a three-dimensional map of an offline scene where the terminal is located.

[0050] In some embodiments, establishing the 2D-3D association between the real-time image and the similar map frame comprises:

[0051] establishing the 2D-3D observation pairs between the local features of the real-time image and the local features of the similar map frame respectively, and composing the 2D-3D association based on all 2D-3D observations.

[0052] In a second aspect, the embodiments of the present application provide an AR large scene long-distance visual positioning system, which comprises a preprocessing module, a relative pose calculation module and an absolute pose calculation module, wherein,

[0053] the preprocessing module is configured to establish a 2D-3D association between a real-time image and a similar map frame, wherein the similar map frame is a map frame in a scene map that matches the real-time image, and the real-time image is sent by a terminal,

[0054] and a virtual camera coordinate system corresponding to the scene map is constructed, and a first observation vector corresponding to a preset road marker point in the scene map is determined in the virtual camera coordinate system,

[0055] and a second observation vector corresponding to each of the preset road marker points in an entity camera coordinate system is obtained, and the 2D-3D association is converted into a 3D-3D association according to the first observation vector and the second observation vector;

[0056] the relative pose calculation module is configured to calculate a relative pose transformation between the virtual camera coordinate system and a world coordinate system according to the 3D-3D association;

[0057] the absolute pose calculation module is configured to optimize the relative pose transformation based on a preset strategy, calculate real scale information with the optimized relative pose transformation as a constraint condition, and restore an absolute pose transformation through the real scale information;

[0058] The positioning result output module is configured to obtain a rigid transformation matrix between the virtual camera coordinate system and the scene map coordinate system, and obtain a positioning result corresponding to the real-time image according to the rigid transformation matrix and the absolute pose transformation, and send the positioning result to the terminal.

[0059] In some embodiments, the preprocessing module establishes a virtual camera coordinate system corresponding to the scene map based on a geometric structure of the scene map,

[0060] The 3D landmark points in the positioning map coordinate system are replaced by orientation vectors in the virtual camera coordinate system, and an observation relationship between a world coordinate system and a physical camera coordinate system is converted into a relationship between cameras.

[0061] The virtual camera coordinate system meets a preset condition, including:

[0062] The field of view of the virtual camera is directed towards the landmark points.

[0063] The preset landmark points in the scene map are uniformly distributed on the left and right of the optical axis of the virtual camera.

[0064] The field of view of the virtual camera covers a preset proportion of the preset landmark points, and the field of view angle is within a preset range.

[0065] In some embodiments, the preprocessing module obtains a second observation vector corresponding to each of the preset landmark points in a physical camera coordinate system, including:

[0066] The camera intrinsic parameters sent by the terminal device are obtained.

[0067] According to the camera intrinsic parameters, the 2D feature points in the real-time image are back-projected to the physical camera coordinate system to obtain the second observation vector.

[0068] In some embodiments, the preprocessing module converts the 2D-3D correlation into 3D-3D correlation according to the first observation vector and the second observation vector, including:

[0069] The 2D feature points and the 3D feature points in the 2D-3D correlation are respectively replaced by the second observation vector and the first observation vector to obtain the 3D-3D correlation,

[0070] A pair of observation vectors forms a 3D-3D observation pair, and all 3D-3D observation pairs form an epipolar geometry relationship between the virtual camera coordinate system and the physical camera coordinate system.

[0071] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method in the first aspect when executing the computer program.

[0072] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method in the first aspect.

[0073] Compared with the related art, the AR long-range visual positioning method provided by the embodiment of the present application has at least the following beneficial effects:

[0074] 1. The 6dof calculation process is constrained in a more accurate range and is hierarchically solved in a combined manner of map frame priori and landmark point posteriori, so that the accuracy of long-range positioning is improved without additional auxiliary facilities;

[0075] 2. The coordinate system of the positioning map is abstracted as a virtual camera coordinate system, and the observation relationship between the world coordinate system and the to-be-positioned (physical) camera coordinate system is converted into the observation relationship between two cameras, so that the specific position of the 3D landmark point is replaced by the vector direction, and the adverse effects caused by the high uncertainty of the long-range 3D point are avoided;

[0076] 3. In the long-range positioning, since the calculability of part of the dimensions in 6dof is lower, a coarse-to-fine relative pose solving process from the minimal solver to the batch least square solver is established by using RANSAC in the present application, so that the 6dof is separately solved in a progressive manner, and the positioning result is more robust.

[0077] 4. In the traditional method, the positioning result is generally calculated by maximizing the posterior probability distribution once, in the present application, the Bayesian probability inference is used, the dimension in the long-range positioning problem that cannot be constrained by the visual feature is constrained to a reasonable range by the similar frame priori constraint provided by the positioning map on the basis of maximizing the joint rate distribution. BRIEF DESCRIPTION OF DRAWINGS

[0078] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate the illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0079] Figure 1 A pose schematic diagram according to an embodiment of the present application;

[0080] Figure 2 Another pose schematic diagram according to an embodiment of the present application;

[0081] Figure 3 is a schematic diagram of a perspective visual positioning uncertainty according to an embodiment of the present application;

[0082] Figure 4 is a schematic diagram of the uncertainty of a 2D point in a perspective image according to an embodiment of the present application;

[0083] Figure 5 is a schematic diagram of another perspective visual positioning uncertainty according to an embodiment of the present application;

[0084] Figure 6 is a schematic diagram of an application environment of an AR perspective positioning method according to an embodiment of the present application;

[0085] Figure 7 is a flowchart of an AR perspective positioning method according to an embodiment of the present application;

[0086] Figure 8 is a schematic diagram of the relationship between a virtual camera coordinate system and a 3D landmark point according to an embodiment of the present application;

[0087] Figure 9 is a schematic diagram of an observation geometric relationship in a scene according to an embodiment of the present application;

[0088] Figure 10 is a schematic diagram of a similar map frame constraint positioning point according to an embodiment of the present application;

[0089] Figure 11 is a structural block diagram of an AR perspective visual positioning system according to an embodiment of the present application;

[0090] Figure 12 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0091] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is described and explained below in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0092] It is apparent that the accompanying drawings described below in the description are only some examples or embodiments of the present application, and for those skilled in the art, the present application can be applied to other similar situations according to the drawings without creative labor. In addition, it can be understood that although the efforts made in this development process can be complex and lengthy, some design, manufacture or production changes based on the technology disclosed in the present application are only routine technical means for those skilled in the art related to the disclosure of the present application, and should not be understood as insufficient disclosure of the present application.

[0093] In the present application, the term "embodiment" means that the specific features, structures or characteristics described in conjunction with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present application can be combined with other embodiments without conflict.

[0094] Unless otherwise defined, the technical terms or scientific terms involved in the present application should be understood as the usual meaning understood by those skilled in the art in the technical field to which the present application belongs. The terms "one", "a", "an", "the", and similar words involved in the present application do not represent quantity limitation, but can represent singular or plural. The terms "include", "contain", "have", and any variations thereof involved in the present application are intended to cover non-exclusive inclusion; for example, a process, method, system, product or device including a series of steps or modules (units) is not limited to the listed steps or units, but can also include steps or units not listed, or can also include other steps or units inherent to the process, method, product or device. The terms "connected", "connected", "coupled" and similar words involved in the present application are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The term "multiple" refers to two or more. The association between the associated objects is described by the term "and / or", which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects. The terms "first", "second", "third", and the like involved in the present application are only to distinguish similar objects, and do not represent a specific order for the objects.

[0095] In this document, it should be understood that the terms involved can be technical means or other summary technical terms for implementing part of the present application. For example, the terms can include:

[0096] Pose: Position and orientation (facing), for example, in 2D it is generally (x, y, yaw), and in 3D it is generally (x, y, z, yaw, pitch, roll), including 6 degrees of freedom (6 of). The last three elements describe the object's orientation, where yaw is the heading angle, rotating around the Z-axis; pitch is the pitch angle, rotating around the Y-axis; and roll is the roll angle, rotating around the X-axis. Figure 1 and Figure 2 As shown.

[0097] Uncertainty: Used to describe the magnitude of error in a state (such as a pose), usually represented by standard deviation (one-dimensional variable) or covariance matrix (multi-dimensional variable); the larger the value, the less reliable it is, and the smaller the value, the more reliable it is; uncertainty is different from error, it describes the error range, such as 1 standard deviation, which represents a probability of about 68%, and is an error of 1 standard deviation.

[0098] like Figure 3 and Figure 4 As shown: distant 3D landmarks have significant uncertainty; in addition, the feature points that can be extracted from distant images are mainly various macroscopic contour edges (higher levels of the image pyramid), and the 2D position uncertainty of higher-level features is often higher, resulting in higher 2D observation uncertainty of distant images.

[0099] like Figure 5 As shown, visual positioning is least accurate along the optical axis, and this uncertainty is positively correlated with the distance between the 3D point and the camera's optical center. For distant scenes, the distance between the 3D point and the camera's optical center is often greater than 200 meters, and in some cases even reaches the kilometer level, making positioning along the optical axis extremely inaccurate. The standard deviation of its optical axis positioning may reach about 20 meters, corresponding to huge (tens of meters) jumps in real AR.

[0100] Visual positioning system: A system that determines the position and orientation of an object in a map coordinate system using an image, with the positioning result being "the pose of object a in map b at time t".

[0101] The AR far-field visual positioning method provided in this embodiment can be applied to, for example... Figure 6 In the environment shown, Figure 6 This is a schematic diagram illustrating the application environment of the AR distant positioning method according to an embodiment of this application. For example... Figure 6As shown, the user takes a real-time image of the offline scene at the AR experience point through the terminal device 10, and uploads the real-time image to the server 11. The server 11 obtains the positioning information (6dof) when the user takes the real-time image after matching and calculating in the positioning map according to the real-time image, and returns it to the terminal device 10. It should be noted that the above offline scene can be a scenic spot, a park, etc.; the terminal device 10 can be, but is not limited to, a smart phone, a tablet computer, and a PC computer, etc., and the server 11 can be a stand-alone server or a cluster composed of multiple servers.

[0102] Figure 7 is a flowchart of an AR long-range positioning method according to an embodiment of the present application, as shown, the flow includes the following steps: Figure 7

[0103] S701, 2D-3D association is established between the real-time image and the similar map frame, wherein the similar map frame is the map frame in the scene map matched with the real-time image, and the real-time image is sent by the terminal;

[0104] Before establishing the 2D-3D association, the server receives the real-time image and the camera intrinsic parameter sent by the terminal; and performs feature extraction on the real-time image to obtain global features and local features; wherein the real-time image can be an RGB image, and the camera intrinsic parameter can include an intrinsic matrix and a distortion matrix.

[0105] Further, the global features are matched in the scene map to obtain the similar map frame; it can be understood that the scene map is a three-dimensional map of the real offline scene where the terminal is located, wherein the offline scene can be a scenic spot, a park, and a large stadium, etc.

[0106] In this embodiment, matching the global descriptor and the positioning map can extract k nearest neighbor SFM map frames as the similar map frame. The matching of the local descriptor and the local descriptor of the k nearest neighbor SFM map frames in the positioning map can establish a 2D-3D form of association. The 2D-3D form of association means that a landmark point in a world coordinate system and a camera image feature point correspond to each other.

[0107] It should be noted that the way of extracting feature points and global and local descriptors of the image in this embodiment is not specifically limited. Traditional SIFT, SURF, ORB, BOW, VLAD methods can be used, deep learning methods such as D2Net, HFNet, NetVALD, R2D2 methods can also be used, and specific selection can be made flexibly according to the hardware computing power level and scene requirements.

[0108] ​S702, a virtual camera coordinate system corresponding to the scene map is constructed, and a first observation vector corresponding to a preset landmark point in the scene map is determined in the virtual camera coordinate system, denoted as vop[x, y, z];

[0109] The virtual camera coordinate system is established according to the geometric structure of the entire scene, and the real offline scene is abstracted as a coordinate system under a virtual camera view.

[0110] It should be noted that, generally, the environment and the camera are considered as two unrelated things, or the camera is positioned in the environment; in the embodiment, the entire environment is abstracted as a "virtual camera", and through the setting, the observation relationship between the world coordinate system and the to-be-solved entity camera coordinate system can be converted into the observation relationship between two cameras.

[0111] Further, the origin of the world coordinate system to a 3D landmark point can form a three-dimensional direction vector, so that the original coordinate point position can be replaced by a direction vector. It can be understood that no matter how far the 3D landmark point is, it is always constrained on the vector, so that the adverse effects caused by too large uncertainty of the long-range 3D point can be avoided.

[0112] In addition, there is a 6dof rigid body transformation Two between the virtual camera coordinate system and the positioning map coordinate system, and subsequent 6dof solving is performed on the virtual camera coordinate system.

[0113] Specifically, Figure 8 is a schematic diagram of the relationship between the virtual camera coordinate system according to the embodiment of the application and the 3D landmark point, as Figure 8 indicated, the established virtual camera coordinate system needs to meet the following three conditions:

[0114] 1: The virtual camera needs to be directed towards the 3D landmark point.

[0115] 2: The 3D landmark points are uniformly distributed on the left and right of the optical axis of the virtual camera.

[0116] 3: The frustum triangle of the virtual camera covers the 3D landmark points in a preset proportion (95%), and the FOV needs to meet a preset range (45°-60°).

[0117] S703, a second observation vector corresponding to each preset landmark point in the entity camera coordinate system is obtained, and the 2D-3D correlation is converted into 3D-3D correlation according to the first observation vector and the second observation vector.

[0118] For the 2D point on the image to be positioned, the camera intrinsic parameter obtained through step S701 is used to back-project to the camera coordinate system to obtain the second observation vector Voc=[u, v, w].

[0119] The first observation vector can be constituted by the origin of a world coordinate system to a 3D landmark point, and the second observation vector can be constituted by the origin of a camera coordinate system to the 3D landmark point.

[0120] It can be understood that the 3D-3D correlation is a matching pair of a three-dimensional bearing vector Vop in a world coordinate system and a bearing vector Voc in a camera coordinate system, and the bearing vector Voc and the bearing vector Vop are connected through a 6dof transformation [R|t], and all 3D-3D observations constitute an epipolar geometry relationship between a virtual camera coordinate system and a camera coordinate system to be positioned.

[0121] Specifically, the relationship between the first observation vector and the second observation vector is shown in the following formula 1.

[0122]

[0123]

[0124] Voc=R*Vop+t

[0125] The epipolar geometry relationship between the virtual camera coordinate system and the camera coordinate system to be positioned is shown in the following formula 2.

[0126] Voc T R · [t] X *Voc=0

[0127] Wherein, the []x operator is a skew-symmetric matrix calculation of a three-dimensional vector.

[0128] In S704, a relative pose transformation between the virtual camera coordinate system and the world coordinate system is calculated according to the 3D-3D correlation.

[0129] In this embodiment, the relative pose change Twc=[R|t] from the virtual camera to the world coordinate system is directly linearly solved (DLT) in the epipolar geometry RANSAC manner.

[0130] Wherein, the classical 8-point method can be used for solving. The R|t] can be represented as an augmented matrix of 3X4, and the elements of the augmented matrix are unknowns to be solved. Further, the equation set can be accumulated in the form of adding observations, and finally the equation set can be converted into the form of Ax=b, and the matrix SVD decomposition method is used for solving.

[0131] It is understandable that, since the virtual camera coordinate system and the positioning map coordinate system are identical, the above steps are equivalent to obtaining a relative pose transformation from the positioning map coordinate system to the camera coordinate system to be positioned. The relative pose transformation can also be called 6dof positioning without scale information. The 3dof position in the real absolute pose is constrained by the direction vector t, and any position on the direction vector is a possible positioning position.

[0132] It should be noted that, since the computability of some dimensions in 6dof is lower. Therefore, using the traditional way of maximizing the joint probability distribution to solve all 6dof at once may cause unstable solution. In this embodiment, the relative pose is solved first, and then the position dimension in the joint probability distribution is constrained by the translation direction in the relative pose. Through such intermediate state calculation and progressive calculation of the absolute pose, the solution can be more stable.

[0133] Figure 9 is a schematic diagram of an observation geometric relationship in a scene according to an embodiment of the present application.

[0134] S705, optimizing the relative pose transformation based on a preset strategy, calculating the real scale information with the optimized relative pose transformation as a constraint condition, and restoring the absolute pose transformation through the real scale information;

[0135] Wherein, the nonlinear optimization equation is established by "the direction vector in the transformed virtual camera coordinate system is equal to the direction vector in the camera coordinate system to be solved", to optimize the relative pose change Toc, and all 3D-3D observation pairs are introduced into the calculation, so that the obtained relative pose transformation is more accurate.

[0136] In this step, all 3D-3D observation inlier sets M filtered out in step S703 are used as input to establish a least squares optimization problem. The exclusion of abnormal matching points in step S704 can greatly make the relative transformation optimized in this step more stable.

[0137] It should be noted that, as shown in Figure 7 and Figure 8 By observing the actual scene offline, it can be found that, in the long-range positioning problem, the AR experience area has geographical accessibility restrictions; and in the construction of SFM (positioning map), the camera position distribution in the map is degenerated into a low-dimensional distribution of lines and points compared to the entire SFM reconstruction structure.

[0138] Therefore, the AR experience area and the camera position distribution in the map have a close correlation. By using this "correlation", a very strong constraint can be achieved in the positioning process, that is, the potential user positioning result is greatly likely to be constrained to this position.

[0139] The absolute position information to be finally calculated is on the displacement vector of the 6dof positioning without scale information. In this step, the 6dof information of the similar map frame is used as prior knowledge, and such linear constraint can make the solving of the 3dof of the displacement part become the solving of a one-dimensional scale factor, so that the number of unknowns to be solved can be reduced, and a stable and accurate result can be obtained.

[0140] Further, Figure 10 is a schematic diagram of a similar map frame constraint positioning point according to an embodiment of the present application. As Figure 10 indicated, the scale s is calculated by using "the light center of the camera is closest to the k map frame distance under the constraint of the relative pose Toc" as the optimization condition. The specific optimization condition only indicates that a position is found on the vector t, so that the distance cumulative sum of the position to the light center position of the k map frame is minimum;

[0141] S706, obtain the rigid transformation matrix between the virtual camera coordinate system and the scene map coordinate system, and obtain the positioning result corresponding to the real-time image according to the rigid transformation matrix and the absolute pose transformation, and send the positioning result to the terminal.

[0142] The obtained scale s is used to restore the absolute transformation. Since the virtual camera coordinate system and the positioning map coordinate system are connected by the rigid transformation Two, the relative pose transformation Twc from the camera coordinate system to be positioned to the positioning map coordinate system is obtained through the formula Twc=Two×Toc.

[0143] It should be noted that the rigid transformation is represented by a matrix, and the specific numerical value is determined in the process of establishing the virtual camera coordinate system. According to the content recorded in the present application, a person skilled in the art can know how much the parameter is "offset" relative to the positioning map coordinate system in the process of artificially constructing the virtual camera coordinate system. This parameter is the above-mentioned rigid transformation two; how to obtain the numerical value does not affect the inventive points of the present application, and therefore, the embodiment will not be described again.

[0144] Finally, the positioning result is returned to the mobile terminal as the result of this visual positioning. After obtaining the positioning result, the terminal can further perform AR navigation, add interactive special effects, and the like according to the positioning result.

[0145] Through the above steps S701 to S705, compared with the long-range positioning method in the related art, the embodiment of the present application can avoid the instability of the pose solving caused by the inaccurate position of the 3D landmark, and make the distribution of the positioning result converge by introducing the way of Bayesian probability inference, and the hierarchical progressive solving process and the split calculation of the 6dof, so that the long-range positioning has high accuracy. Without adding auxiliary facilities, the outdoor long-range positioning accuracy can be improved, and the rendering effect of the AR virtual-real superposition has better consistency.

[0146] In some embodiments, based on the 3D-3D correlation, a relative pose transformation between the virtual camera coordinate system and the world coordinate system is calculated, including:

[0147] An epipolar geometry relationship corresponding to the 3D-3D correlation is obtained;

[0148] Based on the epipolar geometry relationship, a linear equation system of minimum observation is constructed, and by solving the linear equation system, a relative pose transformation [R|t] between the virtual camera coordinate system and the world coordinate system is calculated, wherein in the solving process of the relative pose transformation, the RANSAC algorithm is combined to screen an inlier set of 3D-3D observation pairs.

[0149] Wherein t only represents a direction vector, and any position on the direction vector is a possible positioning position. The obtained transformation can also be called a 6dof positioning without scale information, and the 3dof position in the real absolute pose is constrained by the direction vector t.

[0150] In some embodiments, the relative pose transformation is optimized based on a preset strategy, specifically including:

[0151] A nonlinear least squares model is established based on the 3D-3D observation pair inlier set screened in the above step S704;

[0152] Based on the nonlinear least squares model, the optimization condition that the orientation vector in the transformed virtual camera coordinate system is equal to the orientation vector in the to-be-solved camera coordinate system is used to calculate the optimized relative pose transformation, and the nonlinear optimization equation is shown in the following formula 3:

[0153] Voc' = R * Vop + t

[0154]

[0155]

[0156] Wherein Rt is the relative pose transformation, voc is the second observation vector, voc is the second observation vector converted according to the relative pose transformation, and the preset road point is constrained on the displacement vector. In the optimization process, the optimization result of the relative pose transformation is solved by iterative updating.

[0157] Preferably, Gauss-Newton or levenberg-marquardt iterative method can be used to update the solution, and the increment of each solution is -1*(Jt*J).inv()*Jt*b; wherein J is the matrix derivative of the objective function to Toc, Jt is the transpose of J, inv is the inverse function, and b is the residual of the objective function under the current parameters. It should be noted that if the roll and pitch are given on the terminal, they can be used as initial values in this optimization step, and the dimension is not optimized, and the optimized relative attitude Toc=[R|t] is obtained.

[0158] In some embodiments, the relative pose transformation obtained by the above steps only contains rotation and heading angle, and does not have real scale. It is necessary to calculate real scale information based on the similar map frame as the positioning prior, and restore the absolute pose transformation through the real scale information, which specifically includes:

[0159] Under the constraint of the relative attitude transformation, the optimization condition is constructed to minimize the distance from the optical center of the virtual camera to multiple similar map frames, wherein the positioning result is on the displacement vector of the relative attitude; according to the optimization condition, the least square equation is constructed and iteratively solved to obtain the real scale information.

[0160] As shown in Figure 10 , a position is found on the vector t, so that the cumulative sum of the distance from the position to the optical center position of the k map frame is minimized, and the objective function applied in this process is shown in the following formula (4):

[0161] e s =∑ i＝1...k ||s*t-c i || 2

[0162] Further, the least square form can be constructed to solve the real scale s, wherein the least square equation is represented by the following formula (5):

[0163] s=argmini||e s || 2

[0164] Similarly, Gauss-Newton or levenberg-marquardt iterative method can be used to update the solution, and the increment of each solution is -1*(Jt*J).inv()*Jt*b, wherein J is the derivative of the objective function to s, Jt is the transpose of J, inv is the inverse function, and b is the residual of the objective function under the current parameters.

[0165] The absolute position information to be calculated finally is on the displacement vector of the 6dof positioning without scale information, and such linear constraint can make the solving of the 3dof of the displacement part become the solving of a one-dimensional scale factor, so that the number of unknowns to be solved can be reduced, and a more stable positioning result can be obtained.

[0166] In some embodiments, the scale s obtained is used to restore the absolute transformation. Since the virtual camera coordinate system and the positioning map coordinate system are connected by the rigid body transformation Two, the absolute pose transformation Twc from the camera coordinate system to be positioned to the positioning map coordinate system can be obtained through the formula Twc = Two x Toc, and the change is transmitted to the mobile terminal as the result of this visual positioning.

[0167] The embodiment also provides an AR long-range visual positioning system, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the terms "module", "unit", "sub-unit" and the like can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware or a combination of software and hardware is also possible and is contemplated.

[0168] Figure 11 is a structural block diagram of an AR long-range visual positioning system according to an embodiment of the present application, as shown in Figure 11 The system includes a preprocessing module 10, a pose calculation module and a positioning result output module 40, wherein,

[0169] The preprocessing module 10 is configured to establish a 2D-3D association between a real-time image and a similar map frame, wherein the similar map frame is a map frame in a scene map matched with the real-time image, the real-time image is sent by a terminal,

[0170] and construct a virtual camera coordinate system corresponding to the scene map, and determine a first observation vector corresponding to a preset landmark point in the scene map in the virtual camera coordinate system,

[0171] and obtain a second observation vector corresponding to each preset landmark point in an entity camera coordinate system, and convert the 2D-3D association into a 3D-3D association according to the first observation vector and the second observation vector;

[0172] The pose calculation module 20 is configured to calculate a relative pose transformation between the virtual camera coordinate system and a world coordinate system according to the 3D-3D association,

[0173] and optimize the relative pose transformation based on a preset strategy, calculate real scale information with the optimized relative pose transformation as a constraint condition, and restore an absolute pose transformation through the real scale information;

[0174] The positioning result output module 40 is configured to obtain a rigid transformation matrix between the virtual camera coordinate system and the scene map coordinate system, and obtain a positioning result corresponding to the real-time image according to the rigid transformation matrix and the absolute pose transformation, and send the positioning result to the terminal.

[0175] By the system, the coordinate system of the positioning map is equivalent to a virtual camera coordinate system. The virtual camera coordinate system can convert the observation relationship between the world coordinate system and the camera coordinate system to be solved into the observation relationship between two cameras. Through such a setting, the world coordinate system origin to a 3D landmark point can constitute the above-mentioned three-dimensional orientation vector.

[0176] In addition, the camera coordinate system origin to the 3D landmark point can constitute another three-dimensional orientation vector mentioned above, and the two sets of three-dimensional vectors are a set of 3D-3D observations. All 3D-3D observations constitute the epipolar geometry relationship between the virtual camera coordinate system and the camera coordinate system to be positioned.

[0177] Since the positioning map landmark point position of the long-range view has a large uncertainty, the method in the embodiment can avoid the influence of the 3D landmark point position uncertainty. Further, after establishing the observation pair, in the embodiment, the above-mentioned epipolar geometry relationship is used to construct a linear equation set for solving the relative pose transformation from the virtual camera coordinate system to the camera coordinate system to be positioned. Since the virtual camera coordinate system and the positioning map coordinate system are equivalent, the relative pose transformation from the positioning map coordinate system to the camera coordinate system to be positioned is actually obtained.

[0178] The obtained transformation can also be called 6dof positioning without scale information. The absolute position information to be calculated is on the displacement vector of the 6dof positioning without scale information. Such linear constraints can make the solution of the displacement 3dof become the solution of a one-dimensional scale factor, and the number of unknowns can be reduced, and a more stable result can be obtained.

[0179] In one embodiment, Figure 12 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, as Figure 12 indicated, an electronic device is provided, which can be a server, and the internal structure diagram thereof can be as Figure 12 indicated. The electronic device includes a processor, a network interface, an internal memory, and a non-volatile memory connected by an internal bus, wherein the non-volatile memory stores an operating system, a computer program, and a database. The processor is configured to provide computing and control capabilities, the network interface is configured to communicate with external terminals through network connection, the internal memory is configured to provide an environment for the operation of the operating system and the computer program, the computer program is executed by the processor to implement an AR long-range visual positioning method, and the database is configured to store data.

[0180] Those skilled in the art can understand that Figure 12 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0181] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0182] The technical features of the above-mentioned embodiments can be combined in any way. In order to make the description simple, all possible combinations of the technical features in the above-mentioned embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0183] The above-mentioned embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. An AR (Augmented Reality) far-field visual positioning method, characterized in that, When applied on the server side, the method includes: A 2D-3D association is established between a real-time image and a similar map frame, wherein the similar map frame is a map frame in the scene map that matches the real-time image, and the real-time image is sent by the terminal; Construct a virtual camera coordinate system corresponding to the scene map, and determine the first observation vector corresponding to the preset road sign point in the scene map in the virtual camera coordinate system; Obtain the second observation vector corresponding to each preset landmark point in the physical camera coordinate system, and convert the 2D-3D association into a 3D-3D association based on the first observation vector and the second observation vector; Based on the 3D-3D association, calculate the relative pose transformation between the virtual camera coordinate system and the world coordinate system; The relative pose transformation is optimized based on a preset strategy. The optimized relative pose transformation is used as a constraint to calculate the real scale information, and the absolute pose transformation is recovered through the real scale information. Obtain the rigid body transformation matrix between the virtual camera coordinate system and the scene map coordinate system, and based on the rigid body transformation matrix and the absolute pose transformation, obtain the positioning result corresponding to the real-time image and send it to the terminal.

2. The method according to claim 1, characterized in that: Based on the geometric structure of the scene map, a virtual camera coordinate system corresponding to the scene map is established. Replace the 3D landmarks in the location map coordinate system with the azimuth vectors in the virtual camera coordinate system, and convert the observation relationship between the world coordinate system and the physical camera coordinate system into the relationship between the cameras; The virtual camera coordinate system meets preset conditions, including: The virtual camera's field of view is directed towards a preset landmark in the scene map; The preset landmarks in the scene map are evenly distributed to the left and right of the optical axis of the virtual camera; The virtual camera's field of view covers a preset proportion of the preset landmark points, and the field of view is within a preset range.

3. The method according to claim 1, characterized in that, Obtain the second observation vector corresponding to each of the preset landmark points in the physical camera coordinate system, including: Obtain camera intrinsic parameters sent by the terminal device; Based on the camera intrinsic parameters, the 2D feature points in the real-time image are back-projected onto the coordinate system of the physical camera to obtain the second observation vector.

4. The method according to claim 3, characterized in that, Based on the first observation vector and the second observation vector, the 2D-3D association is converted into a 3D-3D association, including: The 2D and 3D feature points in the 2D-3D association are replaced with the second observation vector and the first observation vector, respectively, to obtain the 3D-3D association. Among them, a pair of observation vectors forms a set of 3D-3D observation pairs, and all 3D-3D observation pairs form the epipolar geometric relationship between the virtual camera coordinate system and the physical camera coordinate system.

5. The method according to claim 4, characterized in that, Based on the 3D-3D association, the relative pose transformation between the virtual camera coordinate system and the world coordinate system is calculated, including: Obtain the epipolar geometry relationship corresponding to the 3D-3D association; Based on the epipolar geometry, a set of linear equations with minimum observation is constructed. By solving the set of linear equations, the relative attitude transformation between the virtual camera coordinate system and the world coordinate system is calculated.

6. The method according to claim 5, characterized in that, In the process of solving the relative attitude transformation, the RANSAC algorithm is used to select the set of interior points of the 3D-3D observation pair.

7. The method according to claim 6, characterized in that, The optimization of the relative attitude transformation based on a preset strategy includes: A nonlinear least squares model is established using the selected set of 3D-3D observation points. Based on the aforementioned nonlinear least squares model, the optimized relative attitude transformation is calculated with the condition that the azimuth vector in the transformed virtual camera coordinate system is equal to the azimuth vector in the camera coordinate system to be determined.

8. The method according to claim 7, characterized in that, Based on the nonlinear least squares model, and taking the azimuth vector in the transformed virtual camera coordinate system as equal to the azimuth vector in the undetermined camera coordinate system as the optimization condition, the optimized relative attitude pose transformation is calculated using the following formula. Wherein, Rt is the relative attitude transformation, voc is the second observation vector, voc′ is the second observation vector after transformation according to the relative attitude transformation, the preset landmark point is constrained on the displacement vector of the relative attitude transformation, and the optimization result of the relative attitude transformation is solved by iterative update during the optimization process.

9. The method according to claim 8, characterized in that, Calculating true scale information using the optimized relative pose transformation as a constraint, and recovering the absolute pose transformation using the true scale information, includes: Under the constraint of the relative pose transformation, the optimization condition is constructed by minimizing the distance between the optical center of the virtual camera and the distance between multiple similar map frames, wherein the positioning result is on the displacement vector of the relative pose; Based on the optimization conditions, a least squares equation is constructed and iteratively solved to obtain the true scale information.

10. The method according to claim 9, characterized in that, The method further includes: The server collects the pitch angle and roll angle parameters sent by the terminal; In the process of optimizing the relative attitude transformation, the pitch angle parameter and roll angle parameter are used as initial values ​​for optimization, and the optimization of the pitch angle dimension and roll angle dimension is omitted.

11. The method according to claim 1, characterized in that, Before establishing a 2D-3D association between the real-time image sent by the terminal and similar map frames, the method further includes: Receive real-time images and camera intrinsic parameters sent by the receiving terminal; Feature extraction is performed on the real-time image to obtain global and local features; The global features are matched in the scene map to obtain the similar map frame, wherein the scene map is a 3D map of the offline scene where the terminal is located.

12. The method according to claim 11, characterized in that, Establishing 2D-3D associations between real-time images and similar map frames includes: The 2D-3D observation pairs are established between the local features of the real-time image and the local features of the similar map frames, respectively, and the 2D-3D association is formed based on all the 2D-3D observation pairs.

13. An AR large-scene long-distance visual positioning system, characterized in that, The system includes: a preprocessing module, a relative pose calculation module, an absolute pose calculation module, and a positioning result output module, wherein... The preprocessing module is used to establish a 2D-3D association between a real-time image and similar map frames, wherein the similar map frames are map frames in the scene map that match the real-time image, and the real-time image is sent by the terminal. Furthermore, a virtual camera coordinate system corresponding to the scene map is constructed, and within the virtual camera coordinate system, the first observation vector corresponding to the preset landmark point in the scene map is determined. In addition, the second observation vector corresponding to each of the preset landmark points in the physical camera coordinate system is obtained, and the 2D-3D association is converted into 3D-3D association based on the first observation vector and the second observation vector; The relative pose calculation module is used to calculate the relative pose transformation between the virtual camera coordinate system and the world coordinate system based on the 3D-3D association. The absolute pose calculation module is used to optimize the relative pose transformation based on a preset strategy, calculate the real scale information using the optimized relative pose transformation as a constraint, and recover the absolute pose transformation using the real scale information. The positioning result output module is used to obtain the rigid body transformation matrix between the virtual camera coordinate system and the scene map coordinate system, and to obtain the positioning result corresponding to the real-time image based on the rigid body transformation matrix and the absolute pose transformation, and send it to the terminal.

14. The system according to claim 13, characterized in that, Based on the geometry of the scene map, the preprocessing module establishes a virtual camera coordinate system corresponding to the scene map. Replace the 3D landmarks in the location map coordinate system with the azimuth vectors in the virtual camera coordinate system, and convert the observation relationship between the world coordinate system and the physical camera coordinate system into the relationship between the cameras; The virtual camera coordinate system meets preset conditions, including: The virtual camera's field of view is directed towards a preset landmark in the scene map; The preset landmarks in the scene map are evenly distributed to the left and right of the optical axis of the virtual camera; The virtual camera's field of view covers a preset proportion of the preset landmark points, and the field of view is within a preset range.

15. The system according to claim 13, characterized in that, The preprocessing module obtains the second observation vector corresponding to each of the preset landmark points in the physical camera coordinate system, including: Obtain camera intrinsic parameters sent by the terminal device; Based on the camera intrinsic parameters, the 2D feature points in the real-time image are back-projected onto the coordinate system of the physical camera to obtain the second observation vector.

16. The system according to claim 13, characterized in that, The preprocessing module converts the 2D-3D association into a 3D-3D association based on the first observation vector and the second observation vector, including: The 2D and 3D feature points in the 2D-3D association are replaced with the second observation vector and the first observation vector, respectively, to obtain the 3D-3D association. Among them, a pair of observation vectors forms a set of 3D-3D observation pairs, and all 3D-3D observation pairs form the epipolar geometric relationship between the virtual camera coordinate system and the physical camera coordinate system.

17. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 12.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 12.

Images