Method for positioning a camera and computer readable storage medium

Abstract

This document describes a system and method for determining the location of a client device. Specifically, the client device receives images captured by a camera at its location. The client device identifies features in the images. These features can be line intersections, straight lines, curves, or any other features found in the images. The client device retrieves a 3D map of the environment from a map database and compares the identified features with the 3D map, which includes map features such as map line intersections, map lines, map curves, etc. The client device identifies the correspondence between the identified features in the images and the map features, and determines the client device's location in the real world based on this correspondence. The client device can display visualization data representing the location in a virtual world corresponding to its location in the real world.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 952,016, filed December 20, 2019, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to 3D mapping, and more specifically to using 3D line intersections to determine the location of a device. Background Technology

[0004] Parallel reality games can provide shared virtual worlds that are at least partially parallel to the real world, hosting various interactions that can attract a player community. Providing a virtual world with a geographical environment at least partially parallel to the real world allows players to navigate within the virtual world by navigating within the real world. During gameplay, players can view the virtual world using handheld or wearable devices that utilize computer-mediated reality technologies to add to, subtract from, or otherwise alter the player's visual or auditory perception of their environment.

[0005] However, accurately altering a player's visual perception of the environment often involves accurately determining the player's location in the real world. This can be difficult to determine because traditional positioning devices are insufficient to accurately pinpoint a player's location without a considerable margin of error. Therefore, improved methods are needed to determine the player's location in the real world. Summary of the Invention

[0006] In location-based parallel reality games, players navigate a virtual world by moving around in the real world using location-aware client devices such as smartphones. Many of the client devices used by players in parallel reality games can include positioning devices that track the player's location information as the player moves around in the real world while playing the parallel reality game. In various embodiments, the client device uses image data captured by cameras(s) on the device(s) to determine the player's location, which can be used to replace or supplement the player's location information collected from the positioning device. The client device can then generate augmented reality (AR) images based on the determined player location to overlay on the image data.

[0007] According to one embodiment, an AR platform determines the position of a camera in an environment used for parallel reality gaming. Using one or more images captured by the camera, the AR platform identifies line junctions in the images—points where lines intersect in space. The AR platform compares these line junctions to a three-dimensional (3D) map of the environment to determine whether the line junctions correspond to map line junctions in the 3D map. Based on one or more correspondences identified by the AR platform, the AR platform determines the position of the camera in the environment. The AR platform can generate computer-mediated reality images for display on a client device associated with the camera to reflect a position in the virtual world corresponding to a determined position in the real world.

[0008] In an additional or alternative embodiment, the AR platform can receive one or more images representing a near real-time view of the environment as seen by the camera and a geolocation location from a geolocation / global positioning system integrated into the client device. The AR platform estimates the distance between the camera's location at a specific point in time and one or more mapped points in the one or more images. Based on the estimated distance, the AR platform overlays one or more images onto a computer-generated image located at a specific location in a 3D map. As the player moves the camera to different locations in space, the computer-generated image remains positioned at that specific location.

[0009] These and other features, aspects, and advantages can be better understood with reference to the following description and the appended claims. The accompanying drawings illustrate specific embodiments and, together with the specification, serve to explain the various principles. However, the drawings should not be considered limiting. Rather, the scope of protection should be determined by the claims. Attached Figure Description

[0010] Figure 1 This is a block diagram of an AR computing system 100 according to one embodiment.

[0011] Figure 2A A line intersection point according to one embodiment is depicted.

[0012] Figure 2B The intersecting lines at the intersection point of the lines according to one embodiment are depicted.

[0013] Figure 2C The representation of line intersections in 2D and 3D is depicted according to one embodiment.

[0014] Figure 3 This illustrates a method according to one embodiment. Figure 1 The flowchart shows the process that the computing system can execute to generate and display augmented reality data.

[0015] Figure 4A concept diagram depicts a virtual world parallel to the real world according to one embodiment, which can serve as a game board for players in a location-based parallel reality game.

[0016] Figure 5 This is a flowchart illustrating a process for determining the position of a camera in an environment according to one embodiment.

[0017] Figure 6 This illustrates a suitable embodiment for use in... Figure 1 A block diagram of an example computer system used in a network computing environment. Detailed Implementation

[0018] A system and method determine the location of a client device in an environment and enable user interaction with the real world via the client device based on that location. Furthermore, the system can create a three-dimensional (3D) map of the environment (e.g., with a resolution of 1 cm) to determine the location of the client device. In various embodiments, the mapping is performed on the client (e.g., a phone or headset) and paired with a backend server that provides a previously compiled image and maps it back to the client device.

[0019] In one embodiment, the system uses one or more models to evaluate the camera's position (e.g., on a client device) based on one or more images captured by the camera. Using one or more models involves the system detecting line intersections within one or more images and projecting these line intersections between 3D and 2D space to determine if they correspond to map line intersections from a 3D map of the environment surrounding the camera. Based on the correspondence between line intersections and map line intersections, the system can triangulate the camera's position in space. Using this position, the system can enhance one or more images for a parallel reality game, allowing players using the client device to see virtual elements or other virtual information in the virtual world that correspond to their real-world location.

[0020] Example System

[0021] Figure 1 This is a block diagram of an AR computing system 100 according to one embodiment. The AR computing system 100 includes a client device 102 that collaborates with elements accessed via a network 104. For example, this element may be a component of a server device configured to generate AR data. In the illustrated embodiment, the client device 102 includes a game engine 106 and an AR platform 108. The game engine 106 renders a parallel reality game for a user of the client device 102 to play. The game engine may be... Game engine or other physics / rendering engine. AR platform 108 can perform segmentation and object recognition on image data captured by the camera of client device 102.

[0022] Figure 1 The AR platform 108 shown includes a complex visualization module 110, a localization and mapping module 112, a map retrieval module 114, and a deep learning module 116. In other embodiments, the AR platform may include alternative or additional modules, such as a training module or an object recognition module. Furthermore, in some embodiments, the AR platform may be located at a server device with elements and accessed via network 104, or the elements may be located at the AR platform 108.

[0023] The complex computer visualization module 110 performs client-side image processing. Such image processing may include image segmentation, local 3D estimation, etc. In some embodiments, the complex computer visualization module 110 may convert an image into a point cloud or process the image to add features described in the image to a 3D map of the environment surrounding the client device 102.

[0024] The location and mapping module 112 maps the environment surrounding the client device 102 and determines the location of the client device 102 within the environment. The location and mapping module 112 may use one or more of a variety of methods to map the environment and determine the location of the client device 102.

[0025] The positioning and mapping module 112 receives one or more images (or other image data, such as video) from the camera of the client device 102. In some embodiments, the positioning and mapping module 112 receives other sensor data (e.g., lighting data, image data, and / or GPS coordinates) captured by the client device 102, which the positioning and mapping module 112 can use to determine the location of the client device 102. The positioning and mapping module 112 can receive images (and sensor data) in real time as the client device 102 moves in its environment.

[0026] The positioning and mapping module 112 determines the location of the client device 102 in the environment based on the received images. In some embodiments, the positioning and mapping module 112 receives the real-world GPS coordinates of the client device 102 from a Global Positioning System located at the client device 102. The positioning and mapping module 112 can use these GPS coordinates to represent the location of the client device 102, or it can use these GPS coordinates as a reference for the general location of the client device 102, and use one or more other models or systems to determine a more accurate location of the client device 102.

[0027] In some embodiments, the localization and mapping module 112 uses Simultaneous Localization and Mapping (SLAM) software development kit (SDK) functionality to generate a map of the environment. Specifically, the SLAM SDK functionality may include a mapping system that constructs a point cloud based on images or other image data, which is then used to form the environment map. The localization and mapping module 112 may store the map locally or in a map database 124. Based on the map, the localization and mapping module 112 may use tracking to locate the position of the client device 102 in space. The localization and mapping module 112 further uses SLAM processes to reproject one or more animated or enhanced values ​​onto the image for display to the user of the client device. For example, based on the client device's location, the localization and mapping module 112 may add one or more virtual elements or other information from the virtual world to the user's image for viewing. Virtual elements, etc., are related to... Figure 4 Describe it.

[0028] In other embodiments, the positioning and mapping module 112 has multiple models that can be used to determine the location of the client device 102. These models may include point cloud-based models (e.g., provided by SLAM), plane matching models, line matching models, geographic information system (GIS) models, building recognition models, and landscape recognition models. The positioning and mapping module 112 can select one or more models for positioning based on sensor data captured by the client device 102. For example, plane and line matching models may be most accurate indoors, while landscape and building models may be most accurate outdoors. Therefore, the client device 102 can determine whether it is indoors or outdoors based on sensor data and select the more effective model for positioning.

[0029] In one exemplary embodiment, the positioning and mapping module 112 applies a line intersection model to an image to position the client device 102. Specifically, the line intersection model identifies one or more line intersections in one or more images. A line intersection is a point in space where two or more lines intersect, as illustrated. For example, an image of a table in front of a person might show line intersections where each edge (e.g., a line) of the table intersects. In some embodiments, the line intersection model may identify other features in the image, such as curves.

[0030] The 3D map of the line intersection model access environment can be stored locally on the client device or in a map database 124 accessed via network 104. The 3D map can contain map line intersections, which are line intersections already identified in the 3D map, or it can contain other features such as lines and curves. Alternatively, the map retrieval module 114 can access the 3D map used for the line intersection model and input the 3D map into the line intersection model.

[0031] The line intersection model compares identified line intersections (or, in some cases, features) with a 3D map to find corresponding line intersections (or features). Based on the correspondences found by the line intersection model, it determines the location of client device 102 in the environment. The process of using the line intersection model to determine the location of client device 102 will combine... Figures 2A to 2C Further description.

[0032] In one embodiment, to select one or more models, the localization and mapping module 112 assigns a score to each model, indicating the likelihood that the model will generate an accurate location for the client device 102. The score can be determined based on the current state of the client device 102 as indicated by sensor data. The localization and mapping module 112 selects one or more models for localization based on the scores and a set of selection criteria. For example, the client device 102 can select the model with the highest score, the three highest-scoring models, or all models with scores above a threshold.

[0033] Additionally or alternatively, the positioning and mapping module 112 can select one or more models based on which models were previously selected at similar locations. For example, if the GPS coordinates for client device 102 are within a threshold of GPS coordinates for which a plane-matching model was previously successfully used, the positioning and mapping module 112 can increase the score of the plane-matching model or select the plane-matching model, unless the score of another model exceeds the threshold amount for the plane-matching model. Similarly, if a particular model was previously selected and failed to provide an accurate location, the positioning and mapping module 112 can decrease the score of that model. Thus, the success and failure of models at similar GPS coordinates can be used to inform the positioning and mapping module 112 which models(s) to select to determine a more precise location (e.g., within one centimeter) than provided by GPS coordinates.

[0034] In a similar manner, the localization and mapping module 112 can consider the success and failure of models at locations similar to the current location of the client device 102, as indicated by sensor data. For example, the localization and mapping module 112 can classify the location as indoor or outdoor based on whether the client device 102 is currently indoors or outdoors based on sensor data indicating that the client device 102 is indoors or outdoors, and exclude or select certain models (or increase or decrease their corresponding scores). Similarly, the localization and mapping module 112 can select one or more models based on lighting levels (e.g., a model may be particularly effective relative to other models under low lighting conditions, and therefore, if the lighting level detected by the client device 102 is below a threshold, that model is preferred). Those skilled in the art will recognize other methods that can be used to influence the selection of which models(s) to favor those models most likely to produce accurate results under the current conditions.

[0035] The positioning and mapping module 112 can apply one or more selected models to generate a potential location for the client device 102. When only a single model is selected, the positioning and mapping module 112 can use the generated location as the actual location of the client device 102. However, if GPS coordinates are available, and the difference between the location determined using the model and the GPS coordinates exceeds a threshold amount, the positioning and mapping module 112 can apply additional models or otherwise seek to verify the current location.

[0036] When more than one model is selected, each model can generate a potential location for client device 102. Location and mapping module 112 determines the current location for client device 102 based on the potential locations. In one embodiment, location and mapping module 112 calculates the current location as the mean of the locations generated by the models. In another embodiment, location and mapping module 112 uses a weighted average of the potential locations. For example, location and mapping module 112 can use scores from the models in the model selection phase to weight the contribution of each model. Alternatively, location and mapping module 112 can preset weights to prioritize certain models over others. In another embodiment, location and mapping module 112 can combine potential locations in other ways, such as by using the location that best matches the device's GPS coordinates.

[0037] In some embodiments, once the location and mapping module 112 has determined the location of the client device 102, it retrieves AR data describing the virtual world to add to one or more images to be rendered at the client device 102. For example, the location and mapping module 112 may access a map database 124 (stored locally or on a server device accessed via network 104) to retrieve one or more virtual elements or other information from the virtual world to add to the images. Virtual elements, etc., are related to... Figure 4 Describe it.

[0038] Map retrieval module 114 retrieves previously generated maps. For example, map retrieval module 114 may access map database 124 via network 104 and send the information retrieved from the map database to one or more other modules of client device 102, such as location and mapping module 112. In some embodiments, map retrieval module 114 may locally store some maps (e.g., a map for the user's home location). Deep learning module 116 applies machine learning algorithms to object recognition. Deep learning module 116 may acquire machine learning algorithms after training via network 104. In some embodiments, deep learning module 116 may also provide the results of object recognition and / or user feedback to enable additional model training.

[0039] In the illustrated embodiment, components accessed via network 104 (e.g., at a server computing device) include an AR backend engine 118, which communicates with a world mapping module 120, an object recognition module 122, a map database 124, an object database 126, and a deep learning training module 128. In other embodiments, additional or different components may be included. Furthermore, this functionality may be distributed in a manner different from that described herein. For example, some or all of the object recognition functionality may be performed at a client device 102.

[0040] A world mapping module 120 merges different local maps together to create a synthetic real-world map. As previously described, GPS location data from the client device 102 that initially generated the map can be used to identify potentially adjacent or overlapping local maps. Pattern matching can then be used to identify overlapping portions of the maps or two local maps that are adjacent to each other (e.g., because they include representations of opposite sides of the same object). If two local maps are determined to overlap or be adjacent, a mapping indicating how the two maps are related to each other can be stored (e.g., in a map database).

[0041] The object recognition module 122 uses image / object information and collected 3D data to identify features in the real world represented in the data. In this way, the server device determines, for example, the location of a chair in 3D and accesses the object database 126 associated with that location. The deep learning module 128 can be used to fuse map information with object information. In this way, the system 100 can link the 3D information used for object recognition and fusion back to the map.

[0042] Map database 124 includes one or more computer-readable media configured to store map data generated by client device 102. The map data may include local maps of 3D point clouds stored in association with images, as well as other sensor data collected by client device 102 at locations. The map data may also include map information indicating geographic relationships between different local maps. Similarly, object database 126 includes one or more computer-readable media configured to store information about identified objects. For example, object database 126 may include a list of known objects (e.g., chairs, tables, trees, buildings, etc.) with corresponding locations, along with attributes of these objects. Attributes may be generic attributes of the object type or may be specifically defined for each instance of the object (e.g., all chairs may be considered furniture, but each location is individually defined). Although map database 124 and object database 126 are shown as a single entity, they may be distributed across multiple storage media on multiple devices (e.g., as a distributed database).

[0043] Example line intersection model

[0044] The positioning and mapping module 112 may employ a line intersection model, which uses a 3D map to locate the client device 102 based on one or more images it captures. The following paragraphs explain how one embodiment of the model is mathematically defined and applied.

[0045] 1. Parametric 3D intersection points

[0046] like Figure 2A An example of a 3D line intersection point according to one embodiment is depicted. The line intersection point includes a point Q and direction vectors (or, in some embodiments, lines or line segments) D1 and D2. Each direction vector is a unit vector with a length of 1. Each 3D line intersection point has seven degrees of freedom: three degrees of freedom for point Q, representing its position in 3D space, and two degrees of freedom for each direction vector (D1 and D2) because they are unit vectors.

[0047] To avoid constrained optimization, the line intersection model parameterizes the two direction vectors as follows, where X, Y, and Z represent the coordinates of the direction vectors in 3D space, θ represents the direction in which the angle with the +Z axis increases, and α represents the direction in which the angle increases counterclockwise from the +Z axis in the XY plane:

[0048]

[0049] Since the norm of a unit vector is 1, therefore

[0050] (cosθcosα) 2 +(cosθsinα)2 +(sinθ) 2

[0051] =cos 2 θ(cos 2 α+sin 2 α)+sin 2 θ

[0052] =ωs 2 θ+sin 2 θ

[0053] =1 (1) Therefore, the line intersection model can parameterize the 3D line intersection point into 7 vectors.

[0054] 2. Project the 3D intersection point onto 2D space.

[0055] The line intersection model projects the 3D line intersection point into 2D space to obtain two intersecting lines (L1 and L2), such as... Figure 2B As shown. Specifically, using point Q, a direction vector D1, and the camera pose (R, T), the 2D line function is defined as the cross product of two distinct points (P1 and P2) on the line, yielding the following equation:

[0056] P1∧P2

[0057] ~(RQ+T)∧(R(Q+D1)T)

[0058] ~(RQ+T)∧(RQ+T+RD1)

[0059] ~(RQ+T)∧(RQ+T)+(RQ+T)∧(RD1)

[0060] ~(RQ+T)∧(RD1)

[0061] ~R(Q∧D1)+T∧RD1

[0062] ~(RQ) x +T x R)D1 (2) where P1 and P2 are two points on the line, ∧ is the cross product, Q x T x It is a skew-symmetric matrix of vectors Q and T.

[0063] Therefore, for the direction vector D1, the 2D projection is (RQ) x +T x R)D1. The same process can be used to derive the 2D projection of the direction vector D2 from the line intersection model.

[0064] 3. Reprojection error

[0065] Line intersection point models identify keylines in one or more images to determine the error in creating a 2D projection from 3D line intersection points. Keylines are the boundary lines separating different color regions in an image. Since 2D projection is a function of lines, the line intersection point model can compare the point to the line distances from the two endpoints of the detected keyline. If the four endpoints are represented as u, v, x, and y on two keylines respectively, the line intersection point model can determine the distance for each endpoint. For example, the distance for u is... Where L = (A, B, C) is the 2D line equation of 2D line projection, and d 2 =A 2 +B 2 .

[0066] L=(RQ x +T x R)D (3)

[0067] If L1 and L2 are the 2D projections of two 3D lines respectively, then the actual residual is a 4-vector, as shown below:

[0068]

[0069] *Here, u, v, x, y are constants, and L1, L2, d1, d2 are variables of (Q, D1, D2, R, T).

[0070] 4. Optimize settings

[0071] To optimize the 2D projection settings, the line intersection model must solve an unconstrained minimization problem to determine the position of client device 102 (e.g., a camera) in the environment. Due to nonlinearity, in one embodiment, the line intersection model uses a Levenberg-Marquardt (or "LM") solver. The general update steps are as follows:

[0072] ΔP=-(J T J+λI) -1 J t r

[0073] Where J is the Jacobian of the reprojection error, r is the residual, and λ is the req term.

[0074] One or more images contain multiple landmarks (e.g., line junctions, point clouds, lines, etc.), and because typical images contain a large number of landmarks, line junction models can use an alternating direction approach to optimize settings. This allows line junction models to compute the stride of one landmark or frame of an image at a time while keeping all other variables fixed. One advantage of this approach is the relatively small size of the Jacobian matrix, which makes computation for line junction models faster.

[0075] In one embodiment, the pipeline followed by the line intersection model used for optimization can be summarized as follows:

[0076] When there is no convergence:

[0077] For each landmark:

[0078] Calculate LM step size

[0079] Update landmarks

[0080] For each frame:

[0081] Calculate LM step size

[0082] Update the frame.

[0083] The line intersection model uses this process to optimize and compute the Jacobian for each variable, as described below. These derivations are described with respect to the line intersection model, but in some embodiments, the line intersection model may not perform these derivations on every application, or the line intersection model may simply use the resulting equations to determine the location of the client device 102 instead of performing the derivations itself.

[0084] 5. Jacobi – 3D Intersection – Intersection Q

[0085] To calculate for Q, J Q The Jacobian should be 4k × 3, where k is the number of frames in one or more images. For each frame k, the residual rk is as follows:

[0086]

[0087] in

[0088] L k1 1(R k Q k +T k R k D1

[0089] L k2 =(R k Q k +T k R k D2

[0090] and

[0091]

[0092] Based on the residual rk, the Jacobian J for each frame k can be derived from the line intersection model. Q :

[0093]

[0094] Taking the first line as an example, the other lines can be analyzed similarly:

[0095] make

[0096] Let A = -RD x Q, B = T x RD (9) Now, according to (8) and (9):

[0097]

[0098] make N = 2B,A1, + 2B2A2,

[0099]

[0100] Now,

[0101]

[0102] because

[0103] Therefore, using these equations, the line intersection model can solve for Jacobi J. k Solve the first row, and similarly solve the other rows.

[0104] 6. Jacobi – 3D Intersection – Intersection Directions D1, D2

[0105] The line intersection model calculates the Jacobian for D1 and D2. Taking D1 as an example, since the derivation of D2 is essentially the same, the line intersection model starts from the dimensions of residuals and Jacobian, as described in Section 5 above. The residuals are the same as in equation (6), while the Jacobian is:

[0106]

[0107] Unlike Section 5, Both are zero. Therefore, the line intersection model can make J... k The size is reduced to R 2x2 And the residual r k The size is reduced to R 2x1 .

[0108] Furthermore, the line intersection model adds another step to the derivative chain:

[0109] make Using a method similar to that described above, we provide a solution for... The Jacobians are then combined to give the complete Jacobian for a single frame, J. k-R 4x7 .

[0110] 7. Jacobi – R, T

[0111] Deriving the Jacobian matrix R involves more work than deriving the Jacobian matrix for T. The Jacobian matrix R is derived from the line intersection model. The line intersection model aims to determine the Jacobian matrix with respect to three Euler angles. Jacobi.

[0112] Defining the transformation from Euler angles to rotation matrices yields:

[0113]

[0114] in

[0115] If R is expanded, where The model of line intersection points yields:

[0116] a 11 =cosθcosφ

[0117]

[0118] a 21 =cosθsinφ

[0119]

[0120] a 31 =-sinφ

[0121]

[0122] a 33 =cosφcosθ (16)

[0123] The following steps are similar to those in Sections 5 and 6, except that when calculating the derivative with respect to Euler angles, an extra step is inserted in the line intersection model within the chain rule, i.e. After calculation, J r ∈R 4x3 J T ∈R 4x3 ·J RT ∈R 4x6 .

[0124] Intersection triangulation

[0125] Using the equations derived above, the line junction model can determine the pose (position and orientation) of a camera (e.g., client device 102) in an environment given one or more images. In one embodiment, the line junction model uses a 3D map to determine endpoints within the 3D map, which are used as vertices for the 3D line junction. The line junction model uses the predicted pose to project the endpoints from 3D to 2D to determine whether the endpoints are reflected in one or more images. Due to errors in the projection, the line junction model calculates a reprojection error and updates one or more predicted poses of the camera for one or more images to reduce the reprojection error. By changing the camera pose over multiple iterations, the line junction model can converge its estimate of the camera pose until the calculated error is below a threshold. The line junction model uses this determined position as the camera's position in the environment.

[0126] The line intersection model uses the above equation to determine the camera's position in the environment. Specifically, given K frames and the camera's intrinsic and extrinsic parameters, it is expressed as:

[0127]

[0128] Where M i The form is P i The form is and

[0129] In each frame, there exists a combination of two 2D line equations and an intersection point (similar to...). Figure 2A Q or Figure 2B The intersection point J of the 2D lines formed by P) i =(l i1 , l i2 P i The two lines intersect here.

[0130] Note that the equation for a 2D line is the normal vector of the 2D line in 3D space, which is crossed by connecting the center of the camera to the two endpoints of the line. By assuming that all cameras are calibrated, the line intersection model can work with camera coordinates. For example, J i -(l i1 , l 12 P i All of them are in the camera coordinate system.

[0131] Given K correspondences for line intersections, the goal of a line intersection model is to triangulate these correspondences into 3D space to obtain 3D line intersections. The line intersection model can use any suitable 3D point triangulation method (e.g., using 2D and 3D intersections). In one embodiment, a 3D line intersection can be defined as J-(L1, L2, P), where L1 and L2 are two unit vectors indicating two directions, and P is the intersection point where two 3D lines intersect in 3D space.

[0132] Unlike two 2D lines, two 3D lines may not intersect, regardless of whether they are parallel. However, in one embodiment, the line intersection model enforces constraints regarding the intersection of these two 3D lines, allowing the model to represent the 3D line intersection using two direction vectors and a point. Since the two direction vectors are unit vectors as previously described, this representation has seven degrees of freedom. This is due to... Figure 2C express, Figure 2C Depicting the intersection of a point P in world space 220 and lines L1 and L2, and a point P in camera space 210. i and line l i1 and l i2 2D line intersection points. The line intersection point model or localization and mapping module 112 can create similar 2D line intersection point models by projecting one or more images onto the graph of vertices (points) and edges (lines). Figure 2C The environment is represented, and this representation is used to identify the intersection of lines.

[0133] From 3D to 2D

[0134] Figure 2C The relationship between the intersection points of 3D lines and the intersection points of 2D lines is depicted. Camera center O i 2D line intersection point P on the image frame i and any point l(l) on the line i1 The client device 102 provides a plane 200 in camera space 210 for its camera. The normal vector of plane 200 is the first row l i1 The equation.

[0135] To find any point q in the first row of the i-th frame i Find s in line 2 i :

[0136]

[0137] By {O i P i q i Any point on the plane that is crossed (denoted as x) i )satisfy:

[0138] l i1 T x i =0

[0139] Similarly, for {O i ,P i ,S i Any point y on the plane that is crossed i

[0140] P i2 T y i =0 (2)

[0141] Note that the direction vector also represents a point on that plane, but it lies in world space 220, not camera space 210. To bring L1 into camera space 210 on frame i, the line intersection model uses the following transformation.

[0142] q i =R i L1+t i

[0143] However, since L1 is a direction vector, the translation can be ignored, resulting in:

[0144]

[0145] Combining equations (2) and (3), we can derive the direct relationship between the direction vector of the intersection point of the 3D lines and the 2D lines, as shown below:

[0146]

[0147] This applies to all frames i = 1, ..., K.

[0148] The following formula can be used to transfer point P from world space 220 to camera space 210:

[0149] P i =R i P+t i (5)

[0150] Since its image on the frame should intersect two 2D lines simultaneously, combined with equations (2) and (5), the following is given:

[0151]

[0152] From 2D to 3D

[0153] (4) and (6) give the equations for solving 3D intersection points from the correspondence of 2D intersection points.

[0154] make Two k×3 matrices.

[0155] According to equation (4), this is

[0156]

[0157] Since both L1 and L2 are unit vectors, ||L1|| = ||L2|| = 1, the line intersection model can apply singular value decomposition (SVD) to N1 and N2:

[0158]

[0159] Let L1 be the singular vector associated with the minimum singular value, and perform the same operation on L2.

[0160] According to equation (6), let and The intersection point model can then be solved by finding the least squares solution of NP = b to obtain P. Therefore,

[0161]

[0162] As the camera (e.g., client device 102) moves through the environment and captures more images, the line intersection model can use the process described above to calculate the camera's position multiple times.

[0163] Example data stream

[0164] Figure 3 This is a flowchart illustrating a process performed by a client device 102 and a server device to generate and display AR data according to one embodiment. The client device 102 and the server device (collectively referred to as the "device") may be similar to... Figure 1 The examples shown represent server devices as elements connected to client device 102 via network 104. Dashed lines indicate data communication between client device 102 and server devices, while solid lines indicate data communication within a single device. In other embodiments, functionality may be distributed differently among devices, and / or different devices may be used.

[0165] At 302, sensor data is collected at client device 102 via one or more sensors on client device 102. In one embodiment, the sensor data includes images and inertial measurement information. Images are captured by one or more cameras. Inertial measurement information can be collected using a Global Positioning System (e.g., GPS) and a gyroscope / accelerometer module on client device 102. Additional sensor data, such as pressure levels, lighting levels, sound levels, audio data, etc., may be collected.

[0166] At position 304, client device 102 can maintain a local map repository. The local map repository can be included in... Figure 1 This can be in the localization and mapping module 112 shown or at another location on the client device 102. The local map repository includes local point cloud data. The point cloud data includes spatial locations forming a grid surface, which can be constructed into a 3D map of one or more environments, and this 3D map can also be stored in the local map repository.

[0167] If the map is initialized at 306, client device 102 can initiate SLAM functionality at localization and mapping module 112 at 308. The SLAM functionality includes a mapping system that constructs a point cloud and uses tracking to locate the camera's position in space (e.g., on client device 102). This example SLAM process further reprojects animation or augmented values ​​back into ground truth. If the map is not localized at 310, it can be stored in a local map repository at 304, and client device 102 uses the initialization module at 312 to create the map.

[0168] If a new viewpoint is detected in the environment at 314 (e.g., if a corner becomes an area that has not yet been mapped / imaged, or if there is overlap and not all currently visible parts of the real world have been mapped / imaged), then client device 102 can record data collected by local environment inference at client device 102 at 316. For example, when it is determined that client device 102 currently has a new viewpoint, an image captured by a camera having that viewpoint can be sent to the server device. Local environment inference can be used to determine when and how images with 3D data are transmitted between client device 102 and the server device. Local environment inference may include updated keyframes for a local mapping system (e.g., localization and mapping module 112), as well as serialized image and / or map data.

[0169] On the server side, where actions are performed by the server device, new viewpoint data (e.g., point cloud information including top grid data) can be stored in the cloud map repository 320 at 318. If the map is not stored in the cloud map repository 320 at 326, the server device can create a new map at 328 to store it in the cloud map repository 320. The server device can add different portions of the real-world map from the stored cloud map repository 320 and the object database 322. Cloud environment inference at 324 (including added data captured by the client device 102 and processed by the server device) can be sent back to the client device 102. The added data may include points and grids, as well as object data with semantic labels (e.g., walls or beds) to be stored in the local map repository at 304.

[0170] Concept art of virtual world

[0171] Figure 4 A conceptual diagram depicts a virtual world 410 parallel to the real world 400 according to one embodiment, which can serve as a game board for players of a location-based parallel reality game. Figure 1 The client device 102 can host parallel reality games (or other location-based games), and the virtual world 410 corresponds to... Figure 4 The real world shown is 400.

[0172] As shown in the figure, virtual world 410 can include geography parallel to that of real world 400. Specifically, the coordinate range defining a geographic region or space in real world 400 is mapped to a corresponding coordinate range defining a virtual space in virtual world 410. The coordinate range in real world 400 can be associated with towns, neighborhoods, cities, campuses, places, countries, continents, the globe, or other geographic regions. Each geographic coordinate within the geographic coordinate range is mapped to a corresponding coordinate in the virtual space of the virtual world.

[0173] A player's position in the virtual world 410 corresponds to their position in the real world 400. For example, player A, located at position 412 in the real world 400, has a corresponding position 422 in the virtual world 410. Similarly, player B, located at position 414 in the real world, has a corresponding position 424 in the virtual world. As a player moves within the geographic coordinates of the real world 400, they also move within the coordinates defining the virtual space within the virtual world 410. Specifically, when a player navigates within the geographic coordinates of the real world, a positioning system associated with the client device 102 carried by the player (e.g., a GPS system or other system used by the positioning and mapping module 112) can be used to track the player's position. Data associated with the player's position in the real world 400 is used to update the player's position within the corresponding coordinates defining the virtual space within the virtual world 410. In this way, by simply moving between corresponding geographic coordinate ranges in the real world 400, players can navigate a continuous trajectory within the coordinate range of the virtual space defined in the virtual world 410, without having to register at specific discrete locations in the real world 400 or periodically update their location information.

[0174] Parallel reality games can include multiple game objectives that require players to travel to and interact with various virtual elements and / or objects scattered throughout virtual locations in virtual world 410. Players can travel to these virtual locations by going to the corresponding locations of virtual elements or objects in real world 400. For example, the positioning system of client device 102 can continuously track the player's location, so that while the player is continuously navigating in real world 400, the player is also continuously navigating in parallel virtual world 410. The player can then interact with various virtual elements and / or objects at specific locations to achieve or perform one or more game objectives.

[0175] For example, refer to Figure 4 The game objective may require players to capture or claim ownership of virtual elements 430 located at various virtual locations within the virtual world 410. These virtual elements 430 may be linked to landmarks, geographic locations, or objects 440 in the real world 400. Real-world landmarks or objects 440 can be works of art, monuments, buildings, businesses, libraries, museums, or other suitable real-world landmarks or objects. To capture these virtual elements 430, players must travel to a landmark, geographic location, or object 440 linked to the virtual element 430 in the real world and must perform any necessary interactions with the virtual element 430 in the virtual world 410. For example, to interact with or capture a virtual element 430 linked to a specific landmark 440 via client device 102, Figure 4 Player A will have to travel to landmark 440 in the real world 400. Interaction with the virtual element 430 may require actions in the real world 400, such as taking photos and / or verifying, obtaining or capturing other information about the landmark or object 440 associated with the virtual element 430.

[0176] Game objectives may require players to use one or more virtual items collected by the player in a parallel reality game. For example, a player might have to move through virtual world 410 to find virtual items (such as weapons or other items) useful for completing the game objective. These virtual items can be found or collected by traveling to different locations in real world 400 or by performing various actions in either virtual world 410 or real world 400. Figure 4 In the example shown, a player uses virtual item 432 to capture one or more virtual elements 430. Specifically, a player can deploy virtual item 432 in the virtual world 410 near virtual element 430. Deploying one or more virtual items 432 near virtual element 430 can result in the capture of virtual element 430 for a specific player or for a specific player's team and / or faction.

[0177] In one particular implementation, as part of a parallel reality game, players might have to collect virtual energy. For example... Figure 4 As shown, virtual energy 450 can be scattered across different locations in the virtual world 410. Players can collect virtual energy 450 by traveling to the corresponding locations of virtual energy 450 in the real world 400. Virtual energy 450 can be used to power virtual items and / or to perform various game objectives in the parallel reality game. Players who lose all of their virtual energy 450 will be disconnected from the parallel reality game.

[0178] According to this disclosure, parallel reality games can be large-scale, multiplayer, location-based games where each participant in the parallel reality game shares the same virtual world. Players can be divided into different teams or factions and can work together to achieve one or more game objectives, such as capturing or claiming ownership of virtual element 430. In this way, parallel reality games can essentially be social games that encourage cooperation between players within the parallel reality game. Players from opposing teams can compete against each other in the parallel reality game. Players can use virtual item 432 to attack or hinder the progress of players from the opposing team.

[0179] Parallel reality games can have various features to enhance and encourage gameplay. For example, players can accumulate virtual currency or other virtual rewards that can be used throughout the parallel reality game. Players can advance through various levels as they complete one or more game objectives and gain experience in the parallel reality game. Players can communicate with each other through one or more communication interfaces provided in the parallel reality game. Players can also acquire enhanced "powers" or virtual items 432, which can be used to complete game objectives in the parallel reality game. Using the disclosure provided herein, those skilled in the art will understand that various other game features can be included in parallel reality games without departing from the scope of this disclosure.

[0180] Example Method

[0181] Figure 5 This is a flowchart illustrating the steps of a process (or "method") for determining the position of a camera in an environment according to one embodiment. Figure 5 This is shown from the perspective of the execution process 500 by the client device 102. However, some or all of the steps may be performed by other entities or components, such as by a server device. Furthermore, some embodiments may perform these steps in parallel, in a different order, or in different ways.

[0182] exist Figure 5In the illustrated embodiment, process 500 begins with client device 102 receiving 510 an image captured by its camera. In some embodiments, client device 102 may receive additional sensor data from sensors, including video, audio, light levels, sound levels, inertial data, orientation, pressure, and / or any other available data. Client device 102 identifies features in the image 520. Features may be line intersections, straight lines, curves, or any other features found in the image.

[0183] Client device 102 retrieves a 3D map of the environment from map database 124. The 3D map may have been created by client device 102 or server device using image data of the environment captured by one or more cameras and the relative positions of the cameras at the time the image data was captured. Client device 102 compares identified features with the 3D map of the environment 530, the 3D map including map features such as map line intersections, map lines, map curves, etc. Client device 102 identifies 540 the correspondence between the identified features 520 in the image and the map features. In some embodiments, client device 102 may apply pattern matching to the identified features 520 and the 3D map to determine whether the identified features 520 and the 3D map overlap or are adjacent (e.g., correspond). Client device 102 determines 550 the location of client device 102 based on the correspondence(s). This location may be a real-world location corresponding to a location in a virtual world for a parallel reality game. If this location is determined at server device, server device may send a view of the location in the virtual world to client device 102 for the player to view via a display.

[0184] In some embodiments, client device 102 may select and apply other positioning models to generate potential locations for client device 102. For example, each model may output a potential location. The potential location may be defined based on a coordinate system within a local map (e.g., selected based on the GPS coordinates of client device 102). For example, GPS coordinates may indicate where client device 102 might be located within a building, thus using a local map of the building. Client device 102 may compare the determined location 550 with the potential location to determine a more accurate location of client device 102 within the building (e.g., aiming with one centimeter accuracy). In some embodiments, client device 102 may change the determined location 550 based on the potential location, or may combine the determined location 550 with the potential location to generate a new location for client device 102. Furthermore, weighted average combinations or other suitable techniques may be used to generate a new location from the determined location 550 and the potential location.

[0185] Furthermore, the client device 102 can check the feasibility of the determined location 550 based on sensor data. For example, the client device 102 can determine whether the determined location 550 is within a threshold distance of the location indicated by GPS coordinates, whether the illumination level detected in the image is consistent with the expectation for the determined location 550, whether the image captured by the client device 102 is consistent with other images captured at similar locations, and so on.

[0186] In some embodiments, client device 102 can receive one or more additional images from an additional client device connected to client device 102 in a parallel reality game. The additional images may show client device 102 in the environment, and client device 102 can use these additional images to determine the location of the client device.

[0187] Computer Machine Architecture

[0188] Figure 6 This is a high-level block diagram illustrating an example computer 600 suitable for use as a client device 102 or a server. The example computer 600 includes at least one processor 602 coupled to a chipset 604. The chipset 604 includes a memory controller hub 620 and an input / output (I / O) controller hub 622. Memory 606 and a graphics adapter 612 are coupled to the memory controller hub 620, and a display 618 is coupled to the graphics adapter 612. Storage device 608, keyboard 610, pointing device 614, and network adapter 616 are coupled to the I / O controller hub 622. Other embodiments of the computer 600 have different architectures.

[0189] exist Figure 6 In the illustrated embodiment, storage device 608 is a non-transitory computer-readable storage medium, such as a hard disk drive, optical disc read-only memory (CD-ROM), DVD, or solid-state storage device. Memory 606 stores instructions and data used by processor 602. Pointing device 614 is a mouse, trackball, touchscreen, or other type of pointing device and is used in conjunction with keyboard 610 (which may be an on-screen keyboard) to input data into computer system 600. Graphics adapter 612 displays images and other information on display 618. Network adapter 616 couples computer system 600 to one or more computer networks.

[0190] Depend on Figure 1 The type of computer used by the entity may vary depending on the embodiment and the processing power required by the entity. For example, the server may include a distributed database system comprising multiple blade servers working together to provide the described functionality. Furthermore, the computer may lack some of the aforementioned components, such as the keyboard 610, graphics adapter 612, and display 618.

[0191] Those skilled in the art can make various uses, modifications, and deviations from the apparatus and techniques disclosed herein without departing from the described concepts. For example, components or features illustrated or described in this disclosure are not limited to their illustrated or described location, arrangement, or context. Examples of apparatuses according to this disclosure may include all, fewer, or different components compared to those described with reference to one or more of the foregoing figures. Therefore, this disclosure is not limited to the specific implementations described herein but should be given the broadest possible scope consistent with any claims that may be published and their equivalents.

Claims

1. A method for locating said camera in an environment from one or more images captured by a camera, the method comprising: Receive one or more images captured by the camera of the client device; Identify the line intersections in the one or more images; The identified line intersections are compared with the 3D map of the environment by identifying endpoints within the 3D map and using those endpoints as vertices for 3D line intersections in the 3D map; By using the corresponding predicted pose, the determined endpoints are iteratively projected from 3D to 2D to identify predicted poses with calculation errors below a threshold, and the correspondence between the identified line intersections and the 3D line intersections in the 3D map is identified. as well as The correspondence is used to determine the camera's pose in the environment.

2. The method according to claim 1, further comprising: A representation of the environment surrounding the camera is created by projecting the one or more images onto a graph of vertices and edges; The line intersections identified therefrom are identified in the representation, and the line intersections identified include overlapping edges in the representation.

3. The method according to claim 1, further comprising: Image data for the virtual game is received from an additional client device connected to the client device.

4. The method of claim 1, wherein comparing the identified line intersections with a 3D map of the environment comprises: Pattern matching is applied to the intersection points of the marked lines and the 3D map to determine whether the intersection points of the marked lines and the 3D map overlap or are adjacent.

5. The method of claim 1, wherein the 3D map is created using image data of the environment captured by one or more cameras and the relative positions of the one or more cameras when the image data is captured.

6. The method of claim 1, wherein the determined pose corresponds to a position in the virtual world.

7. The method according to claim 6, further comprising: Send a view of the location in the virtual world corresponding to the determined pose to the client device.

8. The method of claim 1, wherein each of the identified line intersections has seven degrees of freedom, wherein three degrees of freedom are for the intersection and two degrees of freedom are for each of the two direction vectors.

9. The method of claim 1, wherein the one or more images are each optimized frame-by-frame by utilizing a Levenburg-Marquardt solver to compute updates for line intersections.

10. The method of claim 1, wherein identifying the line intersection points in the one or more images comprises: Determine 2D line equations for lines in the one or more images, wherein each 2D line equation describes a normal vector in 3D space; Determine the 2D line intersection points in each of the one or more images, wherein each 2D line intersection point includes two 2D line equations and an intersection point; as well as The 2D line intersection point is triangulated into 3D space to obtain the 3D line intersection point.

11. The method of claim 10, wherein triangulation of the 2D line intersection points comprises: Transform the coordinates of the 2D line intersection points from each of the one or more images into a camera coordinate space; as well as The converted 2D line intersection points are transformed into 3D line intersection points.

12. A non-transitory computer-readable storage medium comprising instructions for locating the camera in an environment from one or more images captured by the camera, the instructions comprising: Instructions for receiving one or more images captured by the camera of the client device; Instructions for identifying line intersections in the one or more images; Instructions for comparing identified line intersections with a 3D map of the environment by determining endpoints within a 3D map and using those endpoints as vertices for 3D line intersections in the 3D map; Instructions for iteratively projecting the determined endpoints from 3D to 2D using a corresponding predicted pose to identify predicted poses with computational errors below a threshold, and for identifying the correspondence between the identified line intersections and 3D line intersections in the 3D map. as well as Instructions for using the correspondence to determine the position of the camera within the environment.

13. The non-transitory computer-readable storage medium according to claim 12, wherein the instructions further include: Instructions for creating a representation of the environment around the camera by projecting the one or more images onto a graph of vertices and edges; The line intersections identified therefrom are identified in the representation, and the line intersections identified include overlapping edges in the representation.

14. The non-transitory computer-readable storage medium according to claim 12, wherein the instructions further include: Instructions for receiving image data for a virtual game from an additional client device connected to the client device.

15. The non-transitory computer-readable storage medium of claim 12, wherein the instructions for comparing the identified line intersections with the 3D map of the environment further include: Instructions for applying pattern matching to the intersection of the identified lines and the 3D map to determine whether the intersection of the identified lines and the 3D map overlap or are adjacent.

16. The non-transitory computer-readable storage medium of claim 12, wherein the 3D map is created using image data of the environment captured by one or more cameras and the relative positions of the one or more cameras at the time the image data is captured.

17. The non-transitory computer-readable storage medium of claim 12, wherein the determined location corresponds to a location in a virtual world.

18. The non-transitory computer-readable storage medium according to claim 17, wherein the instructions further include: Instructions for sending to the client device a view of the location in the virtual world corresponding to the determined location.

19. The non-transitory computer-readable storage medium of claim 12, wherein each of the identified line intersections has seven degrees of freedom, wherein three degrees of freedom are for the intersection and two degrees of freedom are for each of the two direction vectors.

20. A computer system, comprising: Computer processor; as well as A non-transitory computer-readable storage medium stores instructions for locating the camera in an environment from one or more images captured by the camera, the instructions performing actions when executed by the computer processor, the actions including: Receive one or more images captured by the camera of the client device; Identify the line intersections in the one or more images; The identified line intersections are compared with the 3D map of the environment by identifying endpoints within the 3D map and using those endpoints as vertices for 3D line intersections in the 3D map; By using a corresponding predicted pose, the determined endpoints are iteratively projected from 3D to 2D to identify predicted poses with computational errors below a threshold, and the correspondence between the identified line intersections and 3D line intersections in the 3D map is identified; and The correspondence is used to determine the position of the camera within the environment.

Images