Merging local maps from mapping devices

Abstract

To provide merging of a local map from a mapping device.SOLUTION: An augmented reality system generates computer-mediated reality on a client device. The client device has a sensor including a camera configured to capture image data of an environment. The augmented reality system generates a first 3D map of an environment around the client device based on captured image data. A server receives image data captured from a second client device in the environment and generates a second 3D map of the environment. The server links the first and second 3D maps together into a single 3D map. The single 3D map may be of a real-world graphical representation using a node representative of a 3D map generated by the image data captured at the client device and an edge representative of a transformation between nodes.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates generally to computer-mediated reality systems, and more particularly to an augmented reality (AR) system that combines 3D maps generated from data collected by client devices into a single 3D map. [Background technology]

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 952,036, filed December 20, 2019, which is incorporated by reference in its entirety.

[0003] Parallel reality games may provide a shared virtual world that parallels at least a portion of the real world, capable of hosting a variety of interactions that can captivate a community of players. Providing a virtual world with a parallel geography to at least a portion of the real world allows players to navigate the virtual world by navigating the real world. During play, players may view the virtual world through a portable or wearable device, and computer-mediated reality technologies are used to add, subtract, or otherwise alter the player's visual or auditory perception of the environment.

[0004] However, accurately altering a player's visual perception of the environment typically involves knowing the player's location in the real world precisely. This can be difficult to ascertain because conventional positioning devices are not accurate enough to determine a player's location without a significant margin of error. Therefore, a system for mapping the real world captured by a player's mobile device's camera is desired to assist in determining the mobile device's location in the future. Summary of the Invention

[0005] In location-based parallel reality games, players use location-aware client devices, such as smartphones, to navigate the virtual world by moving through the real world. Many client devices map the player's environment using image data captured by a camera on the device, which may determine the player's location, augmented reality (AR) imagery to overlay on the captured image data, etc. These maps may describe the same environment, but because they were captured by different client devices, the maps may have different coordinate spaces and capture different perspectives of the environment. To create a single 3D map of the environment, the generated maps may be concatenated together based on the image data, location data, and / or the client device that captured the data.

[0006] According to certain embodiments, a system connected to multiple client devices via a network receives a first set of image data captured by a camera integrated into a first client device. The first set of image data represents a near-real-time view of a first area around the first client device. The system generates a first 3D map based on the first set of image data. The 3D map spatially describes the first area around the first client device. The system receives a second set of image data representing a near-real-time view of a second area around a second client device and generates a second 3D map based on the second set of image data. The system analyzes the first and second 3D maps to identify common features and combines the first and second 3D maps into a single 3D map based on the common features.

[0007] The single 3D map may be a graph of nodes, each representing a 3D map generated by image data captured by a client device. Each node may be associated with a different coordinate space based on the client device that captured the image data, and the graph may include edges between the nodes that represent transformations between the coordinate spaces. The system may use the graph to determine the location of the client device in the environment.

[0008] These and other features, aspects, and advantages may be better understood with reference to the following description and appended claims. The accompanying drawings illustrate particular embodiments and, together with the description, serve to explain various principles. However, the drawings should not be considered limiting. Rather, the scope of protection should be determined from the claims. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates a networked computing environment for generating and displaying augmented reality data, according to one embodiment. [Figure 2] FIG. 1 is a block diagram of a OneWorld mapping module 120 according to one embodiment. [Figure 3] 1 is a flowchart illustrating a process executable by an AR computing system for generating and displaying augmented reality data, according to one embodiment. [Figure 4] FIG. 1 is a conceptual diagram of a virtual world parallel to the real world that can act as a game board for players of a location-based parallel reality game, according to one embodiment. [Figure 5] 1 is a flowchart illustrating concatenating a first 3D map and a second 3D map into a single 3D map of an environment, according to one embodiment. [Figure 6] 1 is a flowchart illustrating generating a single 3D map of an environment based on synchronization, according to one embodiment. [Figure 7]FIG. 1 is a high-level block diagram illustrating an example of a computer suitable for use as a client device or a server, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Systems and methods combine two or more local maps into a single map. The single map can be used to enable augmented reality interaction in a virtual world parallel to the real world. In various embodiments, the local maps are stitched together based on the inclusion of common features, synchronization data indicating the relative locations of the client devices that generated the local maps, or both.

[0011] In one embodiment, the system generates a 3D map using images and Global Positioning System (GPS) coordinates on a client device (e.g., a portable or wearable electronic device). The 3D map is constructed with a camera recording module and an inertial measurement unit (IMU), such as an accelerometer and gyroscope. The images and GPS coordinates are sent to a server. The server and client device process the data together to establish objects and geometry and determine potential interactions. An example of a potential interaction is moving virtual elements in a room with AR animation.

[0012] By using the images and 3D maps together, the system can achieve object detection and geometry estimation using neural networks or other types of models. An example of a neural network is a computational model used in machine learning that uses a large collection of connected simple units (artificial neurons). Software connects the units so that, if the combined input signal is large enough, the units emit a unique output signal. The system may also use deep learning (e.g., multi-layer neural networks) to contextually understand the AR data. Other types of models may include other statistical models or other machine learning models.

[0013] The system aggregates local maps to create one or more global maps (e.g., by linking the local maps together). The aggregated maps are combined on a server into a single global map, providing a digital map of the environment, or "world." For example, two local maps generated by one or more devices may be represented as nodes in different coordinate spaces. Those containing portions of the local maps that match within a predetermined threshold for any combination of similar GPS coordinates, similar images, and similar sensor data may be determined to contain common features (e.g., "overlap" in space). Thus, the system may link the two nodes together with an edge representing a transformation between the coordinate spaces of the nodes. The linked node may be included in a graph of nodes representing other local maps created using images captured by client devices. This graph represents a single global map and may help maintain consistency between virtual worlds represented on diverse client devices.

[0014] Additionally, in some embodiments, the system may stitch a local map to a world map based on edges or common features contained within the local map. The world map may store animations for the virtual world at specific GPS coordinates and may further be indexed through 3D points and visual images down to specific locations in the world (e.g., at a resolution on the order of 1 foot / 30 centimeters). In another example, the system may stitch a local map based on synchronization data indicating the relative location of the client device that generated the local map as it traverses the environment.

[0015] An exemplary process maps data to and from the cloud. In one embodiment, the map is a collection of 3D points in space, such as a point cloud, that represents the world in a manner similar to 3D pixels. Image data is transmitted along with the 3D map, if available and useful. In one example, the 3D map data is transmitted without the image data.

[0016] In various embodiments, the client device generates the 3D map using a 3D algorithm executed by a processor. The client device transmits images, the 3D map, GPS data, and any other sensor data (e.g., IMU data, any other location data) in an efficient manner. For example, images may be selectively transmitted so as not to bog down transmission or processing. In one example, images may be selectively transmitted when they show a new perspective of the environment and not transmitted when they simply show a previously seen perspective of the environment. Images are designated for transmission by the system, for example, when the client device's camera field of view has minimal overlap with previous images from past or recent camera poses, or when a perspective in the image has not been observed for an amount of time that depends on the expected movement of the object. As another example, an image may be provided if a threshold amount of time or more has passed since a previous image from the current (or substantially overlapping) perspective was provided. This may allow stored images associated with the map to be updated to reflect a more current (or at least recent) state of the real-world location depicted by the image.

[0017] In various embodiments, a cloud-side device, such as a server, includes a real-time detection system that uses the 3D data and images to detect objects and estimate the geometry of the real-world environment depicted in the images. For example, a 3D map (e.g., a semi-dense and / or dense 3D reconstruction) of a non-photorealistic room can be determined from the images. The server fuses the images and 3D data with the detection system to build a consistent, easily indexed 3D map of the world, or a synthetic real-world map using GPS data. Once stored, the real-world map can be searched to find previously stored animations and other virtual objects.

[0018] In various embodiments, mapping and tracking occurs on the client device. The client device collects a sparse reconstruction of the real world (digitizes the world) along with the position of the client device's camera relative to the real world. Mapping involves creating a point cloud, or collection of 3D points. The client device returns the sparse representation to the server by serializing and transmitting the point cloud information and GPS data. Cloud processing enables multiplayer functionality (sharing map data in or near real time between independent client devices), working physical memory (storing map and animation data for future experiences not stored locally on the device), and object detection.

[0019] The server includes a database of maps and frames. Each frame includes one or more sensor data, such as the pixels forming the image, a pose with respect to a coordinate space, camera intrinsics (e.g., camera parameters such as focal length), feature points, and / or feature descriptors. The server uses the GPS data to determine whether a real-world map has been previously stored for the real-world location. If the location is identified, the server may transmit the stored map to the client device.

[0020] (Augmented Reality Computing System) 1 is a block diagram of an AR computing system 100 including a client device 102 cooperating with elements accessed over a network 104, according to one embodiment. For example, the elements may be components of a server for generating AR data. In some embodiments, the client device 102 is a computing device that a user can use to access a parallel reality game (e.g., an augmented reality game) or another augmented reality system. The client device captures image data (also referred to as images) via an on-device camera, 3D data, GPS data, etc. The client device 102 includes, for example, a game engine 106 (e.g., the UNITY® game engine or another physics / rendering engine) and an AR platform 108.

[0021] The game engine 106 may facilitate a parallel reality game (or other AR program) on the client device 102. For example, the game engine 106 may receive interactions that a user has with the client device 102, such as the user inputting information through the client device 102's interface or the user moving the client device within the real world. The game engine 106 may display parallel reality game information to the user through the interface based on these interactions. The game engine 106 may locally store information for the parallel reality game, including virtual elements available at virtual locations in the virtual world that correspond to locations in the real world. Alternatively, the game engine 106 may access game board information describing the virtual world on a server and continuously communicate with the server to facilitate the parallel reality game on the client device 102. Parallelism between the virtual world and the real world for parallel reality games is further described in connection with FIG. 4.

[0022] The AR platform 108 may perform segmentation and object recognition on data captured by the client device 102. The AR platform includes a combined vision module 110, a localization and mapping module 112, a map lookup module 114, and a deep learning module 116. In some embodiments, the AR platform includes alternative or additional modules.

[0023] The complex computer vision module 110 performs client-side image processing. The complex computer vision module 110 receives image data captured by a camera on the client device 102 and performs image processing on the image data. The image processing may include image segmentation and local 3D estimation.

[0024] The localization and mapping (e.g., SLAM) module 112 maps the environment around the client device 102 based on image data and GPS data captured by the client device 102. In particular, the SLAM module 112 creates one or more local maps, each representing a portion of the real world as seen by the data captured by the client device 102. The SLAM module 112, in some embodiments, may also determine the location of the client device 102 in the environment. The SLAM module 112 includes a mapping system that creates the local map, which may include point, line, and plane geometry. Additionally, the SLAM module 112 may build a point cloud and determine the position of a camera (e.g., the client device 102) in space using tracking information captured by the client device 102. In other embodiments, the SLAM module may build a map using the image data and tracking information. The SLAM module 112 also reprojects from the virtual world back to the real world by overlaying animated or augmented values ​​on the image data captured by the client device 102, which is presented via the display of the client device 102. In other embodiments, the SLAM module 112 may use different or additional approaches to mapping the environment around the client device 102 and / or determining the position of the client device 102 within that environment.

[0025] In some embodiments, the SLAM module 112 may synchronize the position of the client device 102 with another client device before generating the local map of the environment. For example, the SLAM module may receive image data of a machine-readable code (e.g., a QR code) in the environment and synchronize the position of the client device 102 with other client devices that have captured images of the same machine-readable code. The SLAM module 112 may store this information as synchronization data for the local map indicating the position of the environment. In another example, if the image data includes a field of view of another client device, which the SLAM module 112 may determine from the image data or which a user may indicate via the client device 102, the SLAM module 112 may store synchronization data for the local map indicating that the client device 102 was co-located with the other client device and reference that local map.

[0026] The map lookup module 114 looks up maps generated by the SLAM module 112. The map lookup module 114 looks up previously generated maps from a map database 124 (e.g., via the network 104), which is described further below. In some embodiments, the map lookup module 114 may store several maps locally on the client device 102, such as a map of the user's home location. The map lookup 114 may look up maps based on notifications from the game engine 106 and send the maps to the game engine 106 for use in facilitating the parallel reality game.

[0027] The deep learning module 116 applies machine-learned models for object recognition on the map. The deep learning module 116 receives the map from the map search module 114. The deep learning module 116 applies one or more machine-learned models to perform interest point or feature point detection (e.g., using Scale Invariant Feature Transform (SIFT) or Orientation Fast and Rotation Brief (ORB)) along with object detection and classification. For example, the deep learning module 116 may apply a machine-learned model to the map to determine objects contained within the map. The machine-learned model may be a classifier, a regression model, etc. The deep learning module 116 may obtain the machine-learned model after training in an external system (e.g., via the network 104). In some embodiments, the deep learning module 116 may provide object recognition results and / or user feedback to enable further model training.

[0028] The AR computing system 100 includes elements that the client device 102 can access over a network 104. These elements may be located on a remote server and include an AR backend engine 118 in communication with a one-world mapping module 120, an object recognition module 122, a map database 124, an object database 126, and a deep learning training module 128. In other embodiments, additional or different components may be included. Furthermore, functionality may be distributed differently than described herein. For example, some or all of the object recognition functionality may be performed on the client device 102 in some embodiments.

[0029] The one-world mapping module 120 fuses different local maps to create a composite real-world map (e.g., a single 3D map of the real world). The single 3D map may be represented as a graph of nodes connected to each other by edges. Each node may represent a map generated by a client device 102, which may be the client device 102 illustrated in FIG. 1 or another client device connected to a server for a parallel reality game. Each map may have its own coordinate space based on the client device 102 that generated the map, or the coordinate space of the same device may vary over time (e.g., due to GPS drift or changing conditions). The edges connecting the nodes may represent transformations between the nodes' coordinate spaces. The one-world mapping module 120 may add new nodes and edges to the single 3D map when it receives new maps from the client devices 102 via the network 104. The one-world mapping module 120 stores the single 3D map in a map database 124.

[0030] In an exemplary use case scenario, the one-world mapping module 120 may determine edges between nodes of the local maps even when gaps exist between the local maps. For example, the one-world mapping module 120 may receive nodes of the local maps, each containing a line segment missing a segment connecting two other segments. The one-world mapping module 120 may provisionally extend each line segment beyond that shown in the local map by a specified amount (e.g., 10 centimeters, 1 meter, or up to infinity). Assuming the relative positions of the local maps are known (e.g., based on feature analysis or position synchronization identifying common features, as described above), the one-world mapping module 120 may determine that both line segments in each local map are part of the same line. For example, if a projection of one line overlaps with the other line within a threshold (e.g., 1 millimeter, 1 centimeter, etc.), the one-world mapping module 120 may determine that the two segments are part of the same line. Thus, the one-world mapping module 120 may determine edges between the nodes using the missing segments connecting the lines and add the missing segments to one or both of the local maps. The One World Mapping module 120 and the single 3D map are further described in connection with FIG.

[0031] Map database 124 includes one or more computer-readable media configured to store map data (i.e., "maps") generated by client device 102. The map data may include a local map of a 3D point cloud stored in association with imagery and other sensor data collected by client device 102 at a location. The map data may also include mapping information indicating the geographic relationships between different local maps and a single 3D map representing the real world or a particular environment within the real world. Although map database 124 is shown as a single entity, it may be distributed across various storage media in various devices (e.g., as a distributed database).

[0032] The object recognition module 122 uses object information from the images and 3D data captured by the client devices 102 to identify real-world features represented in the data. For example, the object recognition module 122 may determine that a chair is in a 3D position in the environment and add object information describing the chair's 3D position to the object database 126. The object recognition module 122 may perform object recognition on maps stored in a map database, image data captured by one or more client devices 102, or maps generated by one or more client devices 102. The object recognition module may further update the object information stored in the object database 126 after performing object recognition on new image data of the same environment. The object recognition module 122 may continually receive object information from images captured from various client devices 102 to add to the object database 126.

[0033] In some embodiments, object recognition module 122 may further distinguish detected objects into various categories. In one embodiment, object recognition module 122 may identify objects in a captured image as either stationary or transient. For example, object recognition module 122 may determine that a tree is a stationary object. In a subsequent example, object recognition module 122 may update stationary objects less frequently compared to objects that may be determined to be transient. For example, object recognition module 122 may determine that an animal in a captured image is transient and may remove the object if the animal is no longer present in the environment in a subsequent image.

[0034] The object database 126 includes one or more computer-readable media configured to store object information about recognized objects. For example, the object database 126 may include a list of known objects (e.g., chairs, desks, trees, buildings, etc.) along with the object's corresponding location and object properties. The properties may be general to a type of object or may be specifically defined for each instance of the object (e.g., all chairs may be considered furniture, but the location of each chair may be individually defined). The object database 126 may further distinguish objects based on each object's object type. An object type may group all objects in the object database 126 based on similar characteristics. For example, all objects of a plant object type may be objects identified by the object recognition module 122 as plants, such as trees, bushes, grass, vines, etc. In some embodiments, the system may learn to distinguish between relatively stable (e.g., stationary) features and more dynamic features. For example, the system may learn that chairs tend to move around somewhat, while tables tend to stay in roughly the same place for long periods of time. Although the object database 126 is shown as a single entity, it may be distributed across various storage media on various devices (eg, as a distributed database).

[0035] The deep learning module 128 fuses the map data and the object information. In particular, the deep learning module 128 may retrieve maps from the map database 124 or one or more client devices 102 and retrieve object information from the object database 126. The deep learning module may combine the object information with corresponding map data that includes objects from the object information. The deep learning module 128 may do so using one or more machine learning models trained on the server. The machine learning models may include classifiers, neural networks, regression models, etc. The deep learning module 128 may store the fused information in the map database 124 or another database on the server.

[0036] 2 is a block diagram of one world mapping module 120 according to one embodiment. One world mapping module 120 includes map module 210, graph module 215, combination module 220, image database 230, and map database 240. In additional or alternative embodiments, one world mapping module 120 may include other modules that perform additional operations not described below.

[0037] The map module 210 determines a map of the environment based on data captured by the client device 102. Such data may include image data, sensor data, GPS data, etc. The map module 210 may construct a point cloud based on the captured data, which is used as a map of the environment. In some embodiments, the map module 210 may use other techniques to determine a map of the environment based on data captured by the client device 102. However, in other embodiments, the mapping is performed by the SLAM module 112 rather than the mapping module 210, and the mapping module 210 instead obtains a local map generated by the client device 102 from the SLAM module 112. In some embodiments, one or more of the local maps may be constructed collaboratively using data captured by various client devices 102 in the same environment. The map module 210 may store the local map in the map database 240. The map module 210 transmits the local map to the graph module 215.

[0038] The graph module 215 determines a graphical representation of one or more local maps. The graph module 215 receives the local maps from the map module 210. The graph module 215 may also receive information describing each local map. Such information may include which client device generated the local map and / or captured the data used to generate the local map, when the data was captured (e.g., date and time), etc.

[0039] For each local map, the graph module 215 creates a node representing the local map. In some embodiments, each client device 102 and / or server is also represented by a node created by the graph module 215. Each node has its own independent coordinate system based on information describing the local map, client device, or server it represents. Nodes representing local maps of an environment may further represent not only the spatial extent of the environment but also the temporal extent (e.g., how the environment changes over time). The graph module sends the nodes to the combination module 220 for assembly into a single 3D map as described above. In another embodiment, maps for different times (e.g., different periods of the day, such as morning, noon, evening, and night) are stored in different nodes, and the edges between them indicate mapping in both the spatial and temporal coordinates of the map.

[0040] The combination module 220 uses feature analysis to convert the local maps into a single 3D map of the real world. In some embodiments, the combination module 220 may combine the local maps into one single 3D map. In other embodiments, the combination module creates a 3D map of each environment using the local maps and concatenates the 3D maps in a single 3D map.

[0041] The combination module 220 receives nodes from the graph module 215 that represent one or more local maps. For each pair of nodes, the combination module 220 may determine an edge. An edge represents a transformation between the coordinate spaces of the nodes. In some cases, a pair of nodes may not have an edge between them (e.g., if the nodes represent completely different environments). Otherwise, a pair of nodes may have one or more edges associated with them. In some embodiments, the combination module 220 may determine edges only for nodes in the same environment, which the combination module may determine based on feature matching between the local maps. In one embodiment, the mapping module 210 may identify two local maps as representing a single environment based on the local maps being within a threshold distance from each other, which the combination module 220 may determine from the GPS data used to generate each local map.

[0042] The combination module 220 may form edges based on data captured by various client devices 102. Each client device may have a confidence score associated with it, which may be used to determine the confidence score of an edge. The confidence score of an edge represents the likelihood that using the transformation represented by the edge to travel from a first node to a second node will result in an output node identical to the second node. To determine edges, the combination module may use tracking information (e.g., nodes in local maps captured by the same client device during the same session of a parallel reality game are likely to have an edge), feature-based localization (e.g., locating two local maps of a node based on features contained in the local maps, such as points, lines, etc.), 3D cloud alignment (e.g., via an ICP algorithm), forced overlap between consecutive local maps generated by the same client device 102, post-processing optimization across multiple local maps, and / or machine-readable code-based localization (e.g., synchronization).

[0043] For example, in one embodiment, combination module 220 may perform feature analysis to determine an edge for two nodes. Combination module 220 obtains information from object database 126 for each of two local maps and uses that information to perform feature analysis on each local map to determine whether the local maps both contain common features. If combination module 220 determines that each map contains the same common features, combination module 220 creates an edge based on the common features.

[0044] In another example, the combination module 220 may determine edges between nodes based on synchronization performed by the client devices 102. The combination module 220 obtains synchronization data for each local map that indicates that the client devices 102 are co-located in the same environment. The synchronization data may be determined when the client devices 102 are pointed at each other or when each client device 102 captures an image of a machine-readable code (e.g., a two-dimensional barcode) or other recognizable feature in the environment. Based on the synchronization data, the combination module 220 determines edges for the nodes of the local map.

[0045] For each pair of nodes, the combination module 220 accesses a single 3D map of the real world from the map database 240. The single 3D map includes multiple nodes captured by various client devices 102 connected to the server and represents the layout of the real world. If one or both of the nodes are not present in the single 3D map, the combination module 220 adds the missing node or nodes to the single 3D map. Furthermore, if the combination module 220 determines an edge for the pair of nodes, the combination module 220 connects the edges together in the single 3D map, essentially connecting the local maps into one larger map (e.g., a single 3D map). In some embodiments, the combination module 220 may further stitch the local maps together based on the edges to form a single map that includes at least portions of both local maps.

[0046] The combination module 220 may also add edges between existing nodes within the single 3D map. In some embodiments, the combination module 220 may combine multiple edges between pairs of nodes into a single edge as new edges are determined. In other embodiments, the combination module 220 may retain all edges between pairs of nodes within the single 3D map and indicate which of all the edges is the most recent so that the client device may use the most recent edge to convert between local maps as needed.

[0047] Client devices 102 connected to the server may use the single 3D map to localize themselves within the environment and obtain information about the virtual world at the location for a parallel reality game. Additionally, the system of nodes and edges may be used to reduce drift and outliers in the single 3D map. For example, the combination module 220 may remove nodes that are not connected to other nodes by edges after the nodes have been in the single 3D map for a threshold amount of time.

[0048] (Example Data Flow) 3 is a flowchart illustrating processing performed by a client device 102 and a server to generate and display AR data, according to one embodiment. The client device 102 and server may be similar to those shown in FIG. 1. Dashed lines represent communication of data between the client device 102 and the server, while solid lines represent communication of data within a single device (e.g., within the client device 102 or within the server). In other embodiments, functionality may be distributed differently between devices and / or different devices may be used.

[0049] At 302, raw data is collected at the client device 102 by one or more sensors. In one embodiment, the raw data includes image data, inertial measurement data, and location data. The image data may be captured by one or more cameras physically or wirelessly coupled to the client device 102. The inertial measurement data may be collected using a gyroscope, an accelerometer, or a combination thereof, and may include inertial measurement data with up to six degrees of freedom—i.e., three degrees of translational motion and three degrees of rotational motion. Location data may be collected with a Global Positioning System (GPS) receiver. Additional raw data may be collected by various other sensors, such as pressure levels, light levels, humidity levels, altitude levels, sound levels, audio data, etc. The raw data may be stored in one or more storage modules within the client device 102 that can record raw data acquired by the various sensors of the client device 102 over time.

[0050] The client device 102 may maintain a local map storage at 304. The local map storage includes local point cloud data. The point cloud data includes locations in space that form a constructible mesh surface. The local map storage at 304 may include a hierarchical cache of the local point cloud data so that it can be easily retrieved for use by the client device 102. The local map storage at 304 may additionally include object information fused with the local point cloud data. The object information may specify various objects within the local point cloud data.

[0051] Once the raw data is collected at 302, the client device 102 checks whether a map has been initialized at 306. If the map has been initialized at 306, the client device 102 may begin a SLAM function at 308. The SLAM function includes a mapping system that builds a point cloud and tracking to find the camera's position in space over the initialized map. An exemplary SLAM process further reprojects animated or augmented values ​​to the real world. If the map has not been initialized at 310, the client device 102 may search its local map storage at 304 to find a locally stored map. If the map is found in the local map storage at 304, the client device 102 may retrieve the map for use by the SLAM function. If the map is not found at 310, the client device 102 may use an initialization module to create a new map at 312.

[0052] When a new map is created, the initialization module may store the newly created map in local map storage at 304. The client device 102 may routinely synchronize the map data in the local map storage 304 with the server-side cloud map storage 320. When synchronizing the map data, the local map storage 304 of the client device 102 may send the newly created map to the server. At 326, the server side checks whether the map received from the client device 102 was previously stored in the cloud map storage 320. If not, the server side generates a new map for storage in the cloud map storage 320 at 328. The server may alternatively append the new map at 328 to an existing map in the cloud map storage 320.

[0053] Returning to the client side, the client device 102 determines at 314 whether a new viewpoint has been detected. In some embodiments, the client device 102 determines whether each viewpoint in the stream of captured images has less than a threshold overlap with existing viewpoints stored on the client device 102 (e.g., the local map storage 304 may store viewpoints obtained by the client device 102 or viewpoints obtained from the cloud map storage 320). In other embodiments, the client device 102 determines at 314 whether a new viewpoint has been detected in a multi-stage determination. At a high level, the client device 102 may obtain any existing viewpoints within a local radius of the client device's 102's geolocation. From the existing viewpoints, the client device 102 may begin to identify similar objects or features in the viewpoint in question compared to the existing viewpoints. For example, the client device 102 may identify a tree in the viewpoint and further reduce all existing viewpoints within the local radius that also have trees visible. The client device 102 may use additional layers of filtering to be more robust when matching the viewpoint in question to a filtered set of existing viewpoints. In one example, the client device 102 uses a machine learning model to determine whether the viewpoint in question matches another viewpoint in the filtered set (i.e., the viewpoint in question is not new because it matches an existing viewpoint). If a new viewpoint is detected at 314, the client device 102 records data collected by the local environment estimation at 316. For example, if the client device 102 determines that it currently has a new viewpoint, it may send an image captured at the new viewpoint to the server (e.g., to a server-side map / image database 318). The new viewpoint detection module may be used to determine when and how to send the image with 3D data. The local environment estimation may include updated keyframes of the local mapping system and serialized image and / or map data.The local environment estimation can be used by the server to adapt the new viewpoint relative to other viewpoints at a given location in the map.

[0054] On the server side, the new viewpoint data (e.g., consisting of point cloud information overlaid with mesh data) may be stored in a server-side map / image database 318. The server may add different portions of the real-world map from stored cloud map storage 320 and object database 322. The cloud environment estimation 324 (including the added component data) may be sent back to the client device. The added data may include points and meshes stored in local map storage 304, as well as object data with semantic labels (e.g., walls or beds).

[0055] (Conceptual diagram of the virtual world) 4 illustrates a conceptual diagram of a virtual world 410 parallel to a real world 400 that can serve as a game board for players of a location-based parallel reality game, according to one embodiment. The client device 102 of FIG. 1 may host a parallel reality game (or other location-based game) having a virtual world 410 that corresponds to the real world 400, as shown in FIG. 4.

[0056] As shown, virtual world 410 may include a geography that parallels the geography of real world 400. In particular, a range of coordinates that defines a geographic area or space in real world 400 is mapped to a corresponding range of coordinates that defines a virtual space in virtual world 410. A range of coordinates in real world 400 may be associated with a town, neighborhood, city, campus, locale, country, continent, the entire Earth, or other geographic area. Each geographic coordinate within a range of geographic coordinates is mapped to a corresponding coordinate in a virtual space in the virtual world.

[0057] The position of a player in virtual world 410 corresponds to the position of the player in real world 400. For example, player A, located at location 412 in real world 400, has a corresponding location 422 in virtual world 410. Similarly, player B, located at real world location 414, corresponds to virtual world location 424. As a player moves within a range of geographic coordinates in real world 400, the player also moves within a range of coordinates that define a virtual space in virtual world 410. In particular, a positioning system (e.g., a GPS system or other system used by location estimation and mapping module 112) associated with a client device 102 carried by the player can be used to track the player's position as the player moves within a range of geographic coordinates in the real world. Data associated with the player's position in real world 400 is used to update the player's position in the corresponding range of coordinates that define a virtual space in virtual world 410. In this way, a player can navigate a continuous track within a range of coordinates defining a virtual space in virtual world 410 simply by moving between corresponding ranges of geographic coordinates in real world 400, without having to check in at specific discrete locations in real world 400 or periodically update their location information.

[0058] A parallel reality game may include multiple game objects that require a player to travel to and / or interact with various virtual elements and / or objects scattered throughout various virtual locations in the virtual world 410. The player may travel to these virtual locations by moving to the corresponding locations of the virtual elements or objects in the real world 400. For example, the positioning system of the client device 102 may continuously track the player's location such that as the player continuously navigates the real world 400, the player also continuously navigates the parallel virtual world 410. The player may then interact with the various virtual elements and / or objects at a particular location to accomplish or perform one or more game objects.

[0059] For example, referring to FIG. 4 , a game object may require a player to capture or claim ownership of virtual elements 430 located at various virtual locations in the virtual world 410. These virtual elements 430 may be linked to landmarks, geographic locations, or objects 440 in the real world 400. The real-world landmarks or objects 440 may be works of art, monuments, buildings, businesses, libraries, museums, or other suitable real-world landmarks or objects. To capture these virtual elements 430, the player must travel to the landmark, geographic location, or object 440 linked to the real-world virtual element 430 and perform the necessary interaction with the virtual element 430 in the virtual world 410. For example, Player A in FIG. 4 must travel, via the client device 102, to the landmark 440 in the real world 400 in order to interact with or capture the virtual element 430 linked to that particular landmark 440. Interaction with the virtual element 430 may require actions in the real world 400, such as taking a photograph and / or verifying, obtaining, or capturing other information about a landmark or object 440 associated with the virtual element 430.

[0060] Game objectives may require the use of one or more virtual items that a player collects in the parallel reality game. For example, a player may have to travel through the virtual world 410 in search of virtual items (e.g., weapons or other items) useful to complete a game objective. These virtual items can be found or collected by traveling to different locations in the real world 400 or by completing various actions in either the virtual world 410 or the real world 400. In the example shown in FIG. 4 , a player uses a virtual item 432 to capture one or more virtual elements 430. In particular, a player can deploy the virtual item 432 at a location in the virtual world 410 in proximity to the virtual element 430. Deploying one or more virtual items 432 in proximity to the virtual element 430 may result in the capture of the virtual element 430 for a particular player or for a particular player's team and / or faction.

[0061] In one particular implementation, a player may have to collect virtual energy as part of a parallel reality game. As depicted in FIG. 4 , virtual energy 450 may be scattered at different locations in virtual world 410. A player may collect virtual energy 450 by traveling to the corresponding location of virtual energy 450 in real world 400. Virtual energy 450 may be used to power virtual items and / or to perform various game objectives in the parallel reality game. A player who loses all of their virtual energy 450 may be disconnected from the parallel reality game.

[0062] According to aspects of the present disclosure, a parallel reality game can be a massively multiplayer location-based game in which all participants in the parallel reality game share the same virtual world. Players are divided into separate teams or factions and can cooperate to achieve one or more game objectives, such as capturing or claiming ownership of virtual elements 430. In this manner, a parallel reality game can be essentially a social game that encourages cooperation between players within the parallel reality game. Players on opposing teams can cooperate with each other during the parallel reality game. Players can use virtual items 432 to attack or impede the progress of players on opposing teams.

[0063] A parallel reality game may have a variety of features to enhance and encourage gameplay within the parallel reality game. For example, players may accumulate virtual currency or other virtual rewards that can be used throughout the parallel reality game. Players may progress through various levels by completing one or more game objectives and gaining experience within the parallel reality game. Players may communicate with each other through one or more communication interfaces provided within the parallel reality game. Players may also gain enhanced "powers" or virtual items 432 that can be used to complete game objectives within the parallel reality game. Using the disclosure provided herein, one skilled in the art will understand that various other game features can be included in a parallel reality game without departing from the scope of the present disclosure.

[0064] (Exemplary Method) 5 is a flowchart illustrating a process 500 for combining a first 3D map and a second 3D map into a single 3D map of an environment, according to one embodiment. In some embodiments, the process 500 may be modified to be performed on the client side instead of the server side. In this embodiment, the server receives 510 a first set of image data captured by a camera of the first client device 102. The image data represents a near-real-time view of a first area around the first client device 102 in the environment. The server generates 520 a first 3D map based on the first set of image data and, optionally, location data captured by the first client device 102. The 3D map spatially describes the first area around the first client device 102.

[0065] The server receives a second set of image data captured from a second client device 102 within the environment. The second set of image data describes a second area surrounding the second client device 102, and the server generates 530 a second 3D map based on the second set of image data. The server analyzes 540 the first 3D map and the second 3D map for common features located in both 3D maps. In response to the server finding common features in the first 3D map and the second 3D map, the server concatenates 550 the first 3D map and the second 3D map into a single 3D map that describes the environment. In another embodiment, the client device 102 generates and sends the first and second 3D maps to the server, and the server decides whether and how to concatenate them.

[0066] In some embodiments, the first and second 3D maps may be associated in a graph of nodes. In particular, the first and second 3D maps may each be represented by nodes connected by edges in the graph. Each node is associated with a different coordinate space representing the client device 102 that captured the image data used to generate the 3D map or the time the image data was captured by the respective client device. The edge includes a transformation between the different coordinate spaces of the connected nodes. The server may determine the edges based on analysis 540, which may include one or more of session information, point feature-based location estimation, line feature-based location estimation, 3D cloud alignment, forced overlap, optimization, or two-dimensional barcode-based location estimation.

[0067] 6 is a flowchart illustrating a process 600 for generating a single 3D map of an environment based on synchronization, according to one embodiment. In some embodiments, the process 600 may be modified to be performed on the client side. In this embodiment, a server receives image data 610 captured by a camera of a first client device 102. The image data represents a near-real-time view of a first area around the first client device in the environment. The server synchronizes 620 positions between the first client device 102 and a second client device 102. In some embodiments, the server synchronizes positions by receiving image data from each client device 102 or another client device of a feature, such as a two-dimensional barcode.

[0068] The server generates 630 a first 3D map from the first client device based on the image data. Alternatively, the first 3D map may be generated by the client device 102 rather than the image data and transmitted to the server. The first 3D map spatially describes a first area around the first client device 102. The first 3D map may be a raw image or a point cloud generated by the first client device 102. The server receives image data captured from a second client device 102 in the environment. The image data describes a second area around the second client device 102, and the server generates 640 a second 3D map from the second client device based on the image data. The server generates 650 a single 3D map from the first and second 3D maps based on the synchronization. Because the device positions are synchronized in the environment, the relative positions of features in the first and second 3D maps can be determined even if the maps do not overlap.

[0069] (Computing Machine Architecture) 7 is a high-level block diagram illustrating an exemplary computer 700 suitable for use as a client device 102 or a server. The exemplary computer 700 includes at least one processor 702 coupled to a chipset 704. The chipset 704 includes a memory controller hub 720 and an input / output (I / O) controller hub 722. A memory 706 and a graphics adapter 712 are coupled to the memory controller hub 720, and a display 718 is coupled to the graphics adapter 712. A storage device 708, a keyboard 710, a pointing device 714, and a network adapter 716 are coupled to the I / O controller hub 722. Other embodiments of the computer 700 have different architectures.

[0070] 7, storage device 708 is a non-transitory computer-readable storage medium such as a hard disk, compact disc read-only memory (CD-ROM), DVD, or solid-state memory device. Memory 706 holds instructions and data used by processor 702. Pointing device 714 is a mouse, trackball, touchscreen, or other type of pointing device and is used in combination with keyboard 710 (which may be an on-screen keyboard) to input data into computer system 700. In other embodiments, computer 700 has various other input mechanisms, such as a touchscreen, joystick, buttons, scroll wheel, etc., or any combination thereof. Graphics adapter 712 displays images and other information on display 718. Network adapter 716 couples computer system 700 to one or more computer networks (e.g., network adapter 716 may couple client device 102 to a server via network 104).

[0071] 1 can vary depending on the embodiment and the processing power required by the entities. For example, a server may include a distributed database system including multiple blade servers working together to provide the described functionality. Additionally, a computer may lack some of the components described above, such as keyboard 710, graphics adapter 712, and display 718.

[0072] Those skilled in the art will be able to make numerous uses and modifications of, and deviations from, the devices and techniques disclosed herein without departing from the concepts described. For example, the components or features illustrated or described in this disclosure are not limited to the location, configuration, or context shown or described. Example devices in accordance with this disclosure may include all, fewer, or different components than those described with reference to the preceding figure or figures. Therefore, the present disclosure is not limited to the specific embodiments described herein, but rather is to be accorded the widest possible scope consistent with the appended claims and their equivalents.

Claims

1. 1. A computer-implemented method for combining map data to generate a three-dimensional (3D) map of an environment, comprising: Synchronizing locations between a first client device and a second client device; determining a first 3D map based on a first set of image data, the first set of image data being generated by the first client device; determining a second 3D map based on a second set of image data, the second set of image data being generated by the second client device; and generating a single 3D map from the first 3D map and the second 3D map based on the synchronized positions between the first client device and the second client device; Equipped with generating the single 3D map is further based on a common feature of the first set of image data and the second set of image data, the common feature being a QR code or a third client device. Computer-implemented methods.

2. A computer-implemented method for combining map data to generate a three-dimensional (3D) map of an environment, comprising: Synchronizing locations between a first client device and a second client device; determining a first 3D map based on a first set of image data, the first set of image data being generated by the first client device; determining a second 3D map based on a second set of image data, the second set of image data being generated by the second client device; and generating a single 3D map from the first 3D map and the second 3D map based on the synchronized positions between the first client device and the second client device; Equipped with the first 3D map and the second 3D map do not overlap; Computer-implemented methods.

3. A computer-implemented method for combining map data to generate a three-dimensional (3D) map of an environment, comprising: Synchronizing locations between a first client device and a second client device; determining a first 3D map based on a first set of image data, the first set of image data being generated by the first client device; determining a second 3D map based on a second set of image data, the second set of image data being generated by the second client device; and generating a single 3D map from the first 3D map and the second 3D map based on the synchronized positions between the first client device and the second client device; Equipped with the first 3D map is associated with a first node of a graph; the second 3D map is associated with a second node of the graph; the first node and the second node are connected by an edge determined based on the synchronized positions between the first client device and the second client device. Computer-implemented methods.

4. 4. The computer-implemented method of claim 1, wherein generating the single 3D map comprises stitching the first 3D map and the second 3D map together based on the synchronized positions between the first client device and the second client device.

5. prior to generating the single 3D map from the first 3D map and the second 3D map; Identifying a first feature in the first 3D map; Identifying a second feature in the second 3D map; and determining a relative position of the first feature with respect to the second feature based on the synchronized positions between the first client device and the second client device; The computer-implemented method of claim 1 , 2 or 3 further comprising:

6. The computer-implemented method of claim 5 , wherein generating the single 3D map is further based on the relative position of the first feature with respect to the second feature.

7. The computer-implemented method of claim 5 , wherein the first feature is a feature common to the first 3D map and the second 3D map.

8. prior to generating the single 3D map from the first 3D map and the second 3D map; receiving the first set of image data and the second set of image data; generating the first 3D map based on the first set of image data; generating the second 3D map based on the second set of image data; The computer-implemented method of claim 1 , 2 or 3 further comprising:

9. A non-transitory computer-readable storage medium that, when executed by a computing device, Synchronizing locations between a first client device and a second client device; determining a first 3D map based on a first set of image data, the first set of image data being generated by the first client device; determining a second 3D map based on a second set of image data, the second set of image data being generated by the second client device; and generating a single 3D map from the first 3D map and the second 3D map based on the synchronized positions between the first client device and the second client device; instructions that cause the computing device to perform operations including: generating the single 3D map is further based on a common feature of the first set of image data and the second set of image data, the common feature being a QR code or a third client device. A non-transitory computer-readable storage medium.

10. A non-transitory computer-readable storage medium that, when executed by a computing device, Synchronizing locations between a first client device and a second client device; determining a first 3D map based on a first set of image data, the first set of image data being generated by the first client device; determining a second 3D map based on a second set of image data, the second set of image data being generated by the second client device; and generating a single 3D map from the first 3D map and the second 3D map based on the synchronized positions between the first client device and the second client device; instructions that cause the computing device to perform operations including: the first 3D map and the second 3D map do not overlap; A non-transitory computer-readable storage medium.

11. A non-transitory computer-readable storage medium that, when executed by a computing device, Synchronizing locations between a first client device and a second client device; determining a first 3D map based on a first set of image data, the first set of image data being generated by the first client device; determining a second 3D map based on a second set of image data, the second set of image data being generated by the second client device; and generating a single 3D map from the first 3D map and the second 3D map based on the synchronized positions between the first client device and the second client device; instructions that cause the computing device to perform operations including: the first 3D map is associated with a first node of a graph; the second 3D map is associated with a second node of the graph; the first node and the second node are connected by an edge determined based on the synchronized positions between the first client device and the second client device. A non-transitory computer-readable storage medium.

12. 12. The non-transitory computer-readable storage medium of claim 9, wherein generating the single 3D map comprises stitching the first 3D map and the second 3D map together based on the synchronized positions between the first client device and the second client device.

13. The operation is prior to generating the single 3D map from the first 3D map and the second 3D map; Identifying a first feature in the first 3D map; Identifying a second feature in the second 3D map; and determining a relative position of the first feature with respect to the second feature based on the synchronized positions between the first client device and the second client device; 12. The non-transitory computer-readable storage medium of claim 9, 10, or 11, further comprising:

14. The non-transitory computer-readable storage medium of claim 13 , wherein generating the single 3D map is further based on the relative position of the first feature with respect to the second feature.

15. 14. The non-transitory computer-readable storage medium of claim 13, wherein the first feature is a feature common to the first 3D map and the second 3D map.

16. 1. A computer system comprising: a computer processor system; A computer-readable storage medium that, when executed by the computer processor system, Synchronizing locations between a first client device and a second client device; determining a first 3D map based on a first set of image data, the first set of image data being generated by the first client device; determining a second 3D map based on a second set of image data, the second set of image data being generated by the second client device; and generating a single 3D map from the first 3D map and the second 3D map based on the synchronized positions between the first client device and the second client device; the computer-readable storage medium storing instructions for performing operations comprising: Equipped with generating the single 3D map is further based on a common feature of the first set of image data and the second set of image data, the common feature being a QR code or a third client device. Computer system.

17. A computer system comprising: a computer processor system; A computer-readable storage medium that, when executed by the computer processor system, Synchronizing locations between a first client device and a second client device; determining a first 3D map based on a first set of image data, the first set of image data being generated by the first client device; determining a second 3D map based on a second set of image data, the second set of image data being generated by the second client device; and generating a single 3D map from the first 3D map and the second 3D map based on the synchronized positions between the first client device and the second client device; the computer-readable storage medium storing instructions for performing operations comprising: Equipped with the first 3D map and the second 3D map do not overlap; Computer system.

18. A computer system comprising: a computer processor system; A computer-readable storage medium that, when executed by the computer processor system, Synchronizing locations between a first client device and a second client device; determining a first 3D map based on a first set of image data, the first set of image data being generated by the first client device; determining a second 3D map based on a second set of image data, the second set of image data being generated by the second client device; and generating a single 3D map from the first 3D map and the second 3D map based on the synchronized positions between the first client device and the second client device; the computer-readable storage medium storing instructions for performing operations comprising: Equipped with the first 3D map is associated with a first node of a graph; the second 3D map is associated with a second node of the graph; the first node and the second node are connected by an edge determined based on the synchronized positions between the first client device and the second client device. Computer system.

Images