Fast real-time scene reconstruction from input image data

Abstract

A scene reconstruction model is disclosed that outputs a height field for a sequence of input images. For each input image, the model predicts a depth map and extracts a feature map. The model utilizes the predicted depth map and camera pose of the images to build a 3D model. The model raycasts the 3D model to determine a raw height field of the scene. The model utilizes the raw height field to sample features from the feature map that correspond to positions on the height field. The model aggregates the sampled features into an aggregate feature map. The model regresses a refined height field based on the aggregate feature map. The model determines a final height field based on a combination of the raw height field and the refined height field. Using the final height field, a client device may generate augmented virtual content on real-world images captured by the client device.

Description

[Technical field]

[0001] 1.Technical Field The described subject matter generally relates to three-dimensional (3D) scene reconstruction from input image data. [Background technology]

[0002] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of and priority to U.S. Provisional Application No. 63 / 290,440, filed December 16, 2021, which is incorporated by reference in its entirety.

[0003] 2. Challenges 3D scene reconstruction from input image data is a fundamental task in computer vision. It can be particularly useful for augmented reality applications and / or agent navigation. Recent methods can achieve high-quality reconstruction but at the expense of increased runtime and memory requirements. This can introduce delays in the placement and / or movement of virtual characters, impairing the perception of augmented reality. Similarly, in agent navigation, delayed scene reconstruction can cause delays in navigation guidance decisions. Summary of the Invention

[0004] This disclosure describes an approach to a scene reconstruction model that provides fast, high-quality scene reconstruction in real time. The scene reconstruction model inputs image data, which may include real-time video captured by a user device. The scene reconstruction model builds a fast, real-time 3D representation of the environment. The user device may utilize the 3D representation for virtual character placement, agent navigation, other computer vision tasks, and the like. The scene reconstruction model can reconstruct occluded regions.

[0005] In one or more embodiments, the scene reconstruction model generally hybridizes the generation of a raw height field from a predicted depth map and the regression of a refined height field from a feature map of an image. For each image, the scene reconstruction model predicts a depth map and a feature map based on the input image and the camera pose. The depth map includes depth values ​​for each pixel of the input image. The feature map includes features of the input image. The features may include abstract features (i.e., convolutional features), objects of interest, corners, edges, object contours, horizons, semantic segmentation, other features of the input image, or some combination thereof. To construct the raw height field, the scene reconstruction model builds a 3D model of the environment and then raycasts the 3D voxel array to generate the raw height field. To regress the refined height field, the scene reconstruction model projects the raw height field onto the camera pose of each image and samples features from the feature map based on the visible surface of the projected raw height field. The scene reconstruction model aggregates features sampled from the images and inputs the aggregated feature map to regress a refined height field. Generally speaking, the raw height field has sharp definition but may be incomplete in certain parts of the environment that have low signal from the image. However, the refined height field regresses and fills in the low signal parts but may have less definition. Therefore, the scene reconstruction model creates a final height field by blending the raw height field and the refined height field, thereby leveraging the advantages of each. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 illustrates a networked computing environment in accordance with one or more embodiments. [Diagram 2]FIG. 1 illustrates a representation of a virtual world having a parallel geography to the real world, according to one or more embodiments. [Diagram 3] FIG. 1 illustrates an exemplary game interface for a parallel reality game, according to one or more embodiments. [Figure 4] FIG. 4 illustrates an example architecture of a scene reconstruction model 400 according to one or more embodiments. [Diagram 5] 1 is a flowchart illustrating a method for generating a height field using a scene reconstruction model in accordance with one or more embodiments. [Figure 6] 1 is a flowchart illustrating a method of applying a height field generated by a scene reconstruction model in accordance with one or more embodiments. [Figure 7] FIG. 1 illustrates an example computer system suitable for use in training or applying a depth estimation model, according to one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] The drawings and the following description set forth specific embodiments for purposes of illustration only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structure and method may be employed without departing from the principles described. Reference will now be made to certain embodiments, examples of which are illustrated in the accompanying drawings.

[0008] (An exemplary location-based parallel reality gaming system) Various embodiments are described in the context of a parallel reality game that includes augmented reality content in a virtual world geography that parallels at least a portion of the real world geography, such that player movements and actions in the real world affect actions in the virtual world and vice versa. Using the disclosure provided herein, one of ordinary skill in the art will appreciate that the subject matter described is applicable to other situations in which scene reconstruction from image data is beneficial. In addition, the inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions between components of the system. For example, systems and methods according to aspects of the present disclosure can be implemented using a single computing device or across multiple computing devices (e.g., connected by a computer network).

[0009] FIG. 1 illustrates a networked computing environment 100, according to one or more embodiments. The networked computing environment 100 provides for interaction with a player in a virtual world having a geography that parallels the real world. In particular, geographical regions of the real world may be directly linked or mapped to corresponding regions in the virtual world. A player may navigate the virtual world by moving to various geographical locations in the real world. For example, the player's location in the real world may be tracked and used to update the player's location in the virtual world. Typically, the player's location in the real world is determined by finding the location of the client device 120 at which the player is interacting with the virtual world and assuming that the player is in the same (or approximately the same) location. For example, in various embodiments, a player may interact with a virtual element if the player's location in the real world is within a threshold distance (e.g., 10 meters, 20 meters, etc.) of a real-world location that corresponds to the virtual location of the virtual element in the virtual world. For convenience, various embodiments are described with reference to the "location of the player," but one skilled in the art will understand that such reference may also refer to the location of the player's client device 120.

[0010] Reference is now made to FIG. 2, which illustrates a conceptual diagram of a virtual world 210 parallel to a real world 200 that may serve as a game board for players of a parallel reality game, according to one embodiment. As illustrated, the virtual world 210 may include a geography that parallels the geography of the real world 200. In particular, a range of coordinates that defines a geographical area or space in the real world 200 is mapped to a corresponding range of coordinates that defines a virtual space in the virtual world 210. A range of coordinates in the real world 200 may be associated with a town, neighborhood, city, campus, neighborhood, country, continent, the entire globe, or other geographical area. Each geographic coordinate within the range of geographic coordinates is mapped to a corresponding coordinate in a virtual space in the virtual world.

[0011] The position of the player in the virtual world 210 corresponds to the position of the player in the real world 200. For example, player A, located at location 212 in the real world 200, has a corresponding location 222 in the virtual world 210. Similarly, player B, located at location 214 in the real world, has a corresponding location 224 in the virtual world. As the player moves within a range of geographic coordinates in the real world, the player also moves within a range of coordinates that define a virtual space in the virtual world 210. In particular, a positioning system (e.g., a GPS system) associated with a mobile computing device carried by the player may be used to track the player's position as the player navigates the range of geographic coordinates in the real world. Data associated with the player's position in the real world 200 is used to update the player's position in the corresponding range of coordinates that define a virtual space in the virtual world 210. In this manner, a player may navigate along a continuous track within a range of coordinates defining a virtual space in the virtual world 210 by simply moving within a corresponding range of geographic coordinates in the real world 200, without checking in at a specific individual location in the real world 200 or periodically updating location information.

[0012] A location-based game may include multiple game objectives that require the player to travel to or interact with various virtual elements and / or virtual objects scattered at various virtual locations within the virtual world. The player may travel to these virtual locations by moving to the corresponding locations of the virtual elements or objects in the real world. For example, a positioning system may continuously track the player's location such that as the player continually navigates the real world, the player also continually navigates the parallel virtual world. The player may then interact with the various virtual elements and objects at a particular location to accomplish or perform one or more game objectives.

[0013] For example, a game goal may have the player interact with virtual elements 230 located at various virtual locations in the virtual world 210. These virtual elements 230 may be linked to landmarks, geographic locations, or objects 240 in the real world 200. The real world landmarks or objects 240 may be works of art, monuments, buildings, businesses, libraries, museums, or other suitable real world landmarks or objects. The interactions may include capturing, claiming ownership, using some virtual items, using some virtual currency, etc. To capture these virtual elements 230, the player must travel to the landmark or geographic location 240 linked to the virtual elements 230 in the real world and perform any necessary interactions with the virtual elements 230 in the virtual world 210. For example, player A in FIG. 2 may have to travel to a landmark 240 in the real world 200 to interact with or capture the virtual element 230 linked to that particular landmark 240. Interaction with the virtual element 230 may require an action in the real world, such as taking a picture of a landmark or object 240 associated with the virtual element 230 and / or verifying, obtaining, or capturing other information about the landmark or object 240.

[0014] A game objective may require a player to use one or more virtual items that are collected by the player in a location-based game. For example, a player may travel through the virtual world 210 in search of virtual items (e.g., weapons, creatures, power-ups, or other items) that may be useful in achieving a game objective. These virtual items may be found or collected by traveling to different locations in the real world 200 or by completing various actions in either the virtual world 210 or the real world 200. In the example shown in FIG. 2, a player uses a virtual item 232 to capture one or more virtual elements 230. In particular, a player may place a virtual item 232 at a location in the virtual world 210 that is in proximity to or within the virtual element 230. Placing one or more virtual items 232 in this manner may result in the capture of the virtual element 230 for a particular player or a particular player's team / faction.

[0015] In one particular implementation, a player may have to collect virtual energy as part of a parallel reality game. As shown in FIG. 2, virtual energy 250 may be scattered at different locations in the virtual world 210. A player may collect virtual energy 250 by traveling to the corresponding location of the virtual energy 250 in the real world 200. The virtual energy 250 may be used to power virtual items or to perform various game objectives within the game. A player who loses all of the virtual energy 250 may be disconnected from the game.

[0016] According to aspects of the present disclosure, a parallel reality game may be a massively multiplayer location-based game in which all participants of the game share the same virtual world. Players may be divided into separate teams or factions and may cooperate to achieve one or more game objectives, such as capturing or claiming ownership of virtual elements. In this way, a parallel reality game may essentially be a social game that encourages cooperation between players in the game. Players on opposing teams may play against each other (or sometimes cooperate to achieve each other's objectives) during the parallel reality game. Players may use virtual items to attack or impede the progress of players on the opposing team. In some cases, players are encouraged to gather at real-world locations for cooperative or interactive events in the parallel reality game. In these cases, the game server attempts to verify that players are actually physically present and are not faking their locations.

[0017] A parallel reality game may have various features to enhance and encourage gameplay within the parallel reality game. For example, a player may accumulate virtual currency or another virtual reward (e.g., virtual tokens, virtual points, virtual material resources, etc.) that may be used throughout the game (e.g., to purchase in-game items, to redeem other items, to create items, etc.). Players may progress through various levels as they accomplish one or more game objectives and gain experience within the game. In some embodiments, players may communicate with each other through one or more communication interfaces provided within the game. Players may also gain enhanced "powers" or virtual items that can be used to complete game objectives within the game. In some embodiments, players may turn on the camera of their mobile device to provide an augmented reality experience, where real-time image data is displayed augmented with generated virtual content. The generation of the virtual content may depend in part on a 3D representation of the scene (e.g., as determined by a scene reconstruction model). Using the disclosure provided herein, one of ordinary skill in the art should understand that various other game features may be included in a parallel reality game without departing from the scope of the present disclosure.

[0018] In one or more embodiments, the parallel reality game may incorporate an augmented reality experience. Augmented reality generally involves generating and displaying virtual content in a real-world environment. In one or more examples, the client device 110 may be a glasses device or another type of headset with one or more lenses capable of at least partially transmitting light from the environment and displaying the virtual content in conjunction with the transmitted light from the real world. In another example, the client device 110 includes an electronic display that presents a live feed of a camera assembly capturing a real-world environment. The client device 110 may generate virtual content that is overlaid on the live feed. In either case, the effect is similar. There is virtual content presented in conjunction with real-world content. As an example, a virtual character may be generated and displayed in conjunction with a live feed from a camera assembly. The virtual character may be generated to interact realistically with the environment.

[0019] Returning to reference to FIG. 1 , the networked computing environment 100 uses a client-server architecture, with the game server 120 communicating with the client devices 110 over the network 105 to provide the parallel reality game to the players at the client devices 110. The networked computing environment 100 may also include other external systems, such as sponsor / advertiser systems or business systems. Although only one client device 110 is shown in FIG. 1 , any number of clients 110 or other external systems may be connected to the game server 120 over the network 105. Additionally, the networked computing environment 100 may include different or additional elements, and functionality may be distributed between the client devices 110 and the server 120 in a manner different than that described below.

[0020] The client device 110 may be any portable computing device that can be used by a player to interface with the game server 120. For example, the client device 110 may be a wireless device, a personal digital assistant (PDA), a portable gaming device, a mobile phone, a smartphone, a tablet, a navigation system, a handheld GPS system, a wearable computing device, a display having one or more processors, or other such devices. In another example, the client device 110 includes a conventional computer system such as a desktop or laptop computer. Furthermore, the client device 110 may be a vehicle having a computing device. In short, the client device 110 may be any computer device or system that allows a player to interact with the game server 120. As a computing device, the client device 110 may include one or more processors and one or more computer-readable storage media. The computer-readable storage media may store instructions for causing the processor to perform operations. The client device 110 is preferably a portable computing device that can be easily carried or otherwise transported with the player, such as a smartphone or tablet.

[0021] The client device 110 communicates with the game server 120 and provides the game server 120 with sensory data of the physical environment. The client device 110 includes a camera assembly 125 that captures two-dimensional image data of a scene in the physical environment in which the client device 110 resides. In the embodiment shown in FIG. 1, the client device 110 includes software components such as a gaming module 135 and a positioning module 140. The client device 110 also includes a scene reconstruction module 145. The client device 110 may include various other input / output devices for receiving information from a player and / or providing information to a player. Exemplary input / output devices include a display screen, a touch screen, a touch pad, data entry keys, a speaker, and a microphone suitable for voice recognition. The client device 110 may also include various other sensors for recording data from the client device 110, including, but not limited to, motion sensors, accelerometers, gyroscopes, other inertial measurement units (IMUs), barometers, positioning systems, thermometers, light sensors, and the like. The client device 110 may further include a network interface for providing communications over the network 105. The network interface may include any suitable components for interfacing with one or more networks, including, for example, a transmitter, a receiver, a port, a controller, an antenna, or other suitable components.

[0022] The camera assembly 125 captures image data of a scene of an environment in which the client device 110 is present. The camera assembly 125 may utilize various photosensors having various color capture ranges at various capture rates. The camera assembly 125 may include a wide-angle lens or a telephoto lens. The camera assembly 125 may be configured to capture a single image or a video as image data. Additionally, the orientation of the camera assembly 125 may be parallel to the ground with the camera assembly 125 pointed toward the horizon. The camera assembly 125 captures the image data and shares the image data with a computing device on the client device 110. The image data may include metadata that describes other details of the image data, including sensor data (e.g., temperature, brightness of the environment) or capture data (e.g., exposure, warmth, shutter speed, focal length, capture time, etc.). The camera assembly 125 may include one or more cameras that may capture the image data. In one example, the camera assembly 125 includes one camera and is configured to capture monocular image data. In another example, camera assembly 125 includes two cameras and is configured to capture stereoscopic image data. In various other implementations, camera assembly 125 comprises multiple cameras, each configured to capture image data.

[0023] The gaming module 135 provides an interface for players to participate in the parallel reality game. The game server 120 transmits game data over the network 105 to the client device 110 for use by the gaming module 135 of the client device 110 and provides local versions of the game to players remote from the game server 120. The game server 120 may include a network interface for providing communication over the network 105. The network interface may include any suitable components for interfacing with one or more networks, including, for example, a transmitter, a receiver, a port, a controller, an antenna, or other suitable components.

[0024] The gaming module 135 executed by the client device 110 provides an interface between the player and the parallel reality game. The gaming module 135 may present a user interface on a display device associated with the client device 110 that displays a virtual world associated with the game (e.g., renders an image of the virtual world) and allows the user to interact with the virtual world to perform various game objectives. In some embodiments, the gaming module 135 presents real-world image data (e.g., captured by the camera assembly 125) augmented with virtual elements from the parallel reality game. In these embodiments, the gaming module 135 may generate virtual content and / or adjust the virtual content according to other information received from other components of the client device 110. For example, the gaming module 135 may adjust virtual objects displayed on the user interface according to the 3D representation created by the scene reconstruction module 145. Based on the 3D representation, the gaming module 135 may have virtual objects that interact with various surfaces, e.g., a virtual character is displayed hopping on a table.

[0025] The gaming module 135 may also control various other outputs to allow a player to interact with the game without the player needing to view a display screen. For example, the gaming module 135 may control various sounds, vibrations, or other notifications that allow a player to play a game without seeing a display screen. The gaming module 135 may access game data received from the game server 120 to provide an accurate representation of the game to the player. The gaming module 135 may receive and process player input and provide updates to the game server 120 over the network 105. The gaming module 135 may also generate and / or adjust game content displayed by the client device 110. For example, the gaming module 135 may generate virtual elements based on depth information.

[0026] The positioning module 140 can be any device or circuitry for monitoring the location of the client device 110. For example, the positioning module 140 can determine an actual or relative location by using a satellite navigation positioning system (e.g., GPS system, Galileo positioning system, Global Navigation Satellite System (GLONASS), Beidou satellite navigation and positioning system), an inertial navigation system, a dead reckoning system, based on an IP address, by using triangulation and / or proximity to cellular towers or Wi-Fi hotspots, and / or by using other suitable techniques to determine a location. The positioning module 140 may further include various other sensors that can help precisely locate the location of the client device 110.

[0027] As the player moves around with the client device 110 in the real world, the positioning module 140 tracks the player's location and provides the player location information to the gaming module 135. The gaming module 135 updates the player's location in the virtual world associated with the game based on the player's actual location in the real world. Thus, the player may interact with the virtual world simply by carrying or transporting the client device 110 in the real world. In particular, the player's location in the virtual world may correspond to the player's location in the real world. The gaming module 135 may provide the player location information to the game server 120 over the network 105. In response, the game server 120 may implement various techniques to verify the location of the client device 110 to prevent impostors from spoofing the location of the client device 110. It should be understood that the location information associated with the player is utilized only if permission is granted after the player is informed that the player's location information will be accessed and how the location information will be utilized in the context of the game (e.g., to update the player's location in the virtual world). Additionally, any location information associated with the player is stored and maintained in a manner that protects the player's privacy.

[0028] The scene reconstruction module 145 uses the trained scene reconstruction model to create a 3D representation of the scene from image data captured by the camera assembly 125. The scene reconstruction model is trained by the game server 120 or another suitable computing device and provided to the client device 110. In general, the trained scene reconstruction model inputs images from image data captured by the camera assembly 125. The scene reconstruction module 145 uses the trained scene reconstruction model to build and maintain a 3D representation of the scene as the image data is captured by the camera assembly 125. In one or more embodiments, the 3D representation includes a height field. The height field is a two-dimensional (2D) representation of the 3D representation of the scene, where each pixel of the height field corresponds to a height value of a real-world object at that pixel. The height field effectively acts as the terrain of the environment.

[0029] In one or more embodiments, the trained scene reconstruction model includes four stages of generating a 3D representation of the environment or scene. In the first stage, for each image, the trained scene reconstruction model predicts a depth map and a feature map. The depth map includes depth values ​​for each pixel of the input image. The feature map includes features of the input image. The features may include abstract features (i.e., convolutional features), objects of interest, corners, edges, object contours, horizons, semantic segmentation, other features of the input image, or some combination thereof. The feature map may be dimensionally smaller than the input image. In the second stage, the trained scene reconstruction model generates a 3D voxel array. In one or more embodiments, the voxel array is represented as a truncated signed distance field (TSDF) volume that is expanded and / or updated as additional images are captured by the camera assembly 125. The 3D voxel array can be used to generate a raw height field. In the third stage, the trained scene reconstruction model generates a top-down feature map that aggregates the feature maps of the input images. In a fourth stage, the scene reconstruction model creates a final height field based on the raw height field and the aggregated features. To do so, the scene reconstruction model may regress the refined height field based on the top-down feature map. In one or more embodiments, the refined height field is regressed via a deep learning neural network. The raw height field may inform the top-down aggregation of features and / or the regression of the refined height field. The trained scene reconstruction model may generate a final height field based on a combination of the raw height field and the refined height field. In one or more embodiments, the final height field blends the raw height field and the refined height field.

[0030] The game server 120 may be any computing device and may include one or more processors and one or more computer-readable storage media. The computer-readable storage media may store instructions that cause the processor to perform operations. The game server 120 may include or be in communication with a game database 115. The game database 115 stores game data used in the parallel reality game provided to the clients 120 over the network 105.

[0031] The game data stored in the game database 115 may include (1) data associated with the virtual world in the parallel reality game (e.g., image data used to render the virtual world on a display device, geographic coordinates of locations within the virtual world, etc.); (2) data associated with the players of the parallel reality game (e.g., player profiles including, but not limited to, player information, player experience level, player currency, current player location in the virtual / real world, player energy level, player preferences, team information, faction information, etc.); (3) data associated with game goals (e.g., data associated with current game goals, game goal states, past game goals, future game goals, desired game goals, etc.); and (4) data associated with virtual elements in the virtual world (e.g., location of virtual elements, type of virtual elements, etc.). The game data stored in the game database 115 may include (e.g., game objectives associated with the virtual elements, corresponding real world location information of the virtual elements, behavior of the virtual elements, associations of the virtual elements, etc.), (5) data associated with real world objects, landmarks, locations linked to the virtual world elements (e.g., location of the real world object / landmark, description of the real world object / landmark, associations of the virtual elements linked to the real world objects, etc.), (6) game status (e.g., current number of players, current status of game objectives, player leaderboard, etc.), (7) data associated with player actions / inputs (e.g., current player location, past player locations, player movements, player inputs, player queries, player communications, etc.), and (8) any other data used, related to, or obtained during the conduct of the parallel reality game. The game data stored in the game database 115 may be entered either offline or in real time by a system administrator or by data received from users / players of the system 100, such as from client devices 110 over the network 105.

[0032] The game server 120 may be configured to receive requests for game data from the client devices 110 (e.g., via remote procedure calls (RPCs)) and respond to those requests over the network 105. For example, the game server 120 may encode the game data in one or more data files and provide the data files to the client devices 110. In addition, the game server 120 may be configured to receive game data (e.g., player positions, player actions, player inputs, etc.) from the client devices 110 over the network 105. For example, the client devices 110 may be configured to periodically send player inputs and other updates to the game server 120, which uses them to update the game data in the game database 115 to reflect any and all changed conditions of the game.

[0033] In the embodiment shown, server 120 includes universal game module 150, commercial game module 155, data collection module 160, event module 165, and training system 170. As described above, game server 120 interacts with game database 115, which may be part of game server 120 or may be accessed remotely (e.g., game database 115 may be a distributed database accessed over network 105). In other embodiments, game server 120 includes different and / or additional elements. Furthermore, functionality may be distributed among elements in a manner different than that described. For example, game database 115 may be integrated into game server 120.

[0034] The universal game module 150 hosts the parallel reality game for all players and serves as the authoritative source of the current status of the parallel reality game for all players. As a host, the universal game module 150 generates game content for presentation to players, for example, via their respective client devices 110. The universal game module 150 may access the game database 115 to retrieve and / or store game data when hosting the parallel reality game. The universal game module 150 also receives game data (e.g., depth information, player input, player position, player actions, landmark information, etc.) from the client devices 110 and incorporates the received game data into the overall parallel reality game for all players of the parallel reality game. The universal game module 150 may also manage the distribution of game data to the client devices 110 over the network 105. The universal game module 150 may also manage security aspects of the client devices 110, including, but not limited to, securing connections between the client devices 110 and the game server 120, establishing connections between the various client devices 110, and verifying the location of the various client devices 110.

[0035] The commercial games module 155 may be separate from or part of the universal games module 150 in embodiments where it is included. The commercial games module 155 may manage the inclusion of various game features in the parallel reality game that are linked to commercial activities in the real world. For example, the commercial games module 155 may receive requests over the network 105 (via the network interface) from external systems, such as sponsors / advertisers, businesses, or other entities, to include game features linked to commercial activities in the parallel reality game. The commercial games module 155 may then arrange to include these game features in the parallel reality game.

[0036] The game server 120 may further include a data collection module 160. The data collection module 160 may be separate from or part of the universal game module 150 in embodiments where it is included. The data collection module 160 may manage the inclusion of various game features in the parallel reality game that are linked with data collection activities in the real world. For example, the data collection module 160 may modify the game data stored in the game database 115 to include game features linked with data collection activities in the parallel reality game. The data collection module 160 may also analyze data collected by players according to data collection activities and provide the data for access by various platforms.

[0037] The event module 165 manages player access to events within a parallel reality game. While the term "event" is used for convenience, it should be understood that the term does not necessarily refer to a specific event at a specific location or time. Rather, it may refer to any offering of access-controlled game content in which one or more access criteria are used to determine whether a player may access that content. Such content may be part of a larger parallel reality game that includes game content with less or no access control, or may be a stand-alone access-controlled parallel reality game.

[0038] Training system 170 trains a scene reconstruction model, which may be provided to client device 110, for example. Training system 170 receives image data for use in training the scene reconstruction model. In general, training system 170 inputs image data into the scene reconstruction model to predict depth maps and feature maps for each image, determines a 3D voxel array (e.g., a TSDF volume), generates a raw height field from the 3D voxel array, generates a top-down feature map based on the image feature maps, regresses a refined height field based on the top-down feature map, and blends the raw height field and the refined height field into a final height field.

[0039] To train the scene reconstruction model, the training system 170 calculates an overall loss based on the final height field compared to the ground truth height field. In one or more steps, the training system 170 may tune one or more parameters of the scene reconstruction model to minimize the overall loss.

[0040] In some embodiments, training system 170 may train various steps asynchronously, i.e., by utilizing separate ground truth and separate losses for training subsets of steps. For example, training system 170 may asynchronously train a depth network used to predict depth maps and / or asynchronously train a feature network used to generate feature maps. The general process above describes a supervised training algorithm. In one or more embodiments, unsupervised training trains without ground truth knowledge of the height field.

[0041] Once the scene reconstruction model is trained, it receives image data and outputs a 3D representation, which may be the final height field of the scene. The scene reconstruction model may also receive additional image data to iteratively expand and / or update the 3D representation. For example, the scene reconstruction model may input a first image having a viewpoint on a first region of the environment and output a 3D representation of the first region. When another set of images having a viewpoint on a second region of the environment is captured by the camera assembly, the scene reconstruction model may expand the 3D representation to incorporate the second region in addition to the first region. In some examples, the first region and the second region may at least partially overlap. The overlapping portions of the second set of images may refine and / or update the 3D representation of the first region.

[0042] The network 105 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof. The network may also include a direct connection between the client device 110 and the game server 120. In general, communications between the game server 120 and the client device 110 may be transmitted over the network interface using any type of wired and / or wireless connection, using a variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML, JSON), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0043] The technology discussed herein refers to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted to and from such systems. Those skilled in the art will recognize that the inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among components. For example, the server processes discussed herein may be implemented using a single server or multiple servers working in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0044] Additionally, in situations where the systems and methods discussed herein access and analyze personal information about a user, or utilize personal information such as location information, the user may be provided with an opportunity to control whether the program or function collects information and to control whether and / or how the user receives content from the system or other applications. Such information or data will not be collected or used until the user is provided with meaningful notice of what information will be collected and how that information will be used. Information will not be collected or used unless the user consents, and consent can be revoked or changed by the user at any time. Thus, the user may control how information is collected about the user and used by the application or system. Furthermore, certain information or data may be handled in one or more ways before being stored or used, such that any personally identifiable information is removed. For example, a user's identifier may be handled such that personally identifiable information cannot be determined for the user.

[0045] (c) (Example Game Interface) 3 illustrates an embodiment of a game interface 300 that may be presented on the display of the client 120 as part of an interface between the player and the virtual world 210. The game interface 300 includes a display window 310 that may be used to display various other aspects of the game, such as the virtual world 210 and the player position 222 and the locations of the virtual elements 230, virtual items 232, and virtual energy 250 within the virtual world 210. The user interface 300 may also display other information, such as game data information, game communications, player information, client location instructions, and other information associated with the game. For example, the user interface may display player information 315, such as the player's name, experience level, and other information. The user interface 300 may include a menu 320 for accessing various game settings and other information associated with the game. The user interface 300 may also include a communication interface 330 that enables communication between the game system and the player, and between one or more players of a parallel reality game.

[0046] According to aspects of the present disclosure, a player may interact with a parallel reality game by carrying a client device 120 around the real world. For example, a player may play the game by simply accessing an application associated with the parallel reality game on a smartphone and moving through the real world with the smartphone. In this regard, a player does not need to continually see a visual representation of the virtual world on a display screen to play a location-based game. As a result, the user interface 300 may include non-visual elements that allow a user to interact with the game. For example, the game interface may provide audible notifications to the player when the player is approaching a virtual element or object in the game or when an important event occurs in the parallel reality game. The player may control these audible notifications using the audio controls 340. Different types of audible notifications may be provided to the user depending on the type of virtual element or event. The audible notifications may increase or decrease in frequency or volume depending on the player's proximity to the virtual element or object. Other non-visual notifications and signals, such as vibration notifications or other suitable notifications or signals, may be provided to the user.

[0047] In some embodiments, the game interface 300 may be an augmented reality experience. The game interface 300 may display a real-world environment captured by a camera assembly, augmented by virtual content generated by a gaming module. In the example shown in FIG. 3, a ground environment is captured by a camera assembly. Overlaid on the real-world image are virtual objects (e.g., virtual elements 230, virtual items 232, virtual energy 250, virtual characters, etc.). The virtual objects may be generated and positioned based on the 3D representation generated by the scene reconstruction model. For example, virtual objects may be dropped from the sky and bounce off surfaces within the real-world object based on the 3D representation generated by the scene reconstruction model.

[0048] Using the disclosure provided herein, one of ordinary skill in the art will appreciate that numerous game interface configurations and underlying functionality will be apparent in light of the present disclosure, and the present disclosure is not intended to be limited to any one particular configuration.

[0049] (d) (Scene reconstruction model architecture) 4 illustrates an exemplary architecture of a scene reconstruction model 400 according to one or more embodiments. In the embodiment illustrated in FIG. 4, the scene reconstruction model 400 comprises the following components: a depth network 430, a feature network 440, a 3D modeling module 450, a ray casting module 460, a feature aggregation module 470, a top-down network 480, and a blending module 490. In other embodiments, the scene reconstruction model 400 may include additional, fewer, or different components than those listed herein.

[0050] The scene reconstruction model 400 may be a machine learning model trained by the training system 170 using a training set of images 405 with camera poses 410. To supervise the training, the scene reconstruction model 400 may utilize one or more types of ground truth information. For example, the training set of images may be accompanied by a ground truth 3D representation of the environment. The training system 170 may calculate an overall loss between the ground truth 3D representation and the final height field 495 generated by the scene reconstruction model 400. In other examples, the training system 170 may utilize other ground truth data to calculate the loss of one or more components. Once trained, the scene reconstruction model 400 is configured to input a test set of images 405 and the camera poses 410 to generate a final height field 495 of the environment captured in the test set of images 405.

[0051] The input typically includes an image 405 and a camera pose 410 for the image 405. In some embodiments, the camera pose 410 may be estimated based on the images 405. A pose estimation model may be implemented to estimate the relative pose of each image. In other embodiments, the camera pose 410 may be determined based on information captured by a position sensor (e.g., an inertial measurement unit, an accelerometer, a gyroscope, a GPS device, etc.).

[0052] The shared network 420 encodes the image 405. The shared network 420 may encode the image 405 via one or more dimensionality reduction algorithms. For example, the shared network 420 may utilize convolutional layers to reduce the dimensionality of the image 405. In other examples, the shared network 420 may extract one or more deep features of the image 405. The deep features may include, but are not limited to, abstract features, objects of interest, corners, edges, object contours, horizons, semantic segmentation, other features of the input image, or some combination thereof. In some embodiments, the shared network 420 is omitted from the scene reconstruction model 400. In such embodiments, the image 405 and the camera pose 410 are fed to a depth network to predict per-image depths 435, and the image 405 is fed to a feature network to extract per-image features 445.

[0053] The depth network 430 estimates the depth 435 for each image. The depth network 430 may input both the deep features from the shared network 420 and the camera pose 410. The depth network 430 may be trained as a monocular depth estimation model, which inputs an image and outputs a depth map.

[0054] In one or more embodiments, the monocular depth estimation model may be trained in a supervised manner with ground truth depth for a training set of images. The ground truth depth may be captured via a detection and ranging sensor (e.g., a light detection and ranging sensor (LIDAR)). The training system 170 may input the images into the depth network 430 (which may further pass through the shared network 420) and output a depth prediction. The training system 170 may calculate a loss between the depth prediction and the ground truth depth. Training the depth estimation model involves tuning the parameters of the depth estimation model to minimize the loss.

[0055] In some embodiments, the monocular depth estimation model may be trained in an unsupervised manner without ground truth depth, rather using stereo images. The stereo images may have a known or predicted camera pose between the stereo images. The training system 170 predicts the depth of a first image in a stereo image pair and projects the first image onto a second image to generate a composite image, the projection being based on the depth and the camera pose. The training system 170 calculates a loss between the composite image and the second image. Training the depth estimation model includes tuning parameters of the depth estimation model to minimize the loss over a training set of stereo image pairs. In some embodiments, the stereo image pairs may be true stereo image pairs captured by two or more cameras at known relative poses. In other embodiments, the stereo image pairs may be pseudo stereo image pairs captured by cameras at different poses within a proximal timestamp.

[0056] The feature network 440 extracts features 445 per image. Each image has a set of features 445 extracted from the image. The feature network 440 may employ one or more layers to extract features. For example, an edge detection layer may extract edges of objects in the image 405. Other features include abstract features, objects of interest, corners, edges, object contours, horizons, semantic segmentation, other features of the input image, or some combination thereof. The feature network 440 may be trained separately from other components of the scene reconstruction model 400. For example, the edge detection layer may be trained separately and fixed during the training of other components in the scene reconstruction model 400.

[0057] The 3D modeling module 450 inputs the per-image depths 435 to generate a raw 3D model 455 of the environment. The 3D model 455 may be a voxel array in three-dimensional space. The voxel array identifies each voxel as occupied or unoccupied. The 3D modeling module 450 may generate a 3D model with a truncated signed distance field (TSDF). A TSDF is a 3D voxel array that represents objects in a volume of space where each voxel is labeled with the distance to the nearest surface. The multiple input images 405 may help smooth the 3D model 455. Voxel hashing may be utilized to reduce the memory size of the 3D model 455 to speed up computations. The raw height field 465 is generally sharp around areas captured by multiple images, but is incomplete in areas with little signal from the images 405.

[0058] The ray casting module 460 generates a raw height field 465 based on the 3D model 455. For each location in the raw height field, the ray casting module 460 casts a ray downward against the surface of the 3D model 455 to determine the height of the surface at that location. The ray casting module 460 assigns a height to each cell in 2D space based on ray casting of the 3D model 455. The raw height field 465 may be converted to a point cloud representing each cell as a 3D point.

[0059] The feature aggregation module 470 aggregates features 475 from the per-image features 445 based in part on the raw height field 465. For each set of per-image features 445, the feature aggregation module 470 transposes the raw height field 465 or point cloud to the camera pose viewpoint for the image. The feature aggregation module 470 identifies whether the cell is visible or hidden from the camera viewpoint. For visible cells, the feature aggregation module 470 may extract features corresponding to the location of the visible cell. The feature aggregation module 470 may then average the features extracted from each set of per-image features 445 at each cell, thereby resulting in an aggregated feature 475. This depth-based sampling speeds up computations because the scene reconstruction model 400 does not need to project the per-image features 445 to all voxels along a ray for a given pixel in the image. Thus, the depth-based sampling quickly maps features to cells or locations in the point cloud. Aggregated features 475 may collate each type of feature into a separate tensor, for example a first feature type aggregated into a first 2D tensor, a second feature type aggregated into a second 2D tensor, and so on for any additional feature types.

[0060] The top-down network 480 inputs the aggregated features 475 and regresses a refined height field 485. The top-down network may be a machine learning model (e.g., a convolutional neural network (CNN)). The output is a refined height field 485. The refined height field 485 may be more complete but less sharp on shape compared to the raw height field 465. The top-down network 480 may further output a blend map that predicts where the refined height field 485 is closer to the ground truth than the raw height field 465. The training system 170 may train the top-down network 480 using a ground truth height field, for example, determined by more robust 3D modeling utilizing ground truth depth.

[0061] The blending module 490 blends the raw height field 465 and the refined height field 485 into a final height field 495. The blending module 490 utilizes a blend map to determine the contribution of the raw height field 465 and the contribution of the refined height field 485 for each cell.

[0062] When training the scene reconstruction model 400, the training system 170 may utilize a ground truth height field. In some ground truth height fields, the ground truth is sparse. Therefore, the training system 170 may utilize a mask that restricts the loss to comparable regions.

[0063] (e) (Exemplary Method) FIG. 5 is a flow chart illustrating a method 500 for generating a height field using a scene reconstruction model, according to one or more embodiments. During training, the method 500 is typically repeated multiple times with different sets of training image data until one or more training objectives are met. A training system (e.g., training system 170) may perform the training utilizing a loss function to evaluate the predictive ability of the scene reconstruction model. During inference or deployment, a client device (e.g., client device 110) may perform the method 500 to generate a height field for image data captured by a camera assembly (e.g., camera assembly 125). The steps of FIG. 5 are illustrated in terms of a general computing device that performs the method 500. However, some or all of the steps may be performed by other entities and / or components. In addition, some embodiments may perform steps in parallel, perform steps in a different order, or perform different steps.

[0064] The generic computing device predicts 510 a depth map and a feature map for each image in the image data. The scene reconstruction model may include a depth network for predicting the depth map from the images and a feature network for extracting the feature map from the images. In one or more embodiments, the scene reconstruction model further includes an additional sharing network implemented upstream of the depth network and the feature network. In such an embodiment, the generic computing device inputs the image data into the sharing network for encoding the image data.

[0065] The general computing device generates the raw height field by ray casting a predicted depth map of the image. The general computing device may build a 3D voxel array based on the predicted depth map. The 3D voxel array may be represented as a TSDF volume. The TSDF volume is generated by the general computing device by determining whether a voxel is occupied by a volumetric object based on the predicted depth map and the camera pose of the image. The general computing device generates the raw height field by ray casting from a top-down perspective above the environment.

[0066] The general computing device generates an aggregate feature map based on the feature map of the image. The general computing device may utilize the TSDF volume as a starting point. The general computing device may sample features from the feature map according to the raw height field. For example, the general computing device identifies visible and occluded cells from the camera pose of the image. From the visible cells, the general computing device samples features from the feature map. The general computing device may aggregate the sampled features from the image to generate the aggregate feature map. One embodiment of the aggregation includes averaging features across the feature map at a particular location. The feature map may include a tensor for each feature type, e.g., a first tensor for a first feature type, a second tensor for a second feature type, and so on for any additional feature types.

[0067] A typical computing device performs supervised regression to determine a refined height field based on the aggregate feature map. A regression layer may input the aggregate feature map and output a refined height field. The regression layer may implement a deep learning neural network. The top-down network may further output a blend map that predicts cells where the refined height field is likely to be closer to the ground truth height field compared to the raw height field. In other words, the blend map represents the confidence that the top-down network has in the refined height field's representation of the scene's geometry.

[0068] The generic computing device determines the final height field as a combination of the raw height field and the refined height field, and in one or more embodiments, the generic computing device blends the raw height field and the refined height field according to a blend map.

[0069] When training the scene reconstruction model, the training system may utilize the ground truth height field to evaluate the loss between the final height field output by the scene reconstruction model and the ground truth height field. The training system may train the scene reconstruction model by adjusting parameters to minimize the loss. In one example, the overall loss may incorporate a gradient matching loss.

[0070] When training the scene reconstruction model, the training system may synchronously train one or more components of the scene reconstruction model. For example, the training system may synchronously train the top-down network and the blending module using the ground truth height field.

[0071] In other embodiments, the training system may train one or more components asynchronously. For example, the training system may train the deep network separately. Once trained, the training system may fix the deep network, i.e., hold the parameters non-tunable, while training the other components. As another example, the training system may train the feature network separately.

[0072] 6 is a flow chart illustrating a method 600 of applying a height field generated by a scene reconstruction model, according to one or more embodiments. The steps of FIG. 6 are shown from the perspective of a client device performing the method 600. However, some or all of the steps may be performed by other entities and / or components. In addition, some embodiments may perform steps in parallel, perform steps in a different order, or perform different steps.

[0073] The client device receives 610 real-time image data of the scene. The image of the scene may be captured by a camera that is a component of the client device or is external to the client device. In the context of a parallel reality game, the scene may be a real-world location that may be mapped to a virtual location in the virtual world. The image of the scene may also have intrinsics that correspond to the geometric characteristics of the camera that captured the image. The image may be a single image captured by the camera. Alternatively, the image may be a frame from a video being captured by the camera.

[0074] The client device inputs real-time image data of the scene to the trained scene reconstruction model 620. The scene reconstruction model may be trained by a training system, for example, as described in conjunction with Figure 5. The scene reconstruction model receives the image data of the scene.

[0075] The scene reconstruction model generates 630 a 3D model (e.g., including a height field). As mentioned above, the 3D model represents the spatial arrangement of objects in the scene. In one or more embodiments, the 3D model includes a height field, where each 2D location has a height value for the object located at that 2D location.

[0076] The client device may perform various additional operations using the 3D representation and / or the height field.

[0077] In one embodiment, the client device generates 640 virtual content for an augmented reality experience, for example in a parallel reality game. The virtual content may be generated such that the virtual content interacts with surfaces of physical objects as represented in the 3D representation of the scene. As an example, a virtual character may be shown to jump on top of a physical object as informed by the 3D representation and / or a height field.

[0078] The client device displays an image of the scene augmented with the virtual content 650. The client device's display can provide a constant feed of video captured by a camera with the augmented virtual content. For example, physical objects can be augmented with virtual content that interacts with the physical objects.

[0079] In another embodiment, the client device generates 660 navigation instructions for the autonomous agent based on the 3D representation and / or the height field. The navigation instructions may inform navigation of the agent around the terrain of the environment, for example, as represented in the 3D representation and / or the height field. For example, the autonomous vehicle may move around physical objects that stand above the ground floor where the autonomous vehicle is located. In other examples, the autonomous vehicle may ascend or descend various heights. The navigation instructions may instruct the autonomous vehicle to approach a height disparity to ascend or descend.

[0080] The client device navigates the autonomous agent with navigation instructions 670. The navigation instructions may control one or more motors or actuators on the autonomous agent to navigate the agent through the environment.

[0081] In additional embodiments (not shown in FIG. 6), the client device may utilize the 3D representation and / or height field for other applications. In one embodiment, the client device may utilize the height field to determine one or more visual occlusions in the image. A visual occlusion is an area occluded by a physical object captured in the image data. For example, an image of a box placed on the ground hides objects and / or surfaces behind the box from the camera's perspective. The client device may utilize the 3D representation and / or height field to inform visual occlusions in the image data. Based on the visual occlusions, the client device may generate virtual content that may be partially hidden when placed behind a physical object captured in the image data. In another embodiment, the client device may utilize the 3D representation and / or height field to generate a floor map that estimates floor space in the environment. The floor map may inform placement of virtual elements (e.g., from an augmented reality game). In yet another embodiment, the client device may utilize the height field to build a 3D model of the environment.

[0082] (f) (Example of a computing system) FIG. 7 is an exemplary architecture of a computing device, according to one embodiment. Although FIG. 7 illustrates a high-level block diagram showing the physical components of a computer that may be used as part or all of one or more entities described herein, according to one embodiment, the computer may have additional, fewer, or variations of the components provided in FIG. 7. Although FIG. 7 illustrates a computer 700, the diagram is intended as a functional description of various features that may be present in a computer system, rather than as a structural schematic of the implementations described herein. In practice, items shown separately may be combined and some items may be separated, as will be recognized by those skilled in the art.

[0083] 7 shows at least one processor 702 coupled to a chipset 704. Also coupled to the chipset 704 are memory 706, a storage device 708, a keyboard 710, a graphics adapter 712, a pointing device 714, and a network adapter 716. A display 718 is coupled to the graphics adapter 712. In one embodiment, the functionality of the chipset 704 is provided by a memory controller hub 720 and an I / O controller hub 722. In another embodiment, the memory 706 is coupled directly to the processor 702 instead of the chipset 704. In some embodiments, the computer 700 includes one or more communication buses for interconnecting these components. The one or more communication buses optionally include circuitry (also referred to as a chipset) that interconnects and controls communication between the system components.

[0084] The storage device 708 is any non-transitory computer-readable storage medium, such as a hard drive, a compact disk read-only memory (CD-ROM), a DVD, or a solid-state memory device or other optical storage device, a magnetic cassette, a magnetic tape, a magnetic disk storage device or other magnetic storage device, a magnetic disk storage device, an optical disk storage device, a flash memory device, or other non-volatile solid-state storage device. Such a storage device 708 may also be referred to as persistent memory. The pointing device 714 may be a mouse, a trackball, or other type of pointing device and may be used in combination with the keyboard 710 to input data into the computer 700. The graphics adapter 712 displays images and other information on the display 718. The network adapter 716 couples the computer 700 to a local or wide area network.

[0085] The memory 706 holds instructions and data used by the processor 702. The memory 706 may be a non-persistent memory, examples of which include high-speed random access memory such as DRAM, SRAM, DDRRAM, ROM, EEPROM, flash memory, etc.

[0086] As is known in the art, computer 700 may have different and / or other components than those shown in Figure 7. Additionally, computer 700 may lack certain illustrated components. In one embodiment, a computer 700 functioning as a server may lack a keyboard 710, a pointing device 714, a graphics adapter 712, and / or a display 718. Additionally, storage device 708 may be local and / or remote from computer 700 (such as embodied in a storage area network (SAN)).

[0087] As known in the art, computer 700 is adapted to execute computer program modules for providing the functionality described herein. As used herein, the term "module" refers to computer program logic utilized to provide a specified functionality. Thus, a module may be implemented in hardware, firmware, and / or software. In one embodiment, the program module is stored in storage device 708, loaded into memory 706, and executed by processor 702.

[0088] (g) (Additional Considerations) Some portions of the above describe embodiments in terms of algorithmic processes or operations. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. While these operations have been described functionally, computationally, or logically, they will be understood to be implemented by computer programs including instructions for execution by a processor or equivalent electrical circuitry, microcode, or the like. Additionally, and without loss of generality, it may sometimes be convenient to refer to these arrangements of functional operations as modules.

[0089] As used herein, a reference to "one embodiment" or "one embodiment" means that a particular element, feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

[0090] Some embodiments may be described using the terms "coupled" and "connected," along with their derivatives. It should be understood that these terms are not intended as synonyms for each other. For example, an embodiment may be described using the term "connected" to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, an embodiment may be described using the term "coupled" to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other. The embodiments are not limited in this context.

[0091] As used herein, "comprises," "comprising," "includes," "including," "has," "having," or any other variation thereof, is intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent in such process, method, article, or apparatus. Furthermore, unless expressly stated otherwise, "or" refers to an inclusive "or," not an exclusive "or." For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or absent), A is false (or absent) and B is true (or present), and both A and B are true (or present).

[0092] Furthermore, the use of "a" or "an" is employed to describe elements and components of the embodiments herein. This is done merely for convenience and to give a general sense of the disclosure. The description should be read to include one or at least one, and the singular also includes the plural, unless otherwise clearly meant.

[0093] Upon reading this disclosure, those skilled in the art will appreciate that still further alternative structural and functional designs for the system and process for verifying an account with an online service provider correspond to a genuine business. Thus, while specific embodiments and applications have been illustrated and described, it is to be understood that the described subject matter is not limited to the exact structure and components disclosed herein, and that various modifications, changes, and variations that will be apparent to those skilled in the art may be made in the arrangement, operation and details of the disclosed methods and apparatus. The scope of protection should be limited only by the following claims.

Claims

1. 1. A computer-implemented method comprising: Receiving real-time image data of a scene captured by a camera assembly of the mobile device; inputting the real-time image data of the scene into a scene reconstruction model; receiving from the scene reconstruction model a final height field of the scene comprising height values ​​at each 2D position of the scene, the scene reconstruction model for each image of the image data, predicting a depth map based on the image; for each image of the image data, extracting a feature map based on the image; generating a raw height field based on the predicted depth map of the image; generating an aggregate feature map based on the feature map of the image; regressing a refined height field based on the aggregate feature map; and determining the final height field as a combination of the raw height field and the refined height field; generating the final height field by using the final height field to generate virtual content for display in conjunction with the real-time image data of the scene; 23. A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, wherein the real-time image data includes a plurality of images and a camera pose for each image.

3. 3. The computer-implemented method of claim 2, wherein the camera pose for each image is captured by a position sensor of the mobile device.

4. The computer-implemented method of claim 2 , wherein the camera pose for each image is estimated by a pose estimation model based on the image.

5. 2. The computer-implemented method of claim 1, wherein predicting the depth map comprises applying a depth estimation model to the image to determine the depth map.

6. 2. The computer-implemented method of claim 1, wherein extracting the feature map comprises applying a convolutional network to the image to determine the feature map.

7. 2. The computer-implemented method of claim 1, wherein the feature map includes a first tensor of a first feature type and a second tensor of a second feature type.

8. 2. The computer-implemented method of claim 1, wherein the feature map includes a first tensor of a first feature type and a second tensor of a second feature type.

9. generating said raw height field generating a 3D model using the truncated signed distance field together with the predicted depth map; ray-casting the 3D model to generate the raw height field; 2. The computer-implemented method of claim 1, comprising:

10. 10. The computer-implemented method of claim 9, wherein ray-casting the 3D model to generate the raw height field comprises, for each location in the raw height field, casting a ray downward against a surface of the 3D model to determine a height of the surface at that location.

11. Generating the aggregate feature map includes: For each image of the image data, transposing the raw height field to a camera pose viewpoint for the image; identifying whether each location in the raw height field is visible or occluded at the viewpoint of the camera pose; Sampling features from said visible locations; for each location of the aggregate feature map, averaging one or more features of the image data at that location; 2. The computer-implemented method of claim 1, comprising:

12. 2. The computer-implemented method of claim 1, wherein regressing the refined height field comprises applying a machine learning model to the aggregate feature map.

13. 13. The computer-implemented method of claim 12, wherein the scene reconstruction model generates the final height field by further outputting, at each location, a blend map that predicts a confidence level of the refined height field at that location.

14. 14. The computer-implemented method of claim 13, wherein determining the final height field as a combination of the raw height field and the refined height field includes, at each location of the final height field, a contribution of the refined height field at that location based on the confidence level of that location in the blend map.

15. 2. The computer-implemented method of claim 1, wherein generating the virtual content comprises generating virtual objects that lie on a surface within the final height field.

16. 2. The computer-implemented method of claim 1, wherein generating the virtual content includes generating a virtual object that can move between two or more surfaces within the final height field.

17. 1. A non-transitory computer-readable storage medium storing instructions that, when executed by a computer processor, cause the processor to: Receiving real-time image data of a scene captured by a camera assembly of the mobile device; inputting the real-time image data of the scene into a scene reconstruction model; receiving from the scene reconstruction model a final height field of the scene comprising height values ​​at each 2D position of the scene, the scene reconstruction model for each image of the image data, predicting a depth map based on the image; for each image of the image data, extracting a feature map based on the image; generating a raw height field based on the predicted depth map of the image; generating an aggregate feature map based on the feature map of the image; regressing a refined height field based on the aggregate feature map; and generating the final height field by determining the final height field as a combination of the raw height field and the refined height field; using the final height field to generate virtual content for display in conjunction with the real-time image data of the scene; A non-transitory computer-readable storage medium having a program for executing the program.

18. 1. A system comprising: A computer processor; a non-transitory computer-readable storage medium storing instructions that, when executed by the computer processor, cause the processor to: Receiving real-time image data of a scene captured by a camera assembly of the mobile device; inputting the real-time image data of the scene into a scene reconstruction model; receiving from the scene reconstruction model a final height field of the scene comprising height values ​​at each 2D position of the scene, the scene reconstruction model for each image of the image data, predicting a depth map based on the image; for each image of the image data, extracting a feature map based on the image; generating a raw height field based on the predicted depth map of the image; generating an aggregate feature map based on the feature map of the image; regressing a refined height field based on the aggregate feature map; and generating the final height field by determining the final height field as a combination of the raw height field and the refined height field; using the final height field to generate virtual content for display in conjunction with the real-time image data of the scene; A system for executing the above.

19. 1. A computer-implemented method comprising: Receiving real-time image data of a scene captured by a camera assembly of the autonomous vehicle; inputting the real-time image data of the scene into a scene reconstruction model; receiving from the scene reconstruction model a final height field of the scene comprising height values ​​at each 2D position of the scene, the scene reconstruction model for each image of the image data, predicting a depth map based on the image; for each image of the image data, extracting a feature map based on the image; generating a raw height field based on the predicted depth map of the image; generating an aggregate feature map based on the feature map of the image; regressing a refined height field based on the aggregate feature map; and generating the final height field by determining the final height field as a combination of the raw height field and the refined height field; generating navigation instructions for navigating the autonomous vehicle within the scene using the final height field; and Navigating the autonomous vehicle according to the navigation instructions; and 23. A computer-implemented method comprising:

20. 20. The computer-implemented method of claim 19, wherein the real-time image data includes a plurality of images and a camera pose for each image.

21. 21. The computer-implemented method of claim 20, wherein the camera pose for each image is captured by a position sensor of the mobile device.

22. 21. The computer-implemented method of claim 20, wherein the camera pose for each image is estimated based on the image by a pose estimation model.

23. 20. The computer-implemented method of claim 19, wherein predicting the depth map comprises applying a depth estimation model to the image to determine the depth map.

24. 20. The computer-implemented method of claim 19, wherein extracting the feature map comprises applying a convolutional network to the image to determine the feature map.

25. 20. The computer-implemented method of claim 19, wherein the feature map includes a first tensor of a first feature type and a second tensor of a second feature type.

26. 20. The computer-implemented method of claim 19, wherein the feature map includes a first tensor of a first feature type and a second tensor of a second feature type.

27. generating said raw height field generating a 3D model using the truncated signed distance field together with the predicted depth map; ray-casting the 3D model to generate the raw height field; 20. The computer-implemented method of claim 19, comprising:

28. 28. The computer-implemented method of claim 27, wherein ray-casting the 3D model to generate the raw height field comprises, for each location in the raw height field, casting a ray downward against a surface of the 3D model to determine a height of the surface at that location.

29. Generating the aggregate feature map includes: For each image of the image data, transposing the raw height field to a camera pose viewpoint for the image; identifying whether each location in the raw height field is visible or occluded at the viewpoint of the camera pose; Sampling features from said visible locations; for each location of the aggregate feature map, averaging one or more features of the image data at that location; 20. The computer-implemented method of claim 19, comprising:

30. 20. The computer-implemented method of claim 19, wherein regressing the refined height field comprises applying a machine learning model to the aggregate feature map.

31. 31. The computer-implemented method of claim 30, wherein the scene reconstruction model generates the final height field by further outputting, at each location, a blend map that predicts a confidence level of the refined height field at that location.

32. 32. The computer-implemented method of claim 31 , wherein determining the final height field as a combination of the raw height field and the refined height field comprises, at each location of the final height field, a contribution of the refined height field at that location based on the confidence of that location in the blend map.

33. 20. The computer-implemented method of claim 19, wherein generating navigation instructions includes generating navigation instructions for transitioning the autonomous vehicle from a first surface at a first elevation to a second surface at a second elevation.

34. 20. The computer-implemented method of claim 19, wherein generating the navigation instructions includes generating navigation instructions for navigating the autonomous vehicle on a first surface at a first elevation and around a second surface at a second elevation different from the first surface.

35. 1. A non-transitory computer-readable storage medium storing instructions that, when executed by a computer processor, cause the processor to: Receiving real-time image data of a scene captured by a camera assembly of the autonomous vehicle; inputting the real-time image data of the scene into a scene reconstruction model; receiving from the scene reconstruction model a final height field of the scene comprising height values ​​at each 2D position of the scene, the scene reconstruction model for each image of the image data, predicting a depth map based on the image; for each image of the image data, extracting a feature map based on the image; generating a raw height field based on the predicted depth map of the image; generating an aggregate feature map based on the feature map of the image; regressing a refined height field based on the aggregate feature map; and generating the final height field by determining the final height field as a combination of the raw height field and the refined height field; generating navigation instructions for navigating the autonomous vehicle within the scene using the final height field; and Navigating the autonomous vehicle according to the navigation instructions; and A non-transitory computer-readable storage medium having a program for executing the program.

36. 1. A system comprising: A computer processor; and a non-transitory computer-readable storage medium storing instructions that, when executed by the computer processor, cause the processor to: Receiving real-time image data of a scene captured by a camera assembly of the autonomous vehicle; inputting the real-time image data of the scene into a scene reconstruction model; receiving from the scene reconstruction model a final height field of the scene comprising height values ​​at each 2D position of the scene, the scene reconstruction model for each image of the image data, predicting a depth map based on the image; for each image of the image data, extracting a feature map based on the image; generating a raw height field based on the predicted depth map of the image; generating an aggregate feature map based on the feature map of the image; regressing a refined height field based on the aggregate feature map; and generating the final height field by determining the final height field as a combination of the raw height field and the refined height field; generating navigation instructions for navigating the autonomous vehicle within the scene using the final height field; and Navigating the autonomous vehicle according to the navigation instructions; and A system for executing the above.

37. 1. A computer-implemented method comprising: receiving one or more sets of training image data of one or more scenes captured by one or more camera assemblies along with one or more ground truth height fields for the one or more scenes; inputting each set of training image data of the scene into a scene reconstruction model; receiving from the scene reconstruction model a final height field of the scene comprising height values ​​at each 2D position of the scene, the scene reconstruction model for each image of the set of training image data, predicting a depth map based on the image; for each image in the set of training image data, extracting a feature map based on the image; generating a raw height field based on the predicted depth map of the image; generating an aggregate feature map based on the feature map of the image; regressing a refined height field based on the aggregate feature map; and generating the final height field by determining the final height field as a combination of the raw height field and the refined height field; determining, for each set of training data, a loss between the final height field of the scene and the ground truth height field of the scene; training the scene reconstruction model to minimize the loss; 23. A computer-implemented method comprising:

38. 40. The computer-implemented method of claim 37, wherein the training image data includes a plurality of images and a camera pose for each image.

39. 40. The computer-implemented method of claim 38, wherein the camera pose for each image is captured by a position sensor of the mobile device.

40. 40. The computer-implemented method of claim 38, wherein the camera pose for each image is estimated by a pose estimation model based on the image.

41. 38. The computer-implemented method of claim 37, wherein predicting the depth map comprises applying a depth estimation model to the image to determine the depth map.

42. 38. The computer-implemented method of claim 37, wherein extracting the feature map comprises applying a convolutional network to the image to determine the feature map.

43. 38. The computer-implemented method of claim 37, wherein the feature map includes a first tensor of a first feature type and a second tensor of a second feature type.

44. 38. The computer-implemented method of claim 37, wherein the feature map includes a first tensor of a first feature type and a second tensor of a second feature type.

45. generating said raw height field generating a 3D model using the truncated signed distance field together with the predicted depth map; ray-casting the 3D model to generate the raw height field; 38. The computer-implemented method of claim 37, comprising:

46. 46. ​​The computer-implemented method of claim 45, wherein ray casting the 3D model to generate the raw height field comprises, for each location in the raw height field, casting a ray downward against a surface of the 3D model to determine a height of the surface at that location.

47. Generating the aggregate feature map includes: For each image of the image data, transposing the raw height field to a camera pose viewpoint for the image; identifying whether each location in the raw height field is visible or occluded at the viewpoint of the camera pose; Sampling features from said visible locations; for each location of the aggregate feature map, averaging one or more features of the image data at that location; 38. The computer-implemented method of claim 37, comprising:

48. 38. The computer-implemented method of claim 37, wherein regressing the refined height field comprises applying a machine learning model to the aggregate feature map.

49. 49. The computer-implemented method of claim 48, wherein the scene reconstruction model generates the final height field by further outputting, at each location, a blend map that predicts a confidence level of the refined height field at that location.

50. 50. The computer-implemented method of claim 49, wherein determining the final height field as a combination of the raw height field and the refined height field comprises, at each location of the final height field, a contribution of the refined height field at that location based on the confidence of that location in the blend map.

51. 40. The computer-implemented method of claim 37, wherein training the scene reconstruction model includes synchronously training one or more components of the scene reconstruction model.

52. 40. The computer-implemented method of claim 37, wherein training the scene reconstruction model includes asynchronously training one or more components of the scene reconstruction model.

53. 1. A non-transitory computer-readable storage medium storing instructions that, when executed by a computer processor, cause the processor to: receiving one or more sets of training image data of one or more scenes captured by one or more camera assemblies along with one or more ground truth height fields for the one or more scenes; inputting each set of training image data of the scene into a scene reconstruction model; receiving from the scene reconstruction model a final height field of the scene comprising height values ​​at each 2D position of the scene, the scene reconstruction model for each image of the image data, predicting a depth map based on the image; for each image of the image data, extracting a feature map based on the image; generating a raw height field based on the predicted depth map of the image; generating an aggregate feature map based on the feature map of the image; regressing a refined height field based on the aggregate feature map; and generating the final height field by determining the final height field as a combination of the raw height field and the refined height field; determining, for each set of training data, a loss between the final height field of the scene and the ground truth height field of the scene; training the scene reconstruction model to minimize the loss; 1. A non-transitory computer-readable storage medium comprising:

54. 1. A computer program product comprising: a non-transitory computer-readable storage medium storing a scene reconstruction model, the computer program product comprising: receiving one or more sets of training image data of one or more scenes captured by one or more camera assemblies along with one or more ground truth height fields for the one or more scenes; inputting each set of training image data of the scene into a scene reconstruction model; receiving from the scene reconstruction model a final height field of the scene comprising height values ​​at each 2D position of the scene, the scene reconstruction model for each image of the set of training image data, predicting a depth map based on the image; for each image of the training image data, extracting a feature map based on the image; generating a raw height field based on the predicted depth map of the image; generating an aggregate feature map based on the feature map of the image; regressing a refined height field based on the aggregate feature map; and generating the final height field by determining the final height field as a combination of the raw height field and the refined height field; determining, for each set of training data, a loss between the final height field of the scene and the ground truth height field of the scene; training the scene reconstruction model to minimize the loss; storing the scene reconstruction model on the non-transitory computer readable storage medium; 23. A computer program product manufactured by a process comprising:

Images