Generating and presenting third-party augmented reality content
Patent Information
- Application Number
- JP2023142212
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-08-31
- Filing Date
- 2023-09-01
- Publication Date
- 2026-09-08
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
[Technical field]
[0001] The subject matter described relates generally to augmented reality technology, and more particularly to presenting augmented reality content from third parties to users of online systems. [Background technology]
[0002] assignment Online systems, such as game servers, provide users with augmented reality experiences. In these experiences, virtual content is presented to the user along with the real world such that the virtual content appears to be within the real world. However, many online systems simply present the virtual content without the virtual content appearing to interact with the physical world. For example, a virtual character may appear to be walking in the physical world a certain distance away from the client device, but may be depicted on top of a tree that is closer to the client device than the virtual character. Thus, the augmented reality experiences provided by many online systems appear unnatural to the user, thereby providing a suboptimal experience.
[0003] Some online systems may generate virtual models of the physical world so that virtual objects may appear to interact with physical objects. For example, an online system may generate a virtual model that includes a 3D representation of physical objects in a physical space and animate the virtual objects so that, when rendered, the virtual objects appear to interact more realistically with the physical space. However, generating a virtual model of a physical space can be time and resource intensive. Thus, virtual models of physical spaces tend to be limited to sophisticated online systems with the resources to generate detailed virtual models, or niche projects with only a limited number of physical spaces.
[0004] Thus, content developers who want to create realistic augmented reality content are hampered by the resources required to generate virtual models of physical spaces.
[0005] overview This disclosure describes an online system, such as a game server, that allows third party content developers to create virtual content within virtual models available to the online system. The online system stores a set of virtual models of a set of physical spaces. The virtual models include 3D representations (e.g., 3D meshes) that represent objects in the physical spaces, such as buildings or plants. Each virtual model also includes an origin that serves as a reference point about which the 3D representations are positioned and oriented.
[0006] A content developer may generate custom virtual content for a physical space using one of the virtual models made available by the online system. For example, a system associated with the content developer may send a request for a virtual model to the online system. The online system sends the virtual model to the content developer system, which adds virtual objects to the virtual model. These virtual objects are placed and oriented within the virtual model relative to the origin of the virtual model. The content developer's system sends the modified virtual model to the online system, which stores the modified virtual model in association with the physical space.
[0007] To display the virtual content in the physical space, the user's client device receives the virtual model modified with the virtual objects from the content developer's system. The client device receives pose data describing a pose of the client device and identifies the pose in the physical world based on the pose data. The client device identifies a virtual pose in the virtual model that corresponds to the identified physical pose. The client device may identify the virtual pose by comparing a 3D representation of the physical object to the physical object detected in the pose data, or may identify the virtual pose based on a mapping between an origin of the virtual model and a location in the physical world. The client device modifies the streamed video from the client device's camera to include the virtual object based on the identified virtual pose. For example, the client device may render a virtual object based on the virtual pose of the client device and modify the video feed to include the rendered virtual object. Generally, the client device does not render a 3D representation of the physical object in the physical space such that the virtual object appears to be located in and interact with the physical space. However, in some cases, the client device may render a 3D representation, for example, to test how the virtual object may appear to a user in the physical space.
[0008] By providing a virtual model for a third-party content development system to use to generate custom virtual content, a content developer may easily generate virtual content to be presented as an augmented reality experience at a remote physical location by leveraging the virtual model already developed by the online system. Thus, a content developer may use the online system as a platform to more easily generate augmented reality content. [Brief description of the drawings]
[0009] [Figure 1]FIG. 1 is a diagram illustrating a representation of a virtual world with a parallel geography to the real world, according to one embodiment. [Diagram 2] FIG. 2 illustrates an exemplary game interface for a parallel reality game, according to one embodiment. [Diagram 3] FIG. 3 is a block diagram of a networked computing environment suitable for third-party generation of virtual content, according to one embodiment. [Figure 4] FIG. 4 is an interaction diagram illustrating the interaction between a client device, a game server, and a content developer system for generating and displaying custom virtual content, according to one embodiment. [Diagram 5] FIG. 5 is a flowchart of an exemplary method for a client device to display custom content within a virtual model, according to one embodiment. [Figure 6] FIG. 6 is a diagram illustrating customized virtual content included in a virtual model, according to one embodiment. [Figure 7] FIG. 7 is a diagram illustrating how the virtual content of FIG. 6 can be used to modify a broadcast video, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Detailed Description The drawings and the following description describe specific embodiments for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods may be employed without departing from the principles described. It should be noted that, wherever practicable, similar or similar reference numbers are used in the figures to indicate similar or similar functionality. Where elements share a common numeral, if the elements share a common numeral and are followed by another letter, this indicates that the elements are similar or identical. Unless the context dictates otherwise, reference to a numeral alone generally refers to any one or any combination of such elements.
[0011] Various embodiments are described in the context of a parallel reality game that includes augmented reality content in a virtual world geography that is parallel to at least a portion of the real world geography, such that player movements and actions in the real world affect actions in the virtual world. The subject matter described is applicable to other situations in which it is desirable to present customized virtual content generated by a third-party system. In addition, the inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among the components of the system.
[0012] Location-based parallel reality game example FIG. 1 is a conceptual diagram of a virtual world 110 parallel to a real world 100. The virtual world 110 may function as a game board for players of a parallel reality game. As shown, the virtual world 110 includes a geography that parallels the geography of the real world 100. In particular, a range of coordinates that defines a geographical area or space in the real world 100 is mapped to a corresponding range of coordinates that defines a virtual space in the virtual world 110. The range of coordinates in the real world 100 may be associated with a town, neighborhood, city, campus, neighborhood, country, continent, the entire globe, or other geographical area. Each geographic coordinate within the range of geographic coordinates is mapped to a corresponding coordinate in the virtual space in the virtual world 110.
[0013] The positions of the players in the virtual world 110 correspond to the positions of the players in the real world 100. For example, player A located at position 112 in the real world 100 corresponds to position 122 in the virtual world 110. Similarly, player B located at position 114 in the real world 100 corresponds to position 124 in the virtual world 110. As the players move within a range of geographic coordinates in the real world 100, they also move within a range of coordinates that define a virtual space in the virtual world 110. In particular, a positioning system (e.g., a GPS system, a positioning system, or both) associated with a mobile computing device carried by the player may be used to track the position of the player as the player navigates a range of geographic coordinates in the real world 100. Data associated with the player's position in the real world 100 is used to update the player's position in a corresponding range of coordinates that define a virtual space in the virtual world 110. In this manner, a player may navigate along a continuous track within a range of coordinates defining a virtual space in the virtual world 110 by simply moving within a corresponding range of geographic coordinates in the real world 100, without checking in at a specific individual location in the real world 100 or periodically updating location information.
[0014] Location-based games may include game objectives that require a player to travel to or interact with various virtual elements or objects scattered at various virtual locations within the virtual world 110. The player may navigate to these virtual locations by moving to the corresponding locations of the virtual elements or objects in the real world 100. For example, a positioning system may track the player's position such that as the player navigates the real world 100, the player also navigates the parallel virtual world 110. The player may then interact with the various virtual elements and objects at specific locations to accomplish or perform one or more game objectives.
[0015] The game goal may have the player interact with virtual elements 130 located at various virtual locations in the virtual world 110. These virtual elements 130 may be linked to landmarks, geographic locations, or objects 140 in the real world 100. The real world landmarks or objects 140 may be works of art, monuments, buildings, businesses, libraries, museums, or other suitable real world landmarks or objects. The interactions include capturing, claiming ownership, using some virtual items, using some virtual currency, etc. To capture these virtual elements 130, the player travels to the landmark or geographic location 140 linked to the virtual elements 130 in the real world and performs any necessary interactions (as defined by the rules of the game) with the virtual elements 130 in the virtual world 110. For example, Player A must travel to a landmark 140 in the real world 100 to interact with or capture the virtual element 130 linked to that particular landmark 140. Interaction with the virtual element 130 may require actions such as verifying, obtaining, or capturing other information about a landmark or object 140 associated with the virtual element 130 in the real world.
[0016] A game goal may require a player to use one or more virtual items that are collected by the player in the location-based game. For example, a player may travel through the virtual world 110 in search of virtual items 132 (e.g., weapons, creatures, power-ups, or other items) that may be useful in achieving the game goal. These virtual items 132 may be found or collected by traveling to different locations in the real world 100 or by completing various actions (such as interacting with virtual elements 130, fighting non-player characters or other players, or completing quests) either in the virtual world 110 or in the real world 100. In the example shown in FIG. 1, a player uses a virtual item 132 to capture one or more virtual elements 130. In particular, a player may deploy a virtual item 132 near the virtual world 110 or within the virtual element 130. In this manner, deploying one or more virtual items 132 may result in the capture of the virtual element 130 for the player or the player's team / faction.
[0017] In one particular implementation, a player may have to collect virtual energy as part of a parallel reality game. The virtual energy 150 may be distributed at different locations in the virtual world 110. A player may collect virtual energy 150 by traveling to (or within a threshold distance of) a location in the real world 100 that corresponds to the location of the virtual energy in the virtual world 110. The virtual energy 150 may be used to power virtual items or to perform various game objectives within the game. A player who loses all virtual energy 150 may be disconnected from the game or may be blocked for a certain amount of time or until additional virtual energy 150 is collected.
[0018] According to aspects of the present disclosure, a parallel reality game may be a massively multiplayer location-based game in which all participants of the game share the same virtual world. Players may be divided into separate teams or factions and may cooperate to achieve one or more game objectives, such as capturing or claiming ownership of virtual elements. In this way, a parallel reality game may essentially be a social game that encourages cooperation between players in the game. Players on opposing teams may play against each other during the parallel reality game (or sometimes cooperate to achieve mutual objectives). Players may use virtual items to attack or impede the progress of players on the opposing team. In some cases, players are encouraged to gather at real-world locations for cooperative or interactive events in the parallel reality game. In these cases, the game server attempts to obtain confirmation that players are actually physically present and are not faking their locations.
[0019] 2 illustrates one embodiment of a game interface 200 that may be presented (e.g., on a player's smartphone) as part of an interface between a player and the virtual world 110. The game interface 200 includes a display window 210 that may be used to display the virtual world 110 and various other aspects of the game, such as the player's position 122 and the positions of the virtual elements 130, virtual items 132, and virtual energy 150 within the virtual world 110. The user interface 200 may also display other information, such as game data information, game communications, player information, client location instructions, and other information associated with the game. For example, the user interface may display player information 215, such as player name, experience level, and other information. The user interface 200 may include a menu 220 for accessing various game settings and other information associated with the game. The user interface 200 may also include a communication interface 230 that enables communication between the game system and the player, and between one or more players of the parallel reality game.
[0020] According to aspects of the present disclosure, a player may interact with a parallel reality game by carrying a client device in the real world. For example, a player may access an application associated with the parallel reality game on a smartphone and play the game by navigating the real world using the smartphone. In this regard, a player does not need to continually see a visual representation of the virtual world on a display screen to play a location-based game. As a result, the user interface 200 may include non-visual elements that allow a user to interact with the game. For example, the game interface may provide audible notifications to the player when the player is approaching a virtual element or object in the game or when a significant event occurs in the parallel reality game. In some embodiments, the player may control these audible notifications using the audio controls 240. Audible notifications may be provided to the user depending on the type of virtual element or event. Other non-visual notifications may increase or decrease in frequency or volume depending on the player's proximity to the virtual element or object. Other non-visual notifications and signals, such as vibration notifications or other suitable notifications or signals, may be provided to the user.
[0021] A parallel reality game may have various features to enhance and encourage gameplay within the parallel reality game. For example, a player may accumulate virtual currency or other virtual rewards (e.g., virtual tokens, virtual points, virtual material resources, etc.) that can be used throughout the game (e.g., to purchase in-game items, to redeem other items, to craft items, etc.). A player may progress through various levels of one or more game objectives that the player achieves during the game. A player may also gain enhanced "powers" or virtual items that can be used to complete game objectives within the game.
[0022] Those skilled in the art will appreciate that numerous game interface configurations and underlying functions are possible using the disclosure provided, and the present disclosure is not intended to be limited to any one particular configuration without explicit and coherent opposition to the contrary.
[0023] Exemplary Gaming System FIG. 3 illustrates one embodiment of a networked computing environment 300. The networked computing environment 300 uses a client-server architecture, with a game server 320 communicating with client devices 310 over a network 370 to provide parallel reality games to players of the client devices 310. The networked computing environment 300 may also include other external systems, such as sponsor / advertiser systems or business systems. Although only one client device 310 is shown in FIG. 3, any number of client devices 310 or other external systems may be connected to the game server 320 over the network 370. Additionally, the networked computing environment 300 may include different or additional elements, and functionality may be distributed between the client devices 310 and their servers 320 in a manner different from that described below.
[0024] The networked computing environment 300 provides for interaction with players in a virtual world that has a parallel geography to the real world. In particular, geographical regions of the real world may be directly linked or mapped to corresponding regions in the virtual world. A player may navigate the virtual world by moving to various geographical locations in the real world. For example, the player's location in the real world may be tracked and used to update the player's location in the virtual world. Typically, the player's location in the real world is determined by finding the location of the client device 310 at which the player is interacting with the virtual world and assuming that the player is in the same (or approximately the same) location. For example, in various embodiments, a player may interact with a virtual element if the player's location in the real world is within a threshold distance (e.g., 10 meters, 20 meters, etc.) of a real-world location that corresponds to the virtual location of the virtual element in the virtual world. For convenience, various embodiments are described with reference to the "location of the player," but one skilled in the art will understand that the reference may also refer to the location of the player's client device 310.
[0025] The client device 310 may be any portable computing device that may be used by a user to connect with the game server 320. For example, the client device 310 is preferably a portable wireless device that may be carried by a player, such as a smartphone, a portable gaming device, an augmented reality (AR) headset, a mobile phone, a tablet, a personal digital assistant (PDA), a navigation system, a handheld GPS system, or other such device. In some examples of use, the client device 310 may be a non-mobile device, such as a desktop or laptop. Additionally, the client device 310 may be a vehicle with an embedded computing device.
[0026] The client device 310 communicates with a game server 320 and provides sensory data of the physical environment. In one embodiment, the client device 310 includes a camera assembly 312, a game module 314, a positioning module 316, and a localization module 318. The client device 310 also includes a network interface (not shown) for providing communications over a network 370. In various embodiments, the client device 310 may include different or additional components, such as additional sensors, displays, software modules, etc.
[0027] The camera assembly 312 includes one or more cameras that may capture image data. The cameras capture image data describing an environmental scene surrounding the client device 310 at a particular pose (position and orientation of the camera within the environment). The camera assembly 312 may use different photosensors with different color capture ranges and different capture rates. Similarly, the camera assembly 312 may include cameras with lenses with different ranges, such as wide-angle or telephoto lenses. The camera assembly 312 may be configured to capture a single image or multiple images as frames of video.
[0028] The client device 310 may include additional sensors that collect data about the environment surrounding the client device, such as motion sensors, accelerometers, gyroscopes, barometers, thermometers, light sensors, microphones, etc. Image data captured by the camera assembly 312 may be accompanied by metadata that describes information other than the image data, such as additional sensor data (e.g., temperature, environmental lighting, air pressure, position, pose, etc.) or imaging data (e.g., exposure time, shutter speed, time of capture, etc.).
[0029] The game module 314 provides an interface for players to participate in the parallel reality game. The game server 320 transmits game data to the client device 310 over the network 370 for use by the game module 314 to provide a local version of the game to players remote from the game server. In one embodiment, the game module 314 presents a user interface on the display of the client device 310 that depicts a virtual world (e.g., renders an image of the virtual world) and allows the user to interact with the virtual world to perform various game objectives. In some embodiments, the game module 314 presents an image of the real world (e.g., captured by the camera assembly 312) augmented with virtual elements from the parallel reality game. In these embodiments, the game module 314 may generate or adjust virtual content according to other information received from other components of the client device 310. For example, the game module 314 may adjust virtual objects displayed on the user interface according to a depth map of the scene captured in the image data.
[0030] The game module 314 may also control various other outputs to allow a player to interact with the game without the player needing to view a display screen. For example, the game module 314 may control various sounds, vibrations, or other notifications that allow a player to play the game without seeing a display screen.
[0031] The positioning module 316 may be any device or circuitry for determining a location of the client device 310. For example, the positioning module 316 may determine an actual or relative location by using a satellite navigation positioning system (e.g., a GPS system, a Galileo positioning system, a Global Navigation Satellite System (GLONASS), a Beidou satellite navigation and positioning system), an inertial navigation system, a dead reckoning system, IP address analysis, triangulation and / or proximity to cellular towers or Wi-Fi hotspots, or other suitable techniques.
[0032] As the player moves with the client device 310 in the real world, the positioning module 316 tracks the player's location and provides player location information to the game module 314. The game module 314 updates the player's location in the virtual world associated with the game based on the player's actual location in the real world. Thus, the player may interact with the virtual world simply by carrying or transporting the client device 310 in the real world. In particular, the player's location in the virtual world may correspond to the player's location in the real world. The game module 314 may provide the player location information to the game server 320 over the network 370. In response, the game server 320 may implement various techniques to verify the client device 310's location and prevent impostors from spoofing their location. It should be understood that the location information associated with the player is utilized only if permission is granted after the player is informed that the player's location information will be accessed and how the location information will be utilized in the context of the game (e.g., to update the player's location in the virtual world). Additionally, location information associated with a player is stored and maintained in a manner that protects the player's privacy.
[0033] The regionalization module 318 provides additional or alternative methods for determining a position of the client device 310. In one embodiment, the regionalization module 318 receives a position determined for the client device 310 by the positioning module 316 and refines it by determining a pose of one or more cameras of the camera assembly 312. The regionalization module 318 may use the position generated by the positioning module 316 to select a 3D map of the environment surrounding the client device 310 and position it relative to the 3D map. The regionalization module 318 may retrieve the 3D map from local storage or from the game server 320. The 3D map may be a point cloud, a mesh, or any other suitable 3D representation of the environment surrounding the client device 310. Alternatively, the regionalization module 318 may determine a position or pose of the client device 310 without reference to a coarse position (such as provided by a GPS system), such as by determining the relative position of the client device 310 with respect to another device.
[0034] In one embodiment, the regionalization module 318 applies the trained model to determine the pose of the image captured by the camera assembly 312 with respect to the 3D map. The regionalization model may thus determine an accurate (e.g., within a few centimeters and degrees) determination of the position and orientation of the client device 310. The position of the client device 310 may then be tracked over time using dead reckoning based on sensor readings, periodic re-regionalization, or a combination of both. Having an accurate pose for the client device 310 may enable the gaming module 314 to present virtual content overlaid on the image of the real world (e.g., by displaying virtual elements in conjunction with a real-time feed from the camera assembly 312 on a display) or on the real world itself (e.g., by displaying virtual elements on a transparent display of an AR headset) in a way that gives the impression that the virtual objects are interacting with the real world. For example, a virtual character may hide behind a real tree, a virtual hat may be placed on a real statue, or a virtual creature may run and hide when a real person quickly approaches it.
[0035] The game server 320 includes one or more computing devices that provide game functionality to the client devices 310. The game server 320 may include or be in communication with a game database 330. The game database 330 stores game data used in the parallel reality game that is provided or provided to the client devices 310 over the network 370.
[0036] The game data stored in the game database 330 may include (1) data associated with a virtual world in a parallel reality game (e.g., image data used to render the virtual world on a display device, geographic coordinates of locations in the virtual world, etc.); (2) data associated with a player of the parallel reality game (e.g., player profile including, but not limited to, player information, player experience level, player currency, current player location in the virtual / real world, player energy level, player preferences, team information, faction information, etc.); (3) data associated with game goals (e.g., data associated with current game goals, game goal status, past game goals, future game goals, desired game goals, etc.); and (4) data associated with virtual elements in the virtual world (e.g., the location of virtual elements, etc.). (5) data associated with real-world objects, landmarks, locations linked to virtual world elements (e.g., real-world object / landmark location, real-world object / landmark description, virtual element linked to real-world object / landmark relationship, etc.); (6) game status (e.g., current player count, current status of game goals); (7) data associated with player actions / inputs (e.g., current player location, past player locations, player movement, player input, player queries, player communications, etc.), or (8) any other data used, related, or obtained during implementation of the parallel reality game. The game data stored in the game database 330 may be pre-populated either offline or in real-time by a system administrator or by data received from users (e.g., players), such as from client devices 310 over the network 370.
[0037] In one embodiment, the game server 320 is configured to receive requests for game data from the client devices 310 (e.g., via remote procedure calls (RPCs)) and respond to those requests over the network 370. The game server 320 may encode the game data in one or more data files and provide the data files to the client devices 310. In addition, the game server 320 is configured to receive game data (e.g., player positions, player actions, player input, etc.) from the client devices 310 over the network 370. The client devices 310 may be configured to periodically send player input and other updates to the game server 320, which the game server uses to update the game data in the game database 330 to reflect any and all changed conditions of the game.
[0038] 3, game server 320 includes a general game module 322, a commercial game module 323, a data collection module 324, an event module 326, a mapping system 327, and a 3D map store 329. As noted above, game server 320 interacts with (e.g., may be game database 330), which may be part of the game server or may be accessed remotely. In other embodiments, game server 320 includes different or additional elements. Furthermore, functionality may be distributed among elements in a manner different from that described.
[0039] The generic game module 322 hosts an instance of a parallel reality game for a set of players (e.g., all players of the parallel reality game) and acts as an authoritative source of the current status of the parallel reality game for the set of players. As a host, the generic game module 322 generates game content for presentation to the players (e.g., via their respective client devices 310). The generic game module 322 may access the game database 330 when hosting a parallel reality game to retrieve or store game data. The generic game module 322 may also receive game data (e.g., depth information, player input, player position, player actions, landmark information, etc.) from the client devices 310 and incorporate the received game data into an overall parallel reality game for the entire set of players of the parallel reality game. The generic game module 322 may also manage the distribution of game data to the client devices 310 over the network 370. In some embodiments, the generic game module 322 also governs security aspects of the client device 310's interaction with the parallel reality game, such as securing connections between the client device and the game server 320, establishing connections between various client devices, or verifying the locations of various client devices 310 to prevent players from cheating by spoofing their locations.
[0040] The generic gaming module 321 may generate and store virtual models of physical spaces. The generic gaming module 321 may generate virtual models based on sensor data or images collected from the client device 310 and may use these virtual models to provide virtual or augmented reality experiences to users. In some cases, the generic gaming module 321 provides the virtual models to third party systems for the third party systems to generate custom virtual content for the virtual models. The generation of the virtual models and the modification of the virtual models by third party systems are described in further detail below.
[0041] The commercial game module 323 may be separate from the general game module 322 or a part of the general game module 322. The commercial game module 323 may manage the inclusion of various game features in the parallel reality game that are linked to commercial activities in the real world. For example, the commercial game module 323 may receive a request from an external system, such as a sponsor / advertiser, a business, or other entity, via the network 370, that includes game features that are linked to commercial activities in the real world. The commercial game module 323 may then arrange to include these game features in the parallel reality game upon verifying that the linked commercial activities have occurred. For example, if a business pays an agreed amount to the provider of the parallel reality game, a virtual object identifying the business may be displayed in the parallel reality game in a virtual location that corresponds to the real world location of the business (e.g., a store or restaurant).
[0042] The data collection module 324 may be separate from the generic game module 322 or a part of the generic game module 322. The data collection module 324 may manage the inclusion of various game features in the parallel reality game that are linked to the data collection activities in the real world. For example, the data collection module 324 may modify the game data stored in the game database 330 to include game features that are linked to the data collection activities in the parallel reality game. The data collection module 324 may also analyze data collected by players pursuant to data collection activities and provide the data for access by various platforms.
[0043] The event module 326 manages player access to events within a parallel reality game. While the term "event" is used for convenience, it should be understood that the term does not necessarily refer to a specific event at a specific location or time. Rather, it may refer to any offering of access-controlled game content in which one or more access criteria are used to determine whether a player can access that content. Such content may be part of a larger parallel reality game that includes game content with little or no access control, or may be a stand-alone access-controlled parallel reality game.
[0044] The mapping system 327 generates a 3D map of a geographical region based on the set of images. The 3D map may be a point cloud, a polygon mesh, or any other suitable representation of the 3D geometry of the geographical region. The 3D map may include semantic labels that provide additional contextual information, such as identifiers of objects such as tables, chairs, clocks, lampposts, trees, etc., materials (concrete, water, bricks, grass, etc.), or game characteristics (e.g., traversable by a character, suitable for a particular in-game action, etc.). In one embodiment, the mapping system 327 stores the 3D map according to any semantic / contextual information in a 3D map store 329. The 3D map may be stored in the 3D map store 329 together with location information (e.g., GPS coordinates of the center of the 3D map, a ring fence defining the extent of the 3D map, etc.). Thus, the game server 320 may provide the 3D map to client devices 310 that provide location data indicating that they are within or near the geographical region covered by the 3D map.
[0045] The network 370 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof. The network may also include a direct connection between the client device 310 and the game server 320. In general, communications between the game server 320 and the client device 310 may be transmitted over a network interface using any type of wired or wireless connection, using a variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML, JSON), or protection schemes (e.g., VPN, Secure HTTP, SSL).
[0046] The content developer system 380 is a third-party system that generates custom virtual content that is presented to users via client devices. The content developer system 380 requests virtual models of physical spaces from the game server 320 and populates those virtual models with custom virtual content. The content developer system 380 and its interaction with the game server are described in further detail below. In some embodiments, the content developer system 380 is integrated into the game server 320 and may be accessed by developers locally or remotely (e.g., via a web interface over the network 370).
[0047] This disclosure refers to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted to and from such systems. Those skilled in the art will recognize that the inherent flexibility of computer-based systems permits a wide variety of possible configurations, combinations, and divisions of tasks and functions between and among components. For example, processes disclosed as being implemented by a server may be implemented using a single server or multiple servers operating in combination. Databases and applications may be implemented in a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.
[0048] In situations where the disclosed systems and methods access and analyze personal information about a user or utilize personal information such as location information, the user may be provided with the opportunity to control whether a program or function collects information and whether or how it receives content from the system or other applications. Such information or data will not be collected or used until the user is provided with meaningful notice of what information will be collected and how that information will be used. Information will not be collected or used unless the user consents, and the user may revoke or change consent at any time. Thus, the user may control how information about the user is collected and used by the application or system. Additionally, certain information or data may be handled in one or more ways before being stored or used such that any personally identifiable information is removed. For example, a user's identifier may be handled such that personally identifiable information cannot be determined for the user.
[0049] Exemplary methods for generating and presenting virtual content from a content development system as an augmented reality experience 4 is an interaction diagram illustrating interactions between client devices, game servers, and content developer systems for generating and displaying custom virtual content, according to some embodiments. Alternative embodiments may include more, fewer, or different steps and interactions, and the illustrated steps and interactions may be performed by or between different systems, or in a different order than illustrated in FIG.
[0050] The game server stores 400 virtual models that represent physical spaces within the physical world. Each virtual model represents in 3D physical objects (e.g., plants and buildings) within the physical space. For example, a virtual model may include a 3D mesh of polygons for each of a set of physical objects in the physical space. Each virtual model also includes a corresponding origin. The origin is a reference point about which virtual objects in the virtual model are placed and oriented using a coordinate system. For example, the position of a virtual object may be quantified as a position relative to the origin of the virtual model. The origin may also include a baseline for orientation within the virtual model, and virtual objects are oriented relative to that baseline orientation. In some embodiments, the origin is a six-dimensional point with three dimensions for position and three dimensions for orientation.
[0051] The game server may generate virtual models of physical spaces that correspond to real-world landmarks, such as buildings, signs, parking lots, or parks. For example, the game server may generate a virtual model of a physical space that surrounds several physical locations that correspond to the landmarks. The game server may also generate virtual models of physical spaces based on larger models of larger physical areas. For example, the game server may store a large virtual model for a larger area, such as a city or neighborhood, and generate sub-models for physical spaces within the larger area.
[0052] In some embodiments, the game server generates a virtual model of the physical space based on crowdsourced data collected from multiple client devices passing through the physical space. As a user uses a client application on a client device, the client device may collect data describing the environment around the client device, such as images or sensor data. The client device transmits the collected data to the game server along with pose data indicating a pose of the client device. The game server uses the collected sensor data and pose data to generate the virtual model of the physical space.
[0053] The content developer system sends a request 402 for a virtual model to the game server. The request may include an identifier for the virtual model stored by the game server. The request may also include the physical space to which the virtual model corresponds. In some embodiments, the game server stores multiple virtual models, one for each of a set of physical spaces. In these embodiments, the request 402 may include an identifier for the set of virtual models for the physical space that the content developer system is requesting. The game server accesses the virtual models requested by the content development system and sends the requested virtual models 404 to the content development system.
[0054] The content developer system modifies the virtual model to include custom virtual content that is displayed to the user 405. This custom virtual content is content that is presented to the user via the client device as part of the augmented reality experience. The custom virtual content includes one or more virtual objects that the content developer system added to the virtual model. These virtual objects are positioned and oriented relative to the origin of the virtual model. In some embodiments, the virtual objects include animation instructions that detail how the virtual objects appear to move, change, or interact with the physical space represented by the virtual model. For example, if the physical space includes a tree, the content developer system may add a virtual object for a monkey and may include animation instructions that cause the monkey to appear to hang from a tree branch.
[0055] The content developer system sends the modified virtual model 406 to the game server, which stores the modified virtual model 408. The game server may store the modified virtual model in association with the physical space that the model represents.
[0056] The client device sends a request 412 to the game server. The request 412 may include an identifier of the modified virtual model, and the game server sends the modified model 414 to the client device. Alternatively, the request 412 may indicate a pose of the client device, and the game server may use the pose of the client device to identify the modified virtual model as a virtual model of the physical space near the client device.
[0057] The client device displays 416 the custom content generated by the content developer system from the modified virtual model. Figure 5 is an example of a flow chart of a method for a client device to display custom content in a virtual model. Alternative embodiments may include more, fewer, or different steps in a different order than shown in Figure 5, and the steps may be performed in a different order than shown in Figure 5. Additionally, the steps may be performed entirely by the client device, although some or all of the steps may be performed by the game server.
[0058] As described above, the client device receives 500 the virtual model from the game server. The client device also receives 510 pose data describing a pose of the client device. The pose data may include sensor data describing the position and orientation of the client device from one or more sensors of the client device (e.g., GPS or IMU data). The pose data may further include images or video feeds from a camera of the client device. In some embodiments, the pose data describes the location of physical objects in an area around the client device. For example, the pose data may include images describing objects in the vicinity of the client device, or may include LIDAR data measuring the distance of objects from the client device.
[0059] The client device identifies a pose for the client device in the real world based on the pose data 520. The pose identified to the client device may be within or near the physical space represented by the received virtual model. As noted above, in alternative embodiments, the client device transmits the pose data to a game server and receives the identified pose from the game server.
[0060] The client device identifies 530 a virtual pose in the received virtual model that corresponds to the identified pose of the client device in the real world. The virtual pose is located and oriented in the virtual model relative to an origin of the virtual model. To identify the virtual pose, the client device may use the received pose data that identifies physical objects near the client device. The client device may compare the identified physical objects to 3D representations of the physical objects stored in the virtual model to determine a 3D representation that corresponds to the physical objects, described as post data. The client device may thereby identify that virtual pose in the virtual model.
[0061] Alternatively, the client device may use a physical-to-virtual pose mapping to identify its virtual pose. For example, a game server may store physical poses in the real world that correspond to origins in the virtual model. The client device may use this mapping of virtual model origins to the real world to transform its physical pose from the physical world to a virtual pose in the virtual model.
[0062] The client device modifies 540 the video feed from the client device's camera based on the virtual pose of the client device and the virtual objects of the custom content in the client device. For example, the client device may render a virtual object based on whether the virtual object is within the virtual field of view of the client device in the virtual model. If the virtual object is within the virtual field of view of the client device, the client device renders the virtual object from the perspective of the virtual pose of the client device in the virtual model. The client device modifies the video feed from the camera to include the rendered virtual object such that the rendered virtual object appears as if it were part of the original video feed.
[0063] The client device displays the modified video feed to the user via a display of the client device 550. The client device may also allow the user to interact with the virtual objects through a user interface on the client device or by tracking the user's movements and detecting when the user "touches" a virtual object. The client device may update animations associated with the virtual objects based on the user's interactions.
[0064] The client device generally does not render 3D representations of physical objects in the virtual model. Because the virtual objects are rendered from the perspective of the virtual pose of the client device in the virtual mode, the virtual objects appear to interact with actual physical objects in the video feed when the video feed is modified to include the virtual objects. In some cases, however, the client device may render 3D representations of physical objects as well. For example, a content developer associated with the content developer system may want to test how a virtual object appears in a physical space without visiting the physical space. To test the virtual object, a user of the client device may select an option for a 3D representation of the physical object in the virtual model to be rendered in the same way. The client device may modify the video feed to include the rendered 3D representation, as described above, or may display only the rendered 3D representation and the virtual object.
[0065] While the above description may primarily describe a game server storing the modified virtual model from a content developer system and providing the modified virtual model to a client device, in some embodiments, the content developer system delivers the modified virtual model directly to a client device without going through a game server. For example, after generating the modified virtual model, the content developer system may send the modified virtual model to the client device for storage.
[0066] 6 illustrates customized virtual content 610 included in a virtual model, according to some embodiments. The virtual model includes 3D representations 600 of physical objects in a physical space. In the illustrated example, the virtual model includes 3D representations 600 of buildings and trees. The virtual content 610 shown is a virtual car that the content developer system added to the virtual model. As mentioned above, the 3D representations 600 and the virtual content 610 are localized within the virtual model relative to the origin of the virtual model.
[0067] Figure 7 illustrates how the virtual content of Figure 6 can be used to modify a video feed, according to some embodiments. The original video feed 700 shows a physical space that corresponds to a virtual model. For example, the video feed 700 shows a physical object whose 3D representation 600 was shown in Figure 6. The client device modifies the video feed to include the virtual content 610. However, the client device does not render a 3D virtual representation of the physical object as it is depicted in the video feed 700.
[0068] Additional Considerations Some portions of the above describe embodiments in terms of algorithmic processes or operations. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. While these operations have been described functionally, computationally, or logically, they will be understood to be implemented by computer programs including instructions for execution by a processor or equivalent electrical circuitry, microcode, or the like. Additionally, and without loss of generality, it is sometimes convenient to refer to these arrangements of functional operations as modules.
[0069] Reference to "one embodiment" or "an embodiment" means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearance of the phrase "in one embodiment" in various places in this specification does not necessarily all refer to the same embodiment. Similarly, the use of "a" or "an" before an element or component is done merely for convenience. The description should be understood to mean that one or more of the element or component are present, unless it is clear that it is otherwise meant.
[0070] When values are described as "about" or "substantially" (or derivatives thereof), such values should be construed to be accurate + / - 10%, unless otherwise clear from the context. For example, "about 10" should be understood to mean "within the range of 9 to 11."
[0071] As used herein, the terms "comprise," "comprising," "including," "including," "having," "having," or other variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, provision, or apparatus that includes a list of elements is not necessarily limited to only those elements, but may include elements such as other processes, methods, provisions, or apparatus that are not expressly listed in or inherent in such process, method, provision, or apparatus. Furthermore, unless expressly stated to the contrary, "or" refers to an inclusive or, not an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or absent), A is false (or absent) and B is true (or present), and both A and B are true (or present).
[0072] Upon reading this disclosure, those skilled in the art will recognize still further alternative structural and functional designs for the systems and processes for providing the described functionality. Thus, while specific embodiments and applications have been illustrated and described, it should be understood that the described subject matter is not limited to the precise structures and components disclosed. The scope of protection should be limited only by the scope of the following claims.
Claims
1. 1. A computer-implemented method comprising: receiving, at a client device, a virtual model of the physical space from an online system; The virtual model is The origin and a three-dimensional representation of a physical object in the physical space, the three-dimensional representation being located within the virtual model relative to the origin; and Virtual objects provided by a third-party system; obtaining pose data for the client device, the pose data including sensor data describing a pose of the client device; obtaining a video feed captured by a camera of the client device; identifying a pose of the client device within the physical space based on the pose data and the feed data; identifying a virtual pose in the virtual model of the client device based on a real pose and a virtual pose, the virtual pose being identified with respect to the origin of the virtual model by mapping a 3D mesh to a position of a physical object with respect to the origin; modifying the video feed to include the virtual object provided by the third party system based on the virtual pose and the virtual model; and and causing the modified video feed to be displayed by the client device.
2. The method of claim 1 , wherein the origin is a six-dimensional point that provides a reference position and orientation for the virtual model.
3. The method of claim 1 , wherein the three-dimensional representation comprises a mesh of polygons.
4. The virtual model further includes animation instructions for the virtual objects, and displaying the modified video feed includes: The method of claim 1 , comprising displaying the virtual object using the animation instructions.
5. The method of claim 1 , wherein the pose data comprises sensor data describing the physical domain.
6. The method of claim 5 , wherein the pose data comprises sensor data describing the physical object in the physical space.
7. Identifying the virtual pose of the client device comprises: The method of claim 6 , comprising mapping the three-dimensional representation of the physical object onto the sensor data describing the physical object.
8. Identifying the virtual pose of the client device comprises: The method of claim 1 , comprising mapping the origin of the virtual model to a physical pose in the physical domain.
9. 2. The method of claim 1, wherein modifying the video feed comprises modifying the video feed to include the virtual object without modifying the video feed to include a three-dimensional representation of the physical object.
10. The method of claim 1 , wherein modifying the video feed comprises modifying the video feed to include the three-dimensional representation of the physical object.
11. A non-transitory computer readable medium having instructions stored thereon that, when executed by a processor, cause the processor to: performing, at the client device, an operation of receiving a virtual model of the physical space from the online system; The virtual model is The origin and a three-dimensional representation of a physical object in the physical space, the three-dimensional representation being located within the virtual model relative to the origin; and Virtual objects provided by a third-party system; obtaining pose data for the client device, the pose data including sensor data describing a pose of the client device; obtaining a video feed captured by a camera of the client device; identifying a pose of the client device within the physical space based on the pose data and the feed data; identifying a virtual pose in the virtual model of the client device based on a real pose and a virtual pose, the virtual pose being identified with respect to the origin of the virtual model by mapping a 3D mesh to a position of a physical object with respect to the origin; modifying the video feed to include the virtual object provided by the third party system based on the virtual pose and the virtual model; and and causing the modified video feed to be displayed by the client device.
12. The computer-readable medium of claim 11 , wherein the origin is a six-dimensional point that provides a reference position and orientation for the virtual model.
13. The computer-readable medium of claim 11 , wherein the three-dimensional representation comprises a mesh of polygons.
14. The virtual model further includes animation instructions for the virtual objects, and displaying the modified video feed includes: The computer-readable medium of claim 11 , further comprising displaying the virtual object using the animation instructions.
15. The computer-readable medium of claim 11 , wherein the pose data comprises sensor data describing the physical area.
16. The computer-readable medium of claim 15 , wherein the pose data comprises sensor data describing the physical object in the physical space.
17. Identifying the virtual pose of the client device comprises: The computer-readable medium of claim 16 , further comprising mapping the three-dimensional representation of the physical object onto the sensor data describing the physical object.
18. Identifying the virtual pose of the client device comprises: The computer-readable medium of claim 11 , further comprising mapping the origin of the virtual model to a physical pose in the physical domain.
19. 12. The computer-readable medium of claim 11, wherein modifying the video feed comprises modifying the video feed to include the virtual object without modifying the video feed to include a three-dimensional representation of the physical object.
20. The computer-readable medium of claim 11 , wherein modifying the video feed comprises modifying the video feed to include the three-dimensional representation of the physical object.