Controlling augmented reality content by selecting real-world locations or objects

Abstract

The client device and controller enable a user to control the AR content by selecting a real-world location or object. The client receives position data indicating a position and orientation of the controller, where the position data defines an axis of the controller. The client device performs ray casting to determine a location in the 3D map of the real world that intersects with the axis. The client device receives a selection instruction (e.g., a user pressing a button on the controller). Following the selection instruction, the client device selects a location in the 3D map that intersects with the axis as a waypoint. The client device defines a path for the virtual object based on the waypoint.

Description

[Technical field]

[0001] The described subject matter relates generally to augmented reality (AR) content, and more particularly, to a method for controlling the AR content by selecting a real-world location or object. [Background technology]

[0002] This application claims priority to U.S. Provisional Application No. 63 / 290,645, "Controlling Augmented Reality Content By Selecting Real-World Positions or Objects," filed on December 16, 2021, and is incorporated by reference in its entirety herein. Problem Display devices can be used to present information to users through immersive experiences. A wide range of applications use AR technology to present virtual objects in depictions of real-world scenes. However, interaction between virtual objects and real-world objects is typically limited to those that are not present. Furthermore, virtual objects tend to be static once placed in the world. Summary of the Invention

[0003] In light of the above, therefore, there is a need in the art for methods of controlling the movement and other behavior of virtual objects relative to the real-world environment in which they are displayed.

[0004] The client device can provide (e.g., display) various AR experiences in which virtual content is displayed in conjunction with the real world. In various embodiments, the client device displays the virtual content on a display through which the user sees the real world. For example, the client device is an AR headset (e.g., AR glasses or goggles) worn by the user. Thus, from the user's perspective, virtual objects are located in the real world. To enable this, the virtual content is positioned using a 3D map (e.g., a mesh or point cloud) of the real-world environment around the user. The 3D map may use a virtual coordinate system that has a one-to-one relationship with at least a portion of the real world.

[0005] For example, the client device may estimate depth information for pixels of an image captured by one or more cameras on or associated with the client device. The client device may display virtual objects in the external environment based on the depth information. Semantic segmentation may also be used to identify portions of a 3D map that correspond to real-world objects, such as fire hydrants, trees, cars, people, benches, walls, etc. The client device may also display a virtual object interacting with identified objects in the external environment (e.g., walking, climbing, crawling, etc., on one or more of the identified objects in the external environment). Thus, the virtual object may appear more realistic or lifelike to the user.

[0006] A user may control one or more virtual objects, such as a virtual character, by selecting an object or location in the real world. A user holds a controller (e.g., a smartphone or dedicated controller device) that includes one or more position sensors that track the position and / or orientation of the controller. The user points the controller at a real-world location or object and provides a selection indication (e.g., pressing a button or pulling a trigger on the controller, issuing a verbal command, performing a gesture, etc.) to select that object or location. To determine the selected object or location, ray casting may be performed to determine where an axis defined by the controller first intersects with a surface in the 3D map. This intersection point of the 3D map is the selected location or object. The intersection point may be defined as a waypoint. In some embodiments, a crosshair or other indicator is used to show the user the location or object currently being indicated by the controller. For example, if the user is wearing an AR headset, a crosshair may be displayed indicating which location or object in the real world will be selected when the user issues a selection command.

[0007] The path of the virtual object is determined based on the selected object or location. For example, the selection of an object or location is interpreted as an instruction to move a virtual character (or other object) to the selected location or to have the virtual character interact with the selected object. In some embodiments, the user may select multiple points or objects in sequence, and the virtual character may move to or interact with the selected locations / objects in sequence. In other words, the user may define a path or other sequence of interactions using waypoints defined by pointing to corresponding locations in the real world using the controller.

[0008] In some aspects, the technology described herein relates to a method for defining a path for a virtual object in an augmented reality (AR) application, the method including receiving position data indicating a position and orientation of a controller, the position data defining an axis of the controller; performing ray casting to determine a location or object in a 3D map of the real world that intersects with the axis; receiving a selection instruction; following the selection instruction, selecting a location in the 3D map that intersects with the axis as a waypoint; and defining a path based on the waypoint.

[0009] Other aspects include components, devices, systems, improvements, methods, processes, applications, computer readable media, and other techniques related to any of the above. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating a representation of a virtual world having a parallel geography to the real world, according to one embodiment. [Diagram 2] FIG. 2 illustrates an exemplary game interface for a parallel reality game, according to one embodiment. [Diagram 3] FIG. 3 is a block diagram of a networked computing environment suitable for defining a path for a virtual object in an augmented reality (AR) application, according to one embodiment. [Figure 4A] FIG. 4A is a diagram illustrating an example of a user using a controller 400 to control a virtual character, according to various embodiments. [Figure 4B] FIG. 4B is a diagram illustrating an example of a user using the controller 400 to control a virtual character, according to various embodiments. [Figure 4C] FIG. 4C illustrates an example of a user using the controller 400 to control a virtual character, according to various embodiments. [Diagram 5]FIG. 5 is a flow chart illustrating an example method 500 for defining a path of a virtual object in an augmented reality (AR) application, according to one embodiment. [Figure 6] FIG. 6 illustrates an exemplary computer system suitable for use in the networked computing environment of FIG. 1, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The drawings and the following description describe specific embodiments for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods may be employed without departing from the principles described. Wherever practicable, similar or similar reference numbers are used in the figures to indicate similar or similar functionality. When elements share a common numeral followed by another letter, this indicates that the elements are similar or similar. Unless the context dictates otherwise, reference to a numeral alone generally refers to any one or any combination of such elements.

[0012] Various embodiments are described in the context of a parallel reality game that includes augmented reality content in a virtual world geography that parallels at least a portion of the real world geography, such that player movements and actions in the real world affect actions in the virtual world. The subject matter described is applicable to other situations in which it is desirable to control virtual objects in the real world. In addition, the inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among the components of the system. Location-based parallel reality game example FIG. 1 is a conceptual diagram of a virtual world 110 parallel to a real world 100. The virtual world 110 can serve as a game board for players (players may also be referred to as users) of a parallel reality game. As shown, the virtual world 110 includes a geography parallel to that of the real world 100. In particular, a range of coordinates defining a geographical area or space in the real world 100 is mapped to a corresponding range of coordinates defining a virtual space in the virtual world 110. A range of coordinates in the real world 100 can be associated with a town, neighborhood, city, campus, neighborhood, country, continent, the entire globe, or other geographical area. Each geographic coordinate within the range of geographic coordinates is mapped to a corresponding coordinate in a virtual space in the virtual world 110.

[0013] The position of a player in the virtual world 110 corresponds to the position of the player in the real world 100. For example, player A, located at location 112 in the real world 100, has a corresponding location 122 in the virtual world 110. Similarly, player B, located at location 114 in the real world 100, has a corresponding location 124 in the virtual world 110. As a player moves within a range of geographic coordinates in the real world 100, the player also moves within a range of coordinates that define a virtual space in the virtual world 110. In particular, a positioning system (e.g., a GPS system, a positioning system, or both) associated with a mobile computing device carried by the player may be used to track the player's position as the player navigates the range of geographic coordinates in the real world 100. Data associated with the player's position in the real world 100 is used to update the player's position in the corresponding range of coordinates that define the virtual space in the virtual world 110. In this manner, a player may move along a continuous track within a range of coordinates that define a virtual space in the virtual world 110 by simply moving within a corresponding range of geographic coordinates in the real world 100, without checking in or periodically updating location information at a specific individual location in the real world 100.

[0014] Location-based games can include game objectives that require the player to move to or interact with various virtual objects (also referred to as virtual elements) that are scattered at various virtual locations in the virtual world 110. As described herein, the virtual objects can be displayed AR or VR objects. The virtual objects include animated objects and non-animated objects. The animated objects can be referred to as virtual characters. The virtual characters can represent characters of the game, such as playable characters or non-player characters (NPCs). The player can move to these virtual locations by moving to the corresponding locations of the virtual objects in the real world 100. For example, a positioning system can track the player's location such that as the player moves through the real world 100, the player also moves through the parallel virtual world 110. The player can then interact with the various virtual objects at specific locations to accomplish or perform one or more game objectives.

[0015] A game goal may have the player interact with virtual elements 130 located at various virtual locations in the virtual world 110. These virtual elements 130 may be linked to landmarks, geographic locations, or objects 140 in the real world 100. The real world landmarks or objects 140 may be works of art, monuments, buildings, businesses, libraries, museums, or other suitable real world landmarks or objects. Interactions may include acquiring, claiming ownership, using some virtual items, using some virtual currency, etc. To acquire these virtual elements 130, the player travels to the landmark or geographic location 140 linked to the virtual elements 130 in the real world and performs any necessary interactions (as defined by the rules of the game) with the virtual elements 130 in the virtual world 110. For example, player A 112 may travel to a landmark 140 in the real world 100 to interact with or acquire the virtual element 130 linked to that particular landmark 140. Interaction with the virtual element 130 may require an action in the real world, such as taking a photograph or verifying, obtaining, or acquiring other information about a landmark or object 140 associated with the virtual element 130.

[0016] A game objective may require a player to use one or more virtual items that are collected by the player in a location-based game. For example, a player may travel through the virtual world 110 in search of virtual items 132 (e.g., weapons, creatures, power-ups, or other items) that may be useful in achieving a game objective. These virtual items 132 may be found or collected by traveling to different locations in the real world 100 or by completing various actions (such as interacting with virtual elements 130, fighting non-player characters or other players, or completing quests) either in the virtual world 110 or in the real world 100. In the example shown in FIG. 1, a player uses the virtual items 132 to acquire one or more virtual elements 130. In particular, a player may place the virtual items 132 at a location in the virtual world 110 near or within the virtual elements 130. Placing one or more virtual items 132 in this manner may result in the acquisition of the virtual elements 130 for the player or the player's team / faction.

[0017] In one particular implementation, a player may have to collect virtual energy as part of a parallel reality game. The virtual energy 150 may be distributed at different locations within the virtual world 110. A player may collect virtual energy 150 by traveling to (or within a threshold distance of) a location in the real world 100 that corresponds to the location of the virtual energy in the virtual world 110. The virtual energy 150 may be used to power virtual items or perform various game objectives within the game. A player who loses all of the virtual energy 150 may be disconnected from the game or prevented from playing for a certain amount of time or until they collect additional virtual energy 150.

[0018] According to aspects of the present disclosure, a parallel reality game may be a massively multiplayer location-based game in which all participants of the game share the same virtual world. Players may split into separate teams or factions and cooperate to achieve one or more game objectives, such as acquiring or claiming ownership of virtual elements. In this way, a parallel reality game may essentially be a social game that encourages cooperation between players in the game. Players on opposing teams may play against each other (or sometimes cooperate to achieve mutual objectives) during the parallel reality game. Players may use virtual items to attack or impede the progress of players on the opposing team. In some cases, players are encouraged to gather at real-world locations for cooperative or interactive events in the parallel reality game. In these cases, the game server attempts to verify that players are actually physically present and are not faking their locations.

[0019] FIG. 2 illustrates one embodiment of a game interface 200 that may be presented (e.g., on a player's smartphone) as part of an interface between a player and a virtual world 110. The game interface 200 includes a display window 210 that may be used to display the virtual world 110 and various other aspects of the game, such as the player's position 122 and the positions of the virtual elements 130, virtual items 132, and virtual energy 150 within the virtual world 110. The user interface 200 may also display other information, such as game data information, game communications, player information, client location instructions, and other information associated with the game. For example, the user interface may display player information 215, such as player name, experience level, and other information. The user interface 200 may include a menu 220 for accessing various game settings and other information associated with the game. The user interface 200 may also include a communication interface 230 that enables communication between the game system and the player, and between one or more players of the parallel reality game.

[0020] According to aspects of the present disclosure, a player can interact with a parallel reality game by carrying a client device 310 in the real world. For example, a player can play the game by accessing an application associated with the parallel reality game on a smartphone and moving through the real world with the smartphone. In this regard, a player does not need to continually see a visual representation of the virtual world on a display screen to play a location-based game. As a result, the user interface 200 can include non-visual elements that allow a user to interact with the game. For example, the game interface can provide audible notifications to the player when the player is approaching a virtual element or object in the game or when an important event occurs in the parallel reality game. In some embodiments, the player can control these audible notifications using the audio controls 240. Different types of audible notifications can be provided to the user depending on the type of virtual element or event. The audible notifications can increase or decrease in frequency or volume depending on the player's proximity to the virtual element or object. Other non-visual notifications and signals can be provided to the user, such as vibration notifications or other suitable notifications or signals.

[0021] A parallel reality game may have a variety of features that enhance and encourage gameplay within the parallel reality game. For example, a player may accumulate virtual currency or another virtual reward (e.g., virtual tokens, virtual points, virtual material resources, etc.) that can be used throughout the game (e.g., to purchase in-game items, to redeem other items, to create items, etc.). A player may progress through various levels as they accomplish one or more game objectives and gain experience within the game. A player may also gain enhanced "powers" or virtual items that can be used to complete game objectives within the game.

[0022] Those skilled in the art will appreciate that numerous game interface configurations and underlying functionality are possible using the disclosure provided, and the present disclosure is not intended to be limited to any one particular configuration unless expressly stated to the contrary. Exemplary Gaming System FIG. 3 illustrates one embodiment of a networked computing environment 300. The networked computing environment 300 uses a client-server architecture, with a game server 320 communicating with the client devices 310 over a network 370 to provide a parallel reality game to the players of the client devices 310. The networked computing environment 300 may also include other external systems, such as sponsor / advertiser systems or corporate systems. Although only one client device 310 is shown in FIG. 3, any number of client devices 310 or other external systems may be connected to the game server 320 over the network 370. In addition, the client device 310 is shown as being connected to a controller 380, which allows a user to indicate a real-world location or object (and thus a corresponding location in the virtual world), or an object in the real world. It should be appreciated that some or all of the functionality attributed to the client device 310 may be performed by the controller 380. Additionally, the networked computing environment 300 may include different or additional elements, and functionality may be distributed between the client devices 310 and the server 320 in a manner different from that described below.

[0023] The networked computing environment 300 provides for player interaction in a virtual world that has a parallel geography to the real world. In particular, geographical regions of the real world may be directly linked or mapped to corresponding regions in the virtual world. A player may navigate the virtual world by moving to various geographic locations in the real world. For example, the player's location in the real world may be tracked and used to update the player's location in the virtual world. Typically, the player's location in the real world is determined by finding the location of the client device 310 at which the player is interacting with the virtual world and assuming that the player is in the same (or approximately the same) location. For example, in various embodiments, a player may interact with a virtual element if the player's location in the real world is within a threshold distance (e.g., 10 meters, 20 meters, etc.) of a real-world location that corresponds to the virtual location of the virtual element in the virtual world. For convenience, various embodiments are described with reference to the "player's location," but one skilled in the art will understand that such reference may also refer to the location of the player's client device 310.

[0024] The controller 380 may be a smartphone or other computing device, or a dedicated controller. The controller 380 includes one or more sensors that generate position data indicative of the controller's position and / or orientation. The one or more sensors may enable the controller to be tracked along three or more (e.g., six) degrees of freedom. The controller 380 may also include one or more buttons. In one embodiment, a user may select a real-world location or object by pointing the controller at a real-world location or object and issuing a selection command (e.g., by pressing a button, issuing a verbal command, performing a gesture, etc.). Based on the position and orientation of the controller 380, ray casting may be performed to identify a location in the virtual world that corresponds to the real-world object or location at which the controller is pointed. For example, in AR applications, 3D meshes are used in the virtual world that represent the corresponding geometry of the real-world environment. Assuming the mesh is a reasonably accurate representation of the real-world environment, using ray casting to find where the axis of the controller 380 (e.g., the direction it is being pointed) intersects with the mesh identifies the portion of the mesh that corresponds to the real-world location or object at which the user pointed the controller 380. The real-world location or object may be indicated by a crosshair or other indicator displayed in conjunction with the image captured by the camera assembly 312.

[0025] Instead of, or in addition to, a user using the controller 380 to indicate a location or object, the game module 314 performs (e.g., computer vision) hand tracking that allows the user to indicate a location or object. For example, the user performs a hand gesture, such as pointing. In some embodiments, a computer vision module or algorithm is used (e.g., applied to images from the camera assembly 312) to identify the hand gesture. The identified gesture may then be translated into interactivity in a manner similar to the controller 380. For example, after a hand gesture is identified (e.g., the user is pointing), a position and orientation of the gesture may be determined (the position and orientation may define an axis of the gesture). Based on the position and orientation of the gesture, ray casting may be performed to identify a location or object in a 3D map of the real world to which the gesture is pointing (e.g., where the axis of the gesture intersects with the 3D map). In some embodiments, a user pinching their thumb and index finger together is translated into a grabbing mechanism that activates ray casting, and moving the user's hand accordingly moves the virtual object to move it around the virtual world that corresponds to the real world. In this disclosure, descriptions or references to a user using a controller may also be applicable to a user using hand gestures.

[0026] The client device 310 may be any portable computing device that a player can use to interface with the game server 320. For example, the client device 310 is preferably a portable wireless device that can be carried by a player, such as a smartphone, a portable gaming device, an augmented reality (AR) headset, a mobile phone, a tablet, a personal digital assistant (PDA), a navigation system, a handheld GPS system, or other such device. In some use cases, the client device 310 may be a less mobile device, such as a desktop or laptop computer. Additionally, the client device 310 may be a vehicle with an embedded computing device.

[0027] The client device 310 communicates with a game server 320 to provide sensory data of the physical environment. In one embodiment, the client device 310 includes a camera assembly 312, a game module 314, a positioning module 316, and a localization module 318. The game module 314 includes a depth estimation module 365, an object identifier module 375, a waypoint module, and an object control module 390. The client device 310 also includes a network interface (not shown) for providing communications over a network 370. In various embodiments, the client device 310 may include different or additional components, such as additional sensors, displays, and software modules.

[0028] The camera assembly 312 includes one or more cameras capable of capturing image data. The cameras capture image data describing an environmental scene surrounding the client device 310 at a particular pose (position and orientation of the camera within the environment). The camera assembly 312 may use various photosensors having various color capture ranges and various capture rates. Similarly, the camera assembly 312 may include cameras with different lens ranges, such as wide-angle or telephoto lenses. The camera assembly 312 may be configured to capture a single image or multiple images as frames of video.

[0029] The client device 310 may also include additional sensors for collecting data about the environment surrounding the client device, such as motion sensors, accelerometers, gyroscopes, barometers, thermometers, light sensors, microphones, etc. Image data captured by the camera assembly 312 may be enriched with metadata that describes other information about the image data, such as additional sensory data (e.g., temperature, environmental brightness, air pressure, position, pose, etc.) or the capture data (e.g., exposure length, shutter speed, focal length, capture time, etc.).

[0030] The game module 314 provides an interface for players to participate in the parallel reality game. The game server 320 transmits game data to the client device 310 over the network 370 for the game module 314 to use to provide a local version of the game to players located remotely from the game server. In one embodiment, the game module 314 presents a user interface on the display of the client device 310 that depicts a virtual world (e.g., renders an image of the virtual world) and allows the user to interact with the virtual world to perform various game objectives. In some embodiments, the game module 314 presents an image of the real world (e.g., captured by the camera assembly 312) augmented with virtual elements from the parallel reality game. In these embodiments, the game module 314 may generate or adjust virtual content according to other information received from other components of the client device 310. For example, the game module 314 may adjust virtual objects to be displayed on the user interface according to a depth map of the scene captured in the image data.

[0031] The game module 314 may also control various other outputs to allow a player to interact with a game without the player needing to view a display screen. For example, the game module 314 may control various sounds, vibrations, or other notifications that allow a player to play a game without seeing a display screen.

[0032] The game module 314 may access game data received from the game server 320 to provide an accurate representation of the game to the player. The game module 314 may receive and process player input and provide updates to the game server 320 over the network 370. The game module 314 may also generate or adjust game content displayed by the client device 310. For example, the game module 314 may generate virtual elements based on information describing the real world (e.g., a 3D map). For example, the game module 314 may determine a path for the virtual elements in the real world based on the 3D map.

[0033] In one embodiment, a user may use controller 380 to select paths and identify waypoints for virtual elements in the real world, as described above. Additionally or alternatively, a user may cause a virtual element (e.g., a character) to interact with the real world or virtual objects by using the controller to select the virtual element. For example, a user may point to a bench, and a virtual character may be depicted walking toward the bench and sitting on it.

[0034] The positioning module 316 may be any device or circuitry for determining a location of the client device 310. For example, the positioning module 316 may determine an actual or relative location by using a satellite navigation positioning system (e.g., a GPS system, a Galileo positioning system, a Global Navigation Satellite System (GLONASS), a Beidou satellite navigation and positioning system), an inertial navigation system, a dead reckoning system, IP address analysis, triangulation and / or proximity to cellular towers or Wi-Fi hotspots, or other suitable techniques.

[0035] As the player moves with the client device 310 in the real world, the positioning module 316 tracks the player's location and provides the player location information to the game module 314. The game module 314 updates the player's location in the virtual world associated with the game based on the player's actual location in the real world. Thus, the player can interact with the virtual world by simply carrying or transporting the client device 310 in the real world. In particular, the player's location in the virtual world can correspond to the player's location in the real world. The game module 314 can provide the player location information to the game server 320 over the network 370. In response, the game server 320 can implement various techniques to verify the location of the client device 310 to prevent cheaters from impersonating them. It should be appreciated that the location information associated with the player is utilized only if permission is granted after the player is informed that the player's location information will be accessed and how the location information will be utilized in the context of the game (e.g., to update the player's location in the virtual world). Additionally, the location information associated with the player is stored and maintained in a manner that protects the player's privacy.

[0036] The localization module 318 receives the position determined for the client device 310 by the positioning module 316 and refines it by determining a pose of one or more cameras of the camera assembly 312. In one embodiment, the localization module 318 uses the position generated by the positioning module 316 to select a 3D map of the environment surrounding the client device 310. The localization module 318 may retrieve the 3D map from local storage or from the game server 320. The 3D map may be a point cloud, a mesh, or any other suitable 3D representation of the environment surrounding the client device 310.

[0037] In one embodiment, the localization module 318 applies the trained model to determine the pose of the image captured by the camera assembly 312 relative to the 3D map. Thus, the localization model can determine an accurate (e.g., within a few centimeters and a few degrees) determination of the position and orientation of the client device 310. The position of the client device 310 can then be tracked over time using dead reckoning based on sensor readings, periodic relocalization, or a combination of both. Having an accurate pose for the client device 310 may enable the game module 314 to present virtual content overlaid on images of the real world (e.g., by displaying virtual elements in conjunction with a real-time feed from the camera assembly 312 on a display) or overlaid on the real world itself (e.g., by displaying virtual elements on a transparent display of an AR headset) in a way that gives the impression that the virtual objects are interacting with the real world. For example, a virtual character may hide behind a real tree, a virtual hat may be placed on a real statue, or a virtual creature may run and hide if a real person approaches it too quickly.

[0038] The game server 320 includes one or more computing devices that provide game functionality to the client devices 310. The game server 320 may include or communicate with a game database 330. The game database 330 stores game data used in the parallel reality game that is served or provided to the client devices 310 over a network 370.

[0039] The game data stored in the game database 330 may include (1) data associated with the virtual world in the parallel reality game (e.g., image data used to render the virtual world on a display device, geographic coordinates of locations within the virtual world, etc.); (2) data associated with the players of the parallel reality game (e.g., player profile including, but not limited to, player information, player experience level, player currency, current player location in the virtual / real world, player energy level, player preferences, team information, faction information, etc.); (3) data associated with game goals (e.g., data associated with current game goals, game goal status, past game goals, future game goals, desired game goals, etc.); and (4) data associated with virtual elements in the virtual world (e.g., location of virtual elements, location of virtual elements, etc.). The game data may include (e.g., type, game goal associated with the virtual element, corresponding real world location information of the virtual element, behavior of the virtual element, relevance of the virtual element, etc.), (5) data associated with real-world objects, landmarks, locations linked to virtual world elements (e.g., location of real-world object / landmark, description of real-world object / landmark, relevance of virtual element linked to real-world object, etc.), (6) game status (e.g., current number of players, current status of game goal, player leaderboard, etc.), (7) data associated with player actions / inputs (e.g., current player location, past player location, player movement, player input, player query, player communication, etc.), or (8) other data used, related, or obtained during implementation of the parallel reality game. The game data stored in the game database 330 may be entered either offline or in real time by a system administrator or by data received from users (e.g., players) of the system 300, such as from the client device 310 over the network 370.

[0040] In one embodiment, the game server 320 is configured to receive requests for game data from the client devices 310 (e.g., via remote procedure calls (RPCs)) and respond to those requests over the network 370. The game server 320 may encode the game data in one or more data files and provide the data files to the client devices 310. Additionally, the game server 320 may be configured to receive game data (e.g., player positions, player actions, player input, etc.) from the client devices 310 over the network 370. The client devices 310 may be configured to periodically send player input and other updates to the game server 320, which is used to update the game data in the game database 330 to reflect any and all changed conditions of the game.

[0041] 3, the game server 320 includes a universal game module 322, a commercial game module 323, a data collection module 324, an event module 326, a mapping system 327, and a 3D map 328. As described above, the game server 320 interacts with a game database 330, which may be part of the game server or may be accessed remotely (e.g., the game database 330 may be a distributed database accessed over a network 370). In other embodiments, the game server 320 includes different or additional elements. Furthermore, functionality may be distributed among the elements in a manner different than described.

[0042] The universal game module 322 hosts an instance of a parallel reality game for a set of players (e.g., all players of the parallel reality game) and serves as an authoritative source of the current status of the parallel reality game for the set of players. As a host, the universal game module 322 generates game content for presentation to the players (e.g., via their respective client devices 310). When the universal game module 322 hosts a parallel reality game, it may access the game database 330 to retrieve or store game data. The universal game module 322 may also receive game data (e.g., depth information, player input, player position, player actions, landmark information, etc.) from the client devices 310 and incorporate the received game data into an overall parallel reality game for the entire set of players of the parallel reality game. The universal game module 322 may also manage the distribution of game data to the client devices 310 over the network 370. In some embodiments, the universal game module 322 also manages security aspects of the client device 310's interaction with the parallel reality game, such as ensuring connections between the client device and the game server 320, establishing connections between various client devices, or verifying the locations of various client devices 310 to prevent players from spoofing their locations to commit fraud.

[0043] The commercial game module 323 may be separate from or part of the universal game module 322. The commercial game module 323 may manage the incorporation of various game features linked to commercial activities in the real world within the parallel reality game. For example, the commercial game module 323 may receive a request, via the network 370, from an external system, such as a sponsor / advertiser, a business, or other entity, to include game features linked to commercial activities in the real world. The commercial game module 323 may then arrange to include these game features in the parallel reality game upon verifying that the linked commercial activities have occurred. For example, if a business pays an agreed amount to a provider of the parallel reality game, a virtual object identifying the business may be displayed in the parallel reality game at a virtual location corresponding to the business's real world location (e.g., a store or restaurant).

[0044] The data collection module 324 may be separate from or part of the universal game module 322. The data collection module 324 may manage the incorporation of various game features within the parallel reality game that are linked with the data collection activities in the real world. For example, the data collection module 324 may modify the game data stored in the game database 330 to include game features that are linked with the data collection activities within the parallel reality game. The data collection module 324 may also analyze data collected by players according to data collection activities and provide the data for access by various platforms.

[0045] The event module 326 manages player access to events within a parallel reality game. While the term "event" is used for convenience, it should be understood that the term does not necessarily refer to a specific event at a specific location or time. Rather, it may refer to any offering of access-controlled game content in which one or more access criteria are used to determine whether a player can access that content. Such content may be part of a larger parallel reality game that includes game content with little or no access control, or may be a stand-alone access-controlled parallel reality game.

[0046] The mapping system 327 generates a 3D map of the geographic region based on the set of images. The 3D map may be a point cloud, a polygon mesh, or any other suitable representation of the 3D geometry of the geographic region. The 3D map may include semantic labels that provide additional contextual information, such as identifying objects (tables, chairs, clocks, lampposts, trees, etc.), materials (concrete, water, bricks, grass, etc.), or game properties (e.g., moveable by a character, suitable for a particular in-game action, etc.). In one embodiment, the mapping system 327 stores the 3D map along with any semantic / contextual information in a 3D map store 328. The 3D map may be stored in the 3D map store 328 along with location information (e.g., GPS coordinates of the center of the 3D map, a ring fence that defines the extent of the 3D map, etc.). Thus, the game server 320 may provide the 3D map to a client device 310 that provides location data indicating that the client device 310 is within or near the geographic region covered by the 3D map.

[0047] The network 370 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof. The network may also include a direct connection between the client device 310 and the game server 320. In general, communications between the game server 320 and the client device 310 may be performed over a network interface using any type of wired or wireless connection, using a variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML, JSON), or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0048] This disclosure refers to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted to and from such systems. Those skilled in the art will recognize that the inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among components. For example, a process disclosed as being implemented by a server may be implemented using a single server or multiple servers operating in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0049] In situations where the systems and methods disclose access to and analysis of personal information about a user, or use of personal information such as location information, the user may be provided with an opportunity to control whether the program or function collects information and to control whether or how the user receives content from the system or other applications. Such information or data will not be collected or used until the user is provided with meaningful notice of what information will be collected and how that information will be used. Information will not be collected or used unless the user consents, and the user may revoke or change consent at any time. Thus, the user can control how information about the user is collected and used by the application or system. Additionally, certain information or data may be processed in one or more ways such that any personally identifiable information is removed before it is stored or used. For example, a user's identifier may be processed such that personally identifiable information cannot be determined for the user.

[0050] In some embodiments, the game module 314 includes a depth estimation module 365, an object identifier module 375, a waypoint module 385, and an object control module 390. Although these modules are stored on the client device 310 of FIG. 3, one or more of these modules may be stored on the game server 320. For example, images captured by the camera assembly 325 are transmitted to a module on the game server 320, which later provides display instructions to the client device 310 over the network 370. Additionally, while the modules are part of the game module 314, the modules are not limited to a game context. Modules 365, 375, and 390 are described further below.

[0051] The depth estimation module 365 estimates the distance of pixels in an image captured by the camera assembly 312. The distance of a pixel (e.g., each pixel in the captured image) is the estimated distance from the camera assembly 312 to the real-world object corresponding to the pixel. Based on the estimated distance (also referred to as depth information), the game module 314 may display (e.g., provide instructions to the client device 310 for displaying) a virtual object in the external environment. For example, if a virtual object is displayed near an object, the size of the virtual object may be adjusted based on the estimated distance of the object. In some embodiments, the depth information allows the virtual object to be displayed partially behind the object. For example, if the depth information indicates that a real-world object is closer than the location of the virtual object displayed, the virtual object may be displayed such that it appears that the real-world object is obscuring part of the virtual object. Thus, using the depth information may make the virtual object appear more realistic.

[0052] The depth estimation module 365 may be machine-learned. For example, it is trained using training images that include object distance labels for groups of pixels in the images (e.g., manually labeled). If the client device includes two cameras, the distance may be estimated based on stereo image data. Alternatively, a rule-based approach may be used where rules are applied to an input image to estimate the distance of objects in the input image.

[0053] The object identifier module 375 receives and analyzes images captured by the camera assembly 312 to identify real-world objects in the images (e.g., perform semantic segmentation). The object identifier module 375 may identify inanimate and animate objects (e.g., animals and humans), or even parts of animate objects (e.g., hands or legs). Identifying an object may refer to recognizing that a group of pixels in an image represents an object and determining an object type for the object. The object type describes what the object is (e.g., human, dog, chair, wall, desk, or lamp). For example, the object identifier module 375 determines a group of pixels that represents a chair. The object identifier module 375 may be configured to identify common objects in a room, such as walls, tables, chairs, pictures, and lights. In some embodiments, the object identifier module 375 uses or includes a machine learning classifier that can identify one or more types of objects in an input image (or a time series of images). The classifier may be trained using training images that include object type labels for groups of pixels (e.g., manually labeled). In some embodiments, the object identifier module 375 uses or includes multiple machine learning classifiers, each classifier trained to identify one or more different object types. For example, a first classifier is trained to identify trees, a second classifier is trained to identify grass, a third classifier is trained to identify flowers, etc. Additionally or alternatively, a rule-based approach is used to identify real-world objects where rules are applied to the input image to determine whether an object of a given type is depicted.

[0054] In some embodiments, the object identifier module 375 identifies one or more materials and / or textures of an identified object. For example, after a building is identified, the object identifier module 375 determines whether the exterior (or a portion of the exterior) of the building is made of stone, brick, glass, or other material. In another example, the object identifier module 375 determines the material of the ground (e.g., cement, asphalt, grass, snow, or dirt).

[0055] The waypoint module 385 establishes waypoints. A waypoint is a location or object in the real world selected by a user (e.g., using a controller). More specifically, a waypoint is specified on a 3D map of the real world. As previously described, a waypoint may be used to define a path for a virtual object. The waypoint may be displayed to the user. The ability of a user to select a real-world location or object as a waypoint enables many different AR (e.g., game) mechanisms for controlling a virtual object in the real world.

[0056] To establish a waypoint, the following steps may be performed: The waypoint module 385 receives position data from the controller 380. The waypoint module 385 may analyze the position data to determine the position and orientation of the controller 380. Using the determined position and orientation, the waypoint module 385 may determine the axes of the controller 380 (e.g., axes indicating where the controller is pointing). The waypoint module 385 may perform ray casting to determine locations or objects in a 3D map of the real world that intersect with the controller axes. To obtain a 3D map of the surrounding environment, the 3D map may be determined in real time (e.g., by the mapping system 327) based on images captured by the camera assembly 312 or based on images obtained from the 3D map store 328 (e.g., the 3D map was determined beforehand). The waypoint module 385 may also receive a selection indication (e.g., selection data is received from the controller 380 indicating that the user will press a button or pull a trigger). After receiving (e.g., in response to) a selection instruction, the waypoint module 385 selects a location or object in the 3D map that intersects with the axes of the controller 380 as a waypoint (e.g., the location or object to which the controller was pointing when the selection instruction occurred (or within a threshold time of occurrence)).

[0057] The object control module 390 determines how virtual objects interact with their surrounding environment. This may result in the virtual objects appearing more realistic and real to the user. As previously mentioned, in one embodiment, the user may use the controller 380 to issue commands to a virtual character or other virtual object. In various examples, the user may define a path of movement for a virtual object by selecting waypoints, issuing commands to the virtual character to perform certain actions (e.g., jump, duck, dance, etc.), or directly controlling the movement of the virtual object (e.g., using a joystick or directional pad to indicate the direction in which the virtual object should move). The user may also use the controller 380 to direct the virtual object to interact with physical objects or other virtual objects. For example, the user may select a basketball hoop (either real or virtual) to have the virtual character dunk the basketball.

[0058] In some embodiments, the object control module 390 receives the identified object from the object identifier module 375. The object control module 390 may display a virtual object that directly or indirectly interacts with the identified object. Examples of indirect interactions with an identified object in the real world include a virtual object (e.g., a character) walking around or looking at the object. Examples of direct interactions with an identified object in the real world include a virtual object (e.g., a character) touching, leaning on, standing on, or climbing on the object. For example, the object control module 390 displays a virtual character climbing (e.g., up or down) an object, such as a stick-type object or an object with a ledge or handle. In another example, the object control module 390 displays a virtual character sliding down a surface that is inclined at an angle that exceeds a threshold amount (e.g., less than 80 degrees and more than 5 degrees) relative to what the object control module 390 determines to be the ground plane (e.g., a plane perpendicular to gravity or the average orientation of the visible ground in the camera's field of view). How a virtual object interacts with an identified real-world object may depend on characteristics of the identified object (e.g., the identified object's shape, surface, material, texture, or some combination thereof) and characteristics of the virtual object (e.g., the virtual object's movement characteristics).

[0059] In some embodiments, the object control module 390 displays a virtual object that interacts with the user. For example, the user can "pick up" a virtual object and move it to another location in the environment. In another example, the object control module 390 provides instructions for the virtual object to "follow" the user or person in the external environment. In this example, the object control module 390 may identify the object as a human. The object control module 390 may then display a virtual character moving (e.g., walking) behind the human in the environment. The body components of the human (e.g., legs, torso, and arms) may be identified to display the virtual character walking in lockstep behind the human. In the previous example related below, the user may "select" a person in the real world using the controller. The path of the virtual object may then be determined based on the location of the selected person in the real world. If the person moves, the path may be changed or a new path may be created depending on the location of the person and the location of the virtual object.

[0060] The object control module 390 may define a path for the virtual object based on the waypoints. The object control module 390 may also determine how the virtual object moves along the path. The path defines a real-world path for the virtual object to travel. The path may have a start point and an end point. In some cases, the start point of the path may be within a threshold distance of a current position of a virtual object scheduled to move along the path.

[0061] As previously started, a path may be defined based on one or more waypoints. The relationship between waypoints and paths varies depending on the AR application. For example, the start or end point of a path may be defined by a waypoint (e.g., if the selected waypoint is the end point, the virtual object may move from its current position in the real world to the position of the waypoint in the real world). In another example, if multiple waypoints are established, the path may be defined to intersect (or avoid) all waypoints. In another example, if two waypoints are established, the line connecting the two waypoints may indicate the moving direction and moving speed of the virtual object in the real world (e.g., the distance between the waypoints determines the moving speed of the virtual object).

[0062] The path may be determined based on the identified objects in the real world. For example, the object control module 390 defines a path around or between the identified objects in the real world. In another example, the object control module 390 determines whether the virtual character will step over, jump over, climb, or crawl under the real world objects as the character moves along the path. Additionally, the path may be determined based on the shape, surface, material, texture, or some combination thereof, of the identified objects in the real world. For example, a path for a virtual object moving on land may avoid bodies of water (e.g., ponds and fountains). In another example, if the virtual character can climb walls made of stone or brick but not glass, the object control module 390 may define a path that climbs a brick building that avoids the glass windows. In another example, a virtual object may be able to walk on dirt or asphalt but not in snow. Thus, the object control module 390 may form a path for that virtual object that avoids snow. In another example, a virtual object may have a slower speed of movement moving upwards on an inclined surface and a faster speed of movement moving downwards on an inclined surface.

[0063] As suggested by the example in the previous paragraph, a path may be determined based on traversal characteristics of a virtual object scheduled to traverse the path. The traversal characteristics specify the virtual object's ability to traverse terrain in the real world (e.g., jump height, climbing ability, flying or gliding ability, and walking or running speed on different types of terrain). For example, if a virtual character can jump high, a path may be defined such that the virtual character jumps over certain real-world objects instead of walking around them. In another example, if a virtual character can climb objects, a path may be defined such that the character climbs over or around real-world objects instead of walking around them. In another example, if a virtual object has different movement speeds across different ground materials, a path may be defined to avoid the virtual object's slow ground materials.

[0064] In some embodiments, the game module 314 includes a virtual camera module (not shown) that allows the user to take pictures showing virtual objects in the external environment. In these embodiments, the virtual objects may also be displayed based on the depth information.

[0065] 4A-4C show examples of a user using a controller 400 to control a virtual character (controller 400 is an exemplary embodiment of controller 380), according to various embodiments. In FIG. 4A, the user is using controller 400 to select a point in the real world by pointing at controller 380 and clicking a button on controller 380 to indicate a corresponding point in the real world. This allows the user to control the navigation of a virtual object across a surface in the real world using a "point-and-click" interface. Specifically, FIG. 4A shows a user's hand holding a controller at two different points in time (t1 and t2). In the example of FIG. 4A, controller axes 410 and crosshairs are displayed to the user so that the user can determine where the controller is pointing (although this may not be the case in other embodiments). The selected points in the real world are defined as waypoints (indicated in FIG. 4A by a pair of V-shapes, respectively). A pair of V-shapes are displayed to the user so that the user knows the location of the waypoints. As mentioned above, the location of the waypoint in the real world may be determined by performing ray casting to determine a location in a 3D map of the real world that is of interest to the axis 410 of the controller 400. A path 405 is formed according to the defined waypoints. In the example of FIG. 4A, the path 405 is formed to intersect with all the waypoints. However, in another example, the path may be formed to avoid or bypass the waypoints (e.g., at least a threshold distance). A virtual object (e.g., a character) may traverse along the path 405, thus allowing the user to control the movement of the virtual object. In other words, the user can define a path for the virtual object to follow.

[0066] 4B, a user is causing a virtual character 420 to jump, double jump, or even triple jump by selecting a button on the controller 400. For example, a character named "Tiny Peter" may be caused to jump in the air each time the user presses a jump button on the controller 400, flicks the controller vertically, or issues any other input mapped to a jump command.

[0067] 4C , a user guides a finger along touch-sensitive surface 425 to control the movement of virtual character 420. For example, by dragging a finger to the left side of surface 425, virtual character 420 may move to the left. In another example, virtual character 420 may walk while the user touches surface 425 (or another controller button or trigger) and stop walking when the user removes the finger from surface 425. A user may use one or more of these exemplary features to move virtual character 420 along a path (e.g., predetermined using waypoints). Exemplary Methods 5 is a flow chart illustrating an example method 500 for defining a path of a virtual object in an augmented reality (AR) application, according to one embodiment. The steps of FIG. 5 are illustrated from the perspective of a game module (e.g., 314) of a client device (e.g., 310) performing the method 500. However, some or all of the steps may be performed by other entities or components. In addition, some embodiments may perform steps in parallel, in a different order, or perform different steps. The steps of the method 500 may be stored on a non-transitory computer-readable medium.

[0068] In step 510, the game module receives position data indicating a position and orientation of the controller, the position data defining an axis of the controller.

[0069] In step 520, the game module performs a raycast to determine a location or object in the 3D map of the real world that intersects with the axis.

[0070] In step 530, the game module receives an indication of the selection.

[0071] In step 540, the game module, following (eg, in response to) the selection instruction, selects a location in the 3D map that intersects with the axis as a waypoint.

[0072] In step 550, the game module defines a path based on the waypoint (e.g., the location of the waypoint in the 3D map). The waypoint may define a start point or an end point of the path. If the waypoint is not the start point, the start point of the path may be based on the location of a virtual object in the 3D map.

[0073] In some embodiments, the method 500 further includes a game module that identifies one or more real-world objects in the 3D map, and the defining the path is further based on the identified one or more real-world objects. Further, the game module may identify a material of the identified real-world object, and the defining the path is further based on the identified material.

[0074] In some aspects, the method 500 further includes a game module that provides (e.g., to a client device) AR data including a virtual object traversing along the path. A speed of movement of the virtual object along the path may be based on a material of a real-world object along the path. Additionally or alternatively, the AR data includes the virtual object interacting with the real-world object within or within a threshold distance of the waypoint.

[0075] In some aspects, the method 500 further includes a game module receiving a traversal characteristic of a virtual object that traverses the path, where defining the path is further based on the traversal characteristic of the virtual object.

[0076] In some aspects, the method 500 further includes a game module that selects a plurality of waypoints after (e.g., in response to) receiving a plurality of selection indicators, and a path is defined based on the plurality of waypoints. The path may be defined to intersect with each of the plurality of waypoints. Examples of Computing Systems 6 is a block diagram of an exemplary computer 600 suitable for use as a client device 310 or a game server 320. The exemplary computer 600 includes at least one processor 602 coupled to a chipset 604. The chipset 604 includes a memory controller hub 620 and an input / output (I / O) controller hub 622. A memory 606 and a graphics adapter 612 are coupled to the memory controller hub 620, and a display 618 is coupled to the graphics adapter 612. A storage device 608, a keyboard 610, a pointing device 614, and a network adapter 616 are coupled to the I / O controller hub 622. Other embodiments of the computer 600 have different architectures.

[0077] 6, storage device 608 is a non-transitory computer-readable storage medium such as a hard drive, a compact disc read-only memory (CD-ROM), a DVD, or a solid-state memory device. Memory 606 holds instructions and data used by processor 602. Pointing device 614 is a mouse, trackball, touch screen, or other type of pointing device and is used in combination with keyboard 610 (which may be an on-screen keyboard) to input data into computer system 600. Graphics adapter 612 displays images and other information on display 618. Network adapter 616 couples computer system 600 to one or more computer networks, such as network 370.

[0078] The types of computers used by the entities of Figure 3 may vary depending on the implementation and the processing power required by the entities. For example, game server 320 may include multiple blade servers operating together to provide the described functionality. Additionally, the computers may lack some of the components discussed above, such as keyboard 610, graphics adapter 612, and display 618. Additional Considerations Some portions of the above description describe embodiments in terms of algorithmic processes or operations. These algorithmic descriptions and representations are commonly used by those skilled in the computing arts to effectively convey the substance of their work to others skilled in the art. While these operations have been described functionally, computationally, or logically, they will be understood to be implemented by computer programs including instructions for execution by a processor or equivalent electrical circuitry, microcode, or the like. Additionally, and without loss of generality, it may sometimes be convenient to refer to these arrangements of functional operations as modules.

[0079] Reference to "one embodiment" or "embodiment" means that a particular element, feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. In various places in this specification, the appearances of the phrase "in one embodiment" do not necessarily all refer to the same embodiment. Similarly, the use of "a" or "an" before an element or component is done merely for convenience. The description should be understood to mean that one or more of the element or component are present, unless it is clear that something else is meant.

[0080] When values ​​are described as "about" or "substantially" (or derivatives thereof), such values ​​should be construed to be accurate to + / - 10%, unless otherwise clear from the context. From the example, "about 10" should be understood to mean "in the range of 9 to 11."

[0081] The words "comprise," "comprises," "includes," "including," "has," "having," or any other variations thereof, are intended to include a non-exclusive inclusion. For example, a process, method, article, or apparatus comprising a list of elements is not necessarily limited to only those elements and may include other elements not expressly listed or inherent in such process, method, article, or apparatus. Furthermore, unless expressly stated to the contrary, "or" means an inclusive "or" and not an exclusive "or." For example, a condition A or B is satisfied by any one of the following: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and both A and B are true (or exists).

[0082] Upon reading this disclosure, those skilled in the art will recognize still further alternative structural and functional designs for the systems and processes for providing the described functionality. Thus, while specific embodiments and applications have been illustrated and described, it should be understood that the described subject matter is not limited to the precise structures and components disclosed. The scope of protection should be limited only by the scope of the following claims.

Claims

1. 1. A method for defining a path of a virtual object in an augmented reality (AR) application, the method comprising: receiving position data indicative of a position and orientation of a controller, the position data defining axes of the controller; performing ray casting to determine locations in a 3D map of the real world that intersect with said axis; receiving a selection indication; Following said selection indication, selecting the location within the 3D map that intersects with the axis as a waypoint; defining the route based on the waypoints; A method comprising:

2. The method of claim 1 , further comprising identifying one or more real-world objects in the 3D map, and wherein defining the route is further based on the identified one or more real-world objects.

3. The method of claim 2 , further comprising identifying a material of an identified real-world object, and wherein defining the path is further based on the identified material.

4. The method of claim 1 , further comprising providing AR data to a client device including the virtual object traversed along the path.

5. The method of claim 4 , wherein a speed of movement of the virtual object along the path is based on a material of a real-world object along the path.

6. The method of claim 1 , further comprising receiving a traversal characteristic of the virtual object that traverses the path, and wherein defining the path is further based on the traversal characteristic of the virtual object.

7. The method of claim 1 , wherein the waypoint defines an end point of the path.

8. The method of claim 7 , wherein a starting point of the path is based on a position of the virtual object in the 3D map.

9. The method of claim 1 , wherein the waypoint defines a start of the path.

10. The method of claim 1 , further comprising providing AR data to a client device including the virtual object interacting with a real-world object at the waypoint.

11. The method of claim 1 , wherein a plurality of waypoints are selected after receiving a plurality of selection indications, and the path is defined based on the plurality of waypoints.

12. The method of claim 11 , wherein the path is defined to intersect with each of the plurality of waypoints.

13. A non-transitory computer-readable medium having stored thereon instructions that, when executed by a computing system, cause the computing system to perform operations, the operations including: receiving position data indicative of a position and orientation of a controller, the position data defining an axis of the controller; performing ray casting to determine locations in a 3D map of the real world that intersect with said axis; Receiving a selection indication; Following the selection indication, selecting the location within the 3D map that intersects with the axis as a waypoint; defining a route based on the waypoints; A non-transitory computer readable medium comprising:

14. 14. The non-transitory computer-readable medium of claim 13, wherein the operations further include identifying one or more real-world objects in the 3D map, and wherein defining the path is further based on the identified one or more real-world objects.

15. 15. The non-transitory computer-readable medium of claim 14, wherein the operations further include identifying a material of the identified object, and wherein defining the path is further based on the identified material.

16. The non-transitory computer-readable medium of claim 13 , wherein the operations further comprise providing AR data to a client device that includes a virtual object traversed along the path.

17. 20. The non-transitory computer-readable medium of claim 16, wherein a speed of movement of the virtual object along the path is based on a material of a real-world object along the path.

18. 14. The non-transitory computer-readable medium of claim 13, wherein the operations further include receiving a traversal characteristic of a virtual object that traverses the path, and wherein defining the path is further based on the traversal characteristic of the virtual object.

19. The non-transitory computer-readable medium of claim 13 , wherein the waypoint defines an end point of the path.

20. 1. A method for defining a path of a virtual object in an augmented reality (AR) application, the method comprising: receiving position data indicative of a position and orientation of a controller, the position data defining axes of the controller; performing ray casting to determine objects in a 3D map of the real world that intersect with said axis; receiving a selection indication; in response to the selection indication, selecting the object in the 3D map that intersects with the axis as a waypoint; defining the route based on the waypoints; A method comprising:

Images