Interactive Augmented and Virtual Reality Experiences

Abstract

To provide augmented reality and virtual reality environments in which a virtual object interacts with an object in an external environment.SOLUTION: A client device which can display various augmented reality and virtual reality images to a user is configured to capture images of an external environment, identify objects in the environment, and display, if an object satisfies a virtual environment criterion, a window to a virtual environment at the location of the object in the environment. The client device includes a light assembly that can generate light. When the light assembly is turned on, virtual objects displayed in the environment gather in a region highlighted by the light from the light assembly. The client device estimates depth information of an object in the environment. The client device displays a virtual object in the environment based on the depth information and may also display the virtual object interacting with an object in the external environment.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The subject matter described relates generally to the display of interactive augmented reality (AR) and virtual reality (VR) images. [Background technology]

[0002] A display device can be used to present information to a user through an immersive experience. This information can include traditional still or video images, as well as augmented reality (AR) or virtual reality (VR) images. AR and VR are useful in a variety of fields, including gaming, engineering, medicine, and aviation. However, AR and VR experiences are typically experienced in isolation from one another. Furthermore, a user's ability to interact with virtual objects (either AR or VR objects) is typically limited to a few gestures or control commands. Summary of the Invention

[0003] The client device can display a variety of interactive augmented reality (AR) and virtual reality (VR) images to the user.

[0004] In some embodiments, the client device takes images of an external environment and identifies real-world objects in the external environment. If the objects meet the criteria of the virtual environment, the client device displays a window into the virtual environment at or near the location of the object in the external environment. The client device then displays a portion of the virtual environment through that window. Thus, a user can explore the virtual environment by moving the client device relative to the object.

[0005] In some embodiments, the client device includes a light assembly capable of generating light, and the client device displays a virtual character in an external environment. When a user turns on the light assembly, the virtual character will gather in an area in the external environment highlighted by the light from the light assembly. Thus, a user can control the movement of the virtual character in the external environment by turning on the light assembly and moving the client device.

[0006] In some embodiments, the client device estimates depth information for objects in the external environment. The client device may display virtual objects in the external environment based on this depth information. The client device may also display virtual objects that interact with objects in the external environment. Thus, the virtual objects may appear more realistic or lifelike to the user. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates a networked computing environment according to one or more embodiments.

[0008] [Diagram 2] FIG. 1 illustrates a representation of a virtual world with a parallel geography to the real world, according to one or more embodiments.

[0009] [Diagram 3] FIG. 1 illustrates an exemplary game interface for a parallel reality game according to one or more embodiments.

[0010] [Figure 4A] 1A-1D are a series of example images illustrating the display of a window in a virtual environment according to some embodiments. [Figure 4B] 1A-1D are a series of example images illustrating the display of a window in a virtual environment according to some embodiments. [Figure 4C] 1A-1D are a series of example images illustrating the display of a window in a virtual environment according to some embodiments. [Figure 4D] 1A-1D are a series of example images illustrating the display of a window in a virtual environment according to some embodiments.

[0011] [Figure 5A] 1A-1C are diagrams illustrating a series of images showing virtual objects coming together in accordance with some embodiments. [Figure 5B] 1A-1C are diagrams illustrating a series of images showing virtual objects coming together in accordance with some embodiments.

[0012] [Figure 6] FIG. 6 illustrates a client device 110 displaying image data of two virtual characters 605 in an external environment, according to one embodiment.

[0013] [Figure 7] FIG. 2 shows a flowchart illustrating a general process for displaying a window into a virtual environment, according to one embodiment.

[0014] [Figure 8] FIG. 2 illustrates a flowchart illustrating a general process for moving a virtual object in an external environment, according to one embodiment.

[0015] [Figure 9] FIG. 2 illustrates a flowchart illustrating a general process for displaying virtual objects in an external environment, according to one embodiment.

[0016] [Figure 10] FIG. 1 illustrates an example computer system suitable for use in training or applying a traversable space model in accordance with one or more embodiments.

[0017] The drawings and the following description illustrate certain embodiments by way of example only. Those skilled in the art will readily appreciate from the following description that alternative embodiments of the structure and method may be employed without departing from the principles described. Reference will now be made to several embodiments, which are illustrated in the accompanying drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] An Exemplary Location-Based Parallel Reality Gaming System Various embodiments are described in the context of a parallel reality game that includes augmented reality (AR) content in a virtual world geography that parallels at least a portion of the real world geography, such that player movements and actions in the real world affect actions in the virtual world and vice versa. Using the disclosure provided herein, one skilled in the art will appreciate that the described subject matter is applicable in other situations in which AR and virtual reality (VR) images are displayed. Furthermore, using the disclosure provided herein, one skilled in the art will appreciate that numerous game interface configurations and underlying functions will become apparent in light of the present disclosure. The present disclosure is not intended to be limited to any one particular configuration.

[0019] The inherent flexibility of computer-based systems also allows for various configurations, combinations, and divisions of tasks and functionality among the components of the system. For example, systems and methods according to aspects of the present disclosure can be implemented using a single computing device or across multiple computing devices (e.g., connected in a computer network).

[0020] FIG. 1 illustrates a networked computing environment 100 according to one or more embodiments. The networked computing environment 100 provides for player interaction in a virtual world that has a parallel geography to the real world. In particular, geographical regions of the real world can be directly linked or mapped to corresponding regions in the virtual world. A player can move around in the virtual world by moving to various geographical locations in the real world. For example, the player's location in the real world can be tracked and used to update the player's location in the virtual world. Typically, a player's location in the real world is determined by finding the location of the client device 110 where the player is interacting with the virtual world and assuming that the player is in the same (or nearly the same) location. For example, in various embodiments, a player can interact with a virtual element if the player's location in the real world is within a threshold distance (e.g., 10 meters, 20 meters, etc.) of a real-world location that corresponds to the virtual location of the virtual element in the virtual world. For convenience, a player who is sufficiently close to a virtual element to interact in this way is said to be in the real-world location that corresponds to the virtual element. Also, although various embodiments are described with reference to the "location of the player," those skilled in the art will recognize that such a reference may also be a reference to the location of the player's client device 110.

[0021] Reference is now made to FIG. 2, which illustrates a conceptual diagram of a virtual world 210 parallel to a real world 200 that may serve as a game board for a player of a parallel reality game according to one embodiment. As shown, the virtual world 210 may include a geography that parallels the geography of the real world 200. In particular, a range of coordinates defining a geographical area or space in the real world 200 is mapped to a corresponding range of coordinates defining a virtual space in the virtual world 210. The range of coordinates in the real world 200 may be associated with a town, neighborhood, city, campus, region, country, continent, entire world, or other geographical area. Each geographic coordinate within the range of geographic coordinates is mapped to a corresponding coordinate in the virtual space of the virtual world.

[0022] The positions of the players in the virtual world 210 correspond to the positions of the players in the real world 200. For example, player A, located at location 212 in the real world 200, has a corresponding location 222 in the virtual world 210. Similarly, player B, located at location 214 in the real world 200, has a corresponding location 224 in the virtual world. As the players move about within a range of geographic coordinates in the real world, they also move about within a range of coordinates that define the virtual space of the virtual world 210. In particular, a positioning system (e.g., a GPS system) associated with a portable computing device carried by the player may be used to track the position of the player as he moves through a range of geographic coordinates in the real world. Data associated with the player's position in the real world 200 is used to update the player's position in the corresponding range of coordinates that define the virtual space of the virtual world 210. In this manner, players can move with continuous tracking in a range of coordinates that define the virtual space of virtual world 210 by simply moving between corresponding ranges of geographic coordinates in real world 200, without having to check in at a particular specific location in real world 200 or periodically update location information.

[0023] A location-based game may include multiple game objectives that require the player to navigate to or interact with various virtual elements or objects distributed at various virtual locations in a virtual world. As described herein, the virtual objects may be AR or VR objects displayed by the client device 110. The virtual objects include animate and inanimate objects. Animate objects may be referred to as virtual characters. The virtual characters may represent characters of the game, such as non-player characters (NPCs). The player may navigate to these virtual locations by navigating to the corresponding locations of the virtual elements or objects in the real world. For example, a positioning system may continuously track the player's location such that as the player continues to navigate the real world, the player continues to navigate the parallel virtual world. The player may then interact with the various virtual elements or objects at a particular location to accomplish or achieve one or more game objectives.

[0024] For example, a game goal has the player interact with virtual elements 230 located at various virtual locations in the virtual world 210. These virtual elements 230 can be linked to landmarks, geographic locations, or objects 240 in the real world 200. The real world landmarks or objects 240 can be works of art, monuments, buildings, businesses, libraries, museums, or other suitable real world landmarks or objects. Interactions can include acquiring, claiming ownership of, or using any virtual items, spending any virtual currency, etc. To acquire these virtual elements 230, the player travels to the landmarks or geographic locations 240 linked to the virtual elements 230 in the real world and interacts with the virtual elements 230 in the virtual world 210. For example, player A in FIG. 2 may have to travel to the landmarks 240 in the real world 200 to interact with or acquire the virtual elements 230 linked to a particular landmark 240. Interacting with the virtual element 230 may require a real-world action, such as taking a photograph and / or verifying, obtaining, or acquiring other information about a landmark or object 240 associated with the virtual element 230. In other embodiments, different or additional mechanisms for obtaining the virtual element may be available. For example, an in-game item may enable a player to interact with the virtual element remotely (e.g., from a real-world location that is not a location corresponding to the virtual element).

[0025] A game objective may require the player to use one or more virtual items collected by the player within the location-based game. For example, the player may travel through the virtual world 210 in search of virtual items (e.g., weapons, creatures, power-ups, or other items) that may be useful in completing the game objective. These virtual items may be found or collected by traveling to different locations in the real world 200 or by completing various actions in either the virtual world 210 or the real world 200. In the example shown in FIG. 2, the player uses the virtual item 232 to acquire one or more virtual elements 230. In particular, the player may deploy the virtual item 232 at a location in the virtual world 210 in proximity to or within the virtual element 230. Deploying one or more virtual items 232 in this manner may result in the acquisition of the virtual element 230 for a particular player or a particular player's team / faction.

[0026] In one particular implementation, a player may collect virtual energy as part of a parallel reality game. As depicted in FIG. 2, virtual energy 250 may be dispersed at different locations in the virtual world 210. A player may collect virtual energy 250 by traveling to the corresponding location of the virtual energy 250 in the real world 200. The virtual energy 250 may be used to power virtual items or perform various game goals within the game. A player who loses all of the virtual energy 250 may be disconnected from the game.

[0027] According to aspects of the present disclosure, a parallel reality game can be a large multiplayer location-based game in which all participants in the game share the same virtual world. Players can be divided into different teams or factions and work together to achieve one or more game goals, such as gaining or claiming ownership of virtual elements. In this way, a parallel reality game can be essentially a social game that encourages cooperation among players of the game. Players of opposing teams can work against each other (or, in some cases, cooperate to achieve each other's objectives) during the parallel reality game. Players may use virtual items to attack or slow the progress of players of the opposing team. In some cases, players are prompted to meet at real-world locations for cooperative or interactive events during the parallel reality game. In such cases, the game server verifies that players are actually physically present and are not impersonating others.

[0028] A parallel reality game can have a variety of features that enhance and facilitate gameplay within the parallel reality game. For example, players can accumulate virtual currency or other virtual rewards (e.g., virtual tokens, virtual points, virtual material resources, etc.) that can be used throughout the game (e.g., to purchase in-game items, exchange for other items, create items, etc.). Players can progress through various levels by completing one or more game objectives and gaining experience within the game. In some embodiments, players can communicate with each other through one or more communication interfaces provided within the game. Players can also gain enhanced "powers" or virtual items that can be used to complete game objectives within the game. Using the disclosure provided herein, one of ordinary skill in the art should understand that various other game features can be included in a parallel reality game without departing from the scope of the present disclosure.

[0029] Referring again to FIG. 1 , the networked computing environment 100 uses a client-server architecture in which a game server 120 communicates with the client devices 110 over a network 105 to provide parallel reality gaming to players of the client devices 110. The networked computing environment 100 may also include other external systems, such as sponsor / advertiser systems or business systems. Although only one client device 110 is shown in FIG. 1 , any number of client devices 110 or other external systems may be connected to the game server 120 over the network 105. Furthermore, the networked computing environment 100 may include different or additional elements, and functionality may be distributed between the client devices 110 and the server 120 in other ways than those described below.

[0030] The client device 110 may be any portable computing device that can be used by a player to interface with the game server 120. For example, the client device 110 may be a wireless device, a personal digital assistant (PDA), a portable gaming device, a mobile phone, a smartphone, a tablet, a navigation system, a handheld GPS system, a wearable computing device, a display having one or more processors, or other similar devices. As another example, the client device 110 includes a conventional computer system, such as a desktop or laptop computer. Furthermore, the client device 110 may be a vehicle equipped with a computing device. In short, the client device 110 may be any computing device or system that allows a player to interact with the game server 120. As a computing device, the client device 110 may include one or more processors and one or more computer-readable storage media. The computer-readable storage media may store instructions that cause the processor to perform operations. The client device 110 is preferably a portable computing device that can be easily carried or transported by a player, such as a smartphone or tablet.

[0031] The client devices 110 communicate with the game server 120 and provide the game server 120 with sensory data of the physical environment. The client devices 110 include a camera assembly 125 that captures two-dimensional image data of a scene in the physical environment in which the client device 110 resides. In the embodiment shown in FIG. 1, each client device 110 includes additional software components, such as a gaming module 135, a positioning module 140, and a light assembly 143. The client devices 110 may include various other input / output devices for receiving information from a player or providing information to a player. Examples of input / output devices include a display screen, a touch screen, a touch pad, data entry keys, a speaker, and a microphone suitable for voice recognition. The client devices 110 may include various other sensors for recording data from the client devices 110, including, but not limited to, motion sensors, accelerometers, gyroscopes, other inertial measurement units (IMUs), barometers, positioning systems, thermometers, light sensors, and the like. Client device 110 may further comprise a network interface that provides communication over network 105. The network interface may comprise any suitable components for interfacing with one or more networks, including, for example, a transmitter, a receiver, a port, a controller, an antenna, or other suitable components.

[0032] The camera assembly 125 captures image data of a scene of an environment in which the client device 110 is located. The camera assembly 125 may utilize a variety of different photosensors with varying capture rates and varying color capture ranges. The camera assembly 125 may include a wide-angle lens or a telephoto lens. The camera assembly 125 may be configured to capture a single image or a video as image data. The camera assembly 125 captures the image data and shares the image data with a computing device on the client device 110. The image data may be accompanied by metadata describing other details of the image data, including sensory data (e.g., temperature, brightness of the environment) or capture data (e.g., exposure, warmth, shutter speed, focal length, capture time, etc.). The camera assembly 125 may include one or more cameras capable of capturing the image data. In one example, the camera assembly 125 includes one camera and is configured to capture monocular image data. In another example, the camera assembly 125 includes two cameras and is configured to capture stereo image data. In various other embodiments, camera assembly 125 includes multiple cameras each configured to capture image data.

[0033] The gaming module 135 provides an interface for players to participate in the parallel reality game. The game server 120 transmits game data to the client device 110 over the network 105 for use by the gaming module 135 in the client device 110 to provide a local version of the game to players located remotely from the game server 120. The game server 120 may include a network interface for providing communication over the network 105. The network interface may include any suitable components for interfacing with one or more networks, including, for example, a transmitter, a receiver, a port, a controller, an antenna, or other suitable components.

[0034] The gaming module 135 executed by the client device 110 provides an interface between the player and the parallel reality game. The gaming module 135 can present a user interface on a display device associated with the client device 110 that displays (e.g., renders a representation of) a virtual world associated with the game, allowing the player to interact with the virtual world and accomplish various game objectives. In some other embodiments, the gaming module 135 presents image data (e.g., captured by the camera assembly 125) of the real world augmented with virtual elements or objects of the parallel reality game. In these embodiments, the gaming module 135 may generate or adjust the virtual content in response to other information received from other components of the client device. For example, the gaming module 135 may adjust virtual objects displayed on the user interface in response to depth information of the scene captured in the image data.

[0035] The gaming module 135 may also control various other outputs to allow the player to interact with the game without viewing a display screen. For example, the gaming module 135 may control various sounds, vibrations, or other notifications to allow the player to play the game without viewing a display screen. The gaming module 135 may access game data received from the game server 120 to provide the player with an accurate representation of the game. The gaming module 135 may receive and process player inputs and provide updates to the game server 120 over the network 105. The gaming module 135 may also generate or adjust game content displayed by the client device 110. For example, the gaming module 135 may generate virtual elements based on information describing a traversable space of a scene (e.g., as determined by the traversable space estimation model 130). For example, the gaming module 135 may determine a path for the virtual elements in the scene over the traversable space.

[0036] The positioning module 140 may be a device or circuitry for monitoring the location of the client device 110. For example, the positioning module 140 may determine an actual or relative location based on an IP address using a satellite navigation positioning system (e.g., a GPS system, a Galileo positioning system, a Global Navigation satellite system (GLONASS), a BeiDou satellite navigation positioning system), an inertial navigation system, a dead reckoning system, using triangulation, proximity to cell towers or WiFi hotspots, or other suitable location determining techniques. The positioning module 140 may further include various other sensors that may assist in accurately determining the location of the client device 110.

[0037] As the player moves around with the client device 110 in the real world, the positioning module 140 tracks the player's location and provides the gaming module 135 with the player's location information. The gaming module 135 updates the player's location in the virtual world associated with the game based on the player's actual location in the real world. In this manner, the player can interact with the virtual world by simply carrying or transporting the client device 110 in the real world. In particular, the player's location in the virtual world can be matched with the player's location in the real world. The gaming module 135 can provide the player's location information to the game server 120 over the network 105. In response, the game server 120 can enact various techniques to verify the location of the client device 110 to prevent cheaters from spoofing the location of the client device 110. It should be understood that the location information associated with the player is only used if authorized after being informed that the player's location information is accessed and how it will be used in the context of the game (e.g., updating the player's location in the virtual world). Additionally, any location information associated with the player is stored and maintained in a manner that protects the player's privacy.

[0038] The game server 120 may be any computing device and may include one or more processors and one or more computer-readable storage media. The computer-readable storage media may store instructions that cause the processor to perform operations. The game server 120 may include or communicate with a game database 115. The game database 115 stores game data used in the parallel reality game that is provided or made available to the clients 110 over the network 105.

[0039] The game data stored in the game database 115 may include (1) data associated with the virtual world in the parallel reality game (e.g., drawing data used to render the virtual world on a display device, geographic coordinates of locations within the virtual world, etc.); (2) data associated with the player of the parallel reality game (e.g., a player profile including, but not limited to, player information, player experience level, player currency, player's current location in the virtual / real world, player energy level, player preferences, team information, faction information, etc.); (3) data associated with game goals (e.g., data associated with current game goals, game goal status, past game goals, future game goals, desired game goals, etc.); and (4) data associated with virtual elements in the virtual world (e.g., location of virtual elements, , type of virtual element, game objective associated with the virtual element, corresponding real world location information for the virtual element, virtual element behavior, virtual element relevance, etc.), (5) data associated with real world objects, landmarks, locations linked to the virtual world elements (e.g., real world object / landmark location, real world object / landmark description, virtual element linked to real world object relevance, etc.), (6) game status (e.g., current player count, current status of game objectives, player leaderboard, etc.), (7) data associated with player actions / inputs (e.g., current player location, past player locations, player movements, player inputs, player queries, player communications, etc.), and (8) any other data used, relevant, or obtained during the implementation of the parallel reality. The game data stored in the game database 115 can be added either offline or in real time by a system administrator or by data received over the network 105 from users / players of the system 100, such as client devices 110.

[0040] The game server 120 can be configured to receive requests for game data from the client devices 110 (e.g., via remote procedure calls (RPCs)) and respond to these requests over the network 105. For example, the game server 120 can encode the game data into one or more data files and provide the data files to the client devices 110. The game server 120 can also be configured to receive game data (e.g., player position, player actions, player input, etc.) from the client devices 110 over the network 105. For example, the client devices 110 can be configured to periodically send player input and other updates to the game server 120, which the game server 120 uses to update the game data in the game database 115 to reflect any and all game state changes.

[0041] In the illustrated embodiment, the server 120 includes a universal game module 145, a commercial game module 150, a data collection module 155, an event module 160, and a traversable space estimation learning system 170. As described above, the game server 120 interacts with a game database 115, which may be part of the game server 120 or may be accessed remotely (e.g., the game database 115 may be a distributed database accessed over the network 105). In other embodiments, the game server 120 includes different or additional components, and functionality may be distributed among the components in a manner different from that described. For example, the game database 115 may be integrated within the game server 120.

[0042] The universal game module 145 hosts the parallel reality game for all players and acts as an authoritative source of information for all players' current status in the parallel reality game. As a host, the universal game module 145 generates game content for presentation to the players, for example, via each client device 110. When the universal game module 145 hosts the parallel reality game, it may access the game database 115 to retrieve or store game data. The universal game module 145 also receives game data (e.g., depth information, player input, player position, player actions, landmark information, etc.) from the client device 110 and incorporates the received game data into the overall parallel reality game for all players of the parallel reality game. The universal game module 145 may also manage the transmission of game data over the network 105 to the client device 110. The universal game module 145 may also manage security aspects of the client devices 110, including, but not limited to, ensuring connections between the client devices 110 and the game server 120, establishing connections between various client devices 110, and verifying the locations of various client devices 110.

[0043] Commercial games module 150, in embodiments where it is included, may be separate from or part of universal games module 145. Commercial games module 150 may manage the inclusion of various game features in a parallel reality game that are linked to commercial activity in the real world. For example, commercial games module 150 may receive a request from an external system, such as a sponsor / advertiser, business, or other entity over network 105 (via a network interface), to include game features linked to commercial activity in the parallel reality game. Commercial games module 150 may then arrange for the inclusion of these game features in the parallel reality game.

[0044] The game server 120 may further include a data collection module 155. The data collection module 155 may be separate from or part of the universal game module 145 in embodiments where it is included. The data collection module 155 may manage the inclusion of various game features in the parallel reality game linked to the data collection activity in the real world. For example, the data collection module 155 may modify the game data stored in the database 115 to include game features in the parallel reality game linked to the data collection activity. The data collection module 155 may analyze data collected by players according to the data collection activity and provide the data for access by various platforms.

[0045] The event module 160 manages player access to events in a parallel reality game. While the term "event" is used for convenience, it should be understood that the term does not necessarily refer to a specific event at a specific location or time. Rather, it may refer to the provision of access-controlled game content that uses one or more access criteria to determine whether a player may access that content. Such content may be part of a larger parallel reality game that includes game content with little or no access control, or may be a stand-alone access-controlled parallel reality game.

[0046] The network 105 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or any combination thereof. The network can also include a direct connection between the client device 110 and the game server 120. Typically, communication between the game server 120 and the client device 110 can be performed over a network interface using any type of wired or wireless connection, using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML, JSON,), or secure methods (e.g., VPN, Secure HTTP, SSL).

[0047] The technology discussed herein refers to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. Those skilled in the art will recognize that the inherent flexibility of computer-based systems allows for the configuration, combination, and division of various tasks among components. For example, the server operations discussed herein may be implemented using a single server or multiple servers operating in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0048] Also, in situations where the systems and methods discussed herein access and analyze or use personal information about a user (e.g., a player), such as location information, the user may be given the opportunity to control whether a program or function collects information and whether or how it receives content from the system or other applications. Such information or data will not be collected or used until the user is provided with meaningful notice of what information is being collected and how it will be used. This information will not be collected or used without the user's consent, which may be revoked or changed by the user at any time. Thus, the user can control how information about the user is collected and used by the application or system. Some information or data may also be handled in one or more ways before storage or use, such that personally identifiable information is excluded. For example, the user's identity may be handled such that personally identifiable information cannot be determined for the user.

[0049] <Example Game Interface> 3 illustrates an embodiment of a game interface 300 that can be presented on the display of the client 110 as part of an interface between the player and the virtual world 210. The game interface 300 includes a display window 310 that can be used to display the virtual world 210 and various other aspects of the game, such as the player's position 222, the position of the virtual elements 230, the position of the virtual items 232, and the position of the virtual energy 250 in the virtual world 210. The user interface 300 can also display other information, such as game data information, game communications, player information, client position assessment instructions, and other information associated with the game. For example, the user interface can display player information 315, such as the player's name, experience level, and other information. The user interface 300 can include a menu 320 for accessing various game settings and other information associated with the game. The user interface 300 can also include a communication interface 330 that enables communication between the game system and the player, and between one or more players of the parallel reality game.

[0050] According to aspects of the present disclosure, a player can interact with a parallel reality game by simply carrying a client device 110 in the real world. For example, a player can play the game by simply accessing an application associated with the parallel reality game on a smartphone and moving around the real world with the smartphone. In this regard, a player does not need to keep looking at a visual representation of the virtual world on a display screen to play a position-based game. As a result, the user interface 300 can include multiple non-visual elements to allow a user to interact with the game. For example, the game interface can provide an auditory notification to the player when the player is approaching a virtual element or object in the game or when an important event occurs in the parallel reality game. The player can control such an auditory notification with audio controls 340. Different types of auditory notifications can be provided to the user depending on the type of virtual element or event. The auditory notification can increase or decrease in frequency or volume depending on the player's proximity to the virtual element or object. Other non-visual notifications and signals can be provided to the user, such as vibration notifications or other suitable notifications or signals.

[0051] 1, in some embodiments, gaming module 135 includes object identifier module 175, window module 180, flashlight module 185, object interaction module 190, and depth estimation module 195. These modules are stored on client device 110, although one or more of these modules may be stored on game server 120. For example, images captured by camera assembly 125 are sent to a module on game server 120, which subsequently provides display instructions to client device 110 over network 105. Additionally, these modules are part of gaming module 135, but are not limited to the gaming context. Modules 175, 180, 185, 190, and 195 are further described below.

[0052] <Displaying a window in the virtual environment> In some embodiments, the image data is presented by the client device 110 to provide a combined or interchangeable AR / VR experience for a user (e.g., a player of a game). A combined AR / VR experience includes displaying AR image data to a user, and in some situations, VR image data is displayed to the user. For example, while displaying an image of an external environment (also referred to as the real world), the gaming module 135 displays a window. In the context of this description, a window is a virtual object that shows the user a previously undisplayed virtual environment (also referred to as a window). A window is typically displayed on an object in the external environment. For example, a window is displayed on a photo frame on a wall. Thus, the user can move the photo frame to see a portion of the virtual environment that appears to be behind the photo frame. Such combined AR / VR experiences are further described below with respect to the object identifier module 175 and the window module 180.

[0053] The object identifier module 175 receives and analyzes images captured by the camera assembly 125 to identify objects in the images. The object identifier module 175 can identify inanimate and animate objects (e.g., animals and people), or parts of animate objects (e.g., hands or feet). Identifying an object refers to recognizing that a group of pixels in an image represents an object and determining the object type of the object. The object type describes what the object is (e.g., person, dog, chair, wall, desk, or lamp). For example, the object identifier module 175 determines that a group of pixels represents a chair. The object identifier module 175 may be configured to identify common objects in a room, such as walls, tables, chairs, pictures, and lights. In some embodiments, the object identifier module 175 uses a machine learning classifier that can identify one or more objects in an input image (or a time series of images). The classifier is trained using training images that include object type labels for pixels (e.g., manually labeled). Alternatively, a rule-based approach may be used that applies rules to the input image to determine whether an object of a given type is depicted.

[0054] The object identifier module 175 may determine whether an identified object meets one or more virtual environment criteria. The virtual environment criteria describe an object or characteristics of an object that is suitable to represent a window into a virtual environment. A suitable object typically has a flat surface facing the external environment, where the surface area of ​​that surface is large enough to overlay a portion of the virtual environment (e.g., a surface area greater than 4 square inches). A suitable object may be wall-mounted. For example, the object is preferably thin, rectangular, and wall-mounted. A suitable object may be easily accessible by a user (e.g., the object is visible, there are no obstacles preventing a user from approaching the object, etc.). Thus, the object identifier module 175 may determine whether an object meets one or more of these virtual environment criteria.

[0055] The virtual environment criteria may be a list of object types that are suitable for representing a window. For example, the virtual environment criteria may identify that power outlets, light switches, picture frames, and computer monitors are suitable objects. Thus, in these embodiments, an object may satisfy the virtual environment criteria if it is assigned (by object identifier module 175) to an object type in the list.

[0056] In some embodiments, instead of identifying the entire object, the object identifier module 175 identifies surfaces of objects in the external environment and determines whether any of the identified surfaces are suitable to represent a window (e.g., whether the surface is flat or has a surface area above a threshold).

[0057] The window module 180 displays (e.g., provides instructions for client device 110 to display) a window into the virtual environment to an object that meets one or more virtual environment criteria. If multiple identified objects meet the virtual environment criteria, the window module 180 may select the object that best meets the criteria. Alternatively, the window module 180 may allow the user to select one of the multiple objects as the object that represents the window. In some embodiments, the window may not be displayed until an instruction is received from the user to view the window or until client device 110 is within a threshold distance of the identified object.

[0058] The window appears at the location of the selected object (e.g., on the outward facing face of the object) or near the location of the selected object (e.g., within 12 inches of the object). The face of the window may be approximately parallel (e.g., within 5 degrees) to the outward facing face of the object. The shape and size of the window may correspond to (e.g., match or be approximately similar to) the shape and size of the object. For example, the window's borders match the outline of the object.

[0059] A window allows a user (using client device 110) to see a portion of the virtual environment through the window. In other words, the portion of the virtual environment is seen from the perspective of someone looking through the window. For example, to view other portions of the virtual environment, the user must move client device 110 relative to the object. In some cases, a window contains a door that the user must open (e.g., by selecting the door) to view the portion of the virtual environment.

[0060] A virtual environment typically includes objects of interest to a user playing a game (e.g., a parallel reality game). For example, the virtual environment includes a virtual character associated with the game. In some cases, a virtual character in the virtual environment interacts with virtual or real objects in an external environment (e.g., a virtual character in an external environment). For example, a virtual character in the virtual environment converses with a user through a window. Information related to the virtual environment, such as the size and shape of the virtual environment and the objects in the virtual environment, may be received from the game server 120. For example, the game server 120 may provide instructions to display a virtual room with four walls, a floor, and a ceiling, and to display a particular virtual object within the room.

[0061] In some embodiments, a user may "enter" the virtual environment, for example, by selecting a window or providing an indication to enter the window. In such embodiments, image data of the virtual environment may be displayed in place of image data of the external environment to indicate that the user has "entered" the virtual environment (in other words, the client device displays VR imagery corresponding to the virtual environment).

[0062] In some embodiments, the window module 180 displays an AR visual effect (e.g., an animated effect) on or around the selected object prior to displaying a window on the object. The visual effect is intended to draw the user's gaze to the object and inform the user that a window will be displayed on the object. The visual effect may move or change color to attract the user's attention. For example, the visual effect changes the color of the object. Other examples of visual effects include smoke, explosions, fireworks, and stars.

[0063] 4A-4D are a series of example images showing the display of a window into a virtual environment, according to some embodiments. FIG. 4A illustrates a display 400 of a client device 405 displaying image data of an external environment 410, according to one embodiment. In the example of FIG. 4A, the external environment 410 is a portion of a room that includes two walls, a floor, a table, and a wall-mounted power outlet 415. Although omitted in FIG. 4A, the image data of the external environment 410 may be augmented with virtual objects.

[0064] 4A-4D, wall outlet 415 is selected to represent a window into the virtual environment. To communicate this to the user, windowing module 180 displays visual effects 420 through wall outlet 420 (see FIG. 4B). In this example, visual effects 420 include stars around outlet 415.

[0065] Figures 4C and 4D show a window 425 into a virtual environment 430 displayed above an outlet 415. The size and shape of the window 425 matches the size and shape of the outlet 415. In Figure 4D, the client device 110 is positioned closer to the outlet 415 so that more of the virtual environment 430 can be seen. The virtual environment 430 includes two virtual characters 430 and a virtual table 440 in a room. The room appears behind the wall on which the outlet 415 is mounted.

[0066] <Interaction with light> As described with reference to FIG. 1, the client device 110 may include a light assembly 143 capable of generating light to illuminate an external environment. For example, the light assembly may generate sufficient light to help a user see portions of the external environment in low-light situations. The light assembly 143 may be used by the camera assembly 125 to illuminate the external environment when capturing images. The light assembly may include one or more light sources, such as light-emitting diodes (LEDs), to generate light. The following description assumes that the light source of the light assembly 143 is positioned such that an image captured by the camera assembly 125 includes an area of ​​the external environment illuminated by the light assembly 143. For example, the light assembly 143 faces in the same direction as the camera assembly 125.

[0067] When the light assembly 143 is producing light, it may be said to be "on." When the light assembly 143 is producing no light or a small amount of light, it may be said to be "off." The light assembly 143 may provide a user interface that allows a user to turn the light assembly 143 on and off. The gaming module 135 may also include a user interface that allows a user to control the light assembly 143 (e.g., so that the user can control the light assembly 143 without leaving the game).

[0068] In some gaming situations or modes, a user may use light from the light assembly 143 to interact with virtual objects displayed in the external environment (or virtual environment). For example, when the light assembly 143 is on, the gaming module 143 causes previously undisplayed virtual objects to appear. In another example, after the light assembly 143 is turned on, virtual characters in the external environment may congregate in an area of ​​the external environment that is illuminated by the light assembly 143. Depending on the game, when the light assembly 143 is turned off, the virtual objects may remain in that area or may begin moving again through the external environment. Thus, by turning on the light assembly 143, a user may control the location of the virtual objects in the external environment by moving the client device 110 (and thereby changing the portion of the environment that is illuminated by the light assembly 143). In this example, an instruction to turn on the light assembly 143 may be referred to as a congregation instruction. In some embodiments, a user may instruct the virtual objects to move to a location in the external environment by selecting an area of ​​the display (e.g., representing the location).

[0069] Flashlight module 185 is communicatively coupled to light assembly 143 to indicate movement of the virtual object (e.g., provide an indication to the display of client device 110). Specifically, in response to light assembly 143 being turned on, flashlight module 185 indicates the virtual object moving from its current position in the external environment to an area of ​​the external environment illuminated by light assembly 143.

[0070] Because the light assembly 143 is typically fixed relative to the camera assembly, the flashlight module 185 does not need to identify light from the light assembly 143 in the image data of the external environment. Instead, the flashlight module 185 may move the virtual object to a region of the external environment being displayed by the client device 110 (referred to as a region of the external environment corresponding to a region of the display). In some cases, the virtual object is moved to a particular region of the display, such as a central region. As the client device 110 is moved with the light assembly 143 on, different regions of the environment are illuminated. If the movement exceeds a threshold (e.g., determined by analyzing signals from the positioning module 140), the flashlight module 185 may display the virtual object moving from its previous location to the new region receiving light.

[0071] In some embodiments, the flashlight module 185 displays a visual effect (e.g., an animated effect) that indicates the area illuminated by the light assembly 143. This may help a user identify the area. Example visual effects include a virtual spotlight or highlight on the area.

[0072] 5A and 5B are a series of example images showing virtual objects coming together, according to some embodiments. FIG. 5A illustrates a client device 110 displaying image data of three virtual characters 505 in an external environment 510, according to one embodiment. The virtual characters 505 have human shapes and are placed throughout the external environment 510. The virtual characters 505 may be placed throughout the external environment 510 for any number of reasons. For example, the game module 135 may generate the virtual characters 505 in random locations in the external environment 510, and the game module 135 may display the virtual characters 505 walking around the external environment.

[0073] While playing the game, it may be desirable for the user to gather the three virtual characters 505 in a region of the external environment 510. To gather the virtual characters 505, the user may turn on the light assembly 143, as described above. Figure 5B illustrates one embodiment of a client device 110 displaying image data including the three virtual characters 505 gathered in a region of the external environment 515 illuminated by the light assembly 143. In the example of Figure 5B, the outline of the illuminated region 515 is virtually highlighted with a visual effect that helps the user identify the region 515.

[0074] <Interaction with virtual objects> Referring to FIG. 1 , the object interaction module 190 displays (e.g., provides instructions for the client device 110 to display) how the virtual object interacts with objects in the external environment. This results in the virtual object appearing more realistic and lifelike to the user. To this end, the object interaction module 190 receives an identified object from the object interaction module 190. By analyzing characteristics of the identified object, such as its shape and surface, the object interaction module 190 can display how the virtual object interacts directly or indirectly with the identified object. Examples of indirect interactions include the virtual character walking around or looking at the object. Examples of direct interactions include the virtual character touching, leaning, standing up, or climbing on the object. For example, the object interaction module 190 displays how the virtual character climbs (e.g., climbs up and down) an object, such as a stick-shaped object or an object with a protruding part or handle. In another example, the object interaction module 190 displays the virtual character sliding down a slope having an angle greater than a threshold amount (e.g., greater than 5 degrees and less than 80 degrees) relative to what the object interaction module 190 determines to be the ground (e.g., a plane perpendicular to gravity or the average orientation of the visible ground in the camera's field of view).

[0075] In some embodiments, the object interaction module 190 displays how the virtual object interacts with the user. For example, the user can "pick up" a virtual object and move it to a different location in the environment. In another example, the object interaction module 190 provides instructions for the virtual object to "follow" the user in the external environment. In this example, the object interaction module 190 can identify the object as a person. The object interaction module 190 can then display how the virtual character moves (e.g., walks) behind the person in the environment. The body components of the person (e.g., legs, torso, and arms) can be identified such that the virtual character can be displayed as walking in step behind the person.

[0076] The depth estimation module 195 estimates the distance from the client device 110 to the object (identified by the object identifier module 175). Based on the estimated distance (also referred to as depth information), the gaming module 135 displays (e.g., provides instructions for the client device 110 to display) the virtual object in the external environment. For example, if a virtual object is displayed near an object, the size of the virtual object may be adjusted based on the estimated distance to the object. In some embodiments, the depth information may cause the virtual object to be displayed partially behind the object. For example, if the depth information indicates that the real-world object is closer than the virtual object is to be displayed, the virtual object may be displayed such that the real-world object obscures a portion of the virtual object. In this manner, the use of the depth information makes the virtual object appear more realistic.

[0077] In some embodiments, the gaming module 135 includes a virtual camera module (not shown) that allows the user to take pictures showing virtual objects in the external environment. In these embodiments, the virtual objects may also be displayed based on depth information.

[0078] The depth estimation module 195 may be built by machine learning. For example, the depth estimation module is trained using training images that contain (e.g., manually labeled) object distance labels for pixels in the game. If the client device has two cameras, the distance may be estimated based on stereo image data. Alternatively, a rule-based approach may be used in which rules are applied to the input image to estimate the distance to an object in the input image.

[0079] FIG. 6 illustrates a client device 110 displaying image data of two virtual characters 605 in an external environment 610, according to one embodiment. Character 605A is displayed as standing on a table in the background of the image, and character 605B is displayed as standing on the ground in the foreground of the image. Although the virtual characters 605 have the same size, the characters 605 are displayed at different sizes in the image so that the characters appear to be in the external environment 610. Notably, because character 605A is displayed in the background and character 605B is displayed in the foreground, character 605A is smaller than character 605B, which can provide a sense of depth. The size of the characters 605 may be based on distance information from the depth estimation module 195. For example, the depth estimation module 195 estimates the distance between the client device and the table and determines the heights of characters 605A and 605B based on this distance.

[0080] <Module Interaction> Although the window module 180, the flashlight module 185, the object interaction module 190, and the depth estimation module 195 are described as independent, they may be used together to enhance a user's gaming experience. For example, each virtual object may be displayed based on depth information from the depth estimation module 195.

[0081] In another example, in some gaming situations, the virtual environment may be darkened such that the user cannot see portions of the virtual environment through a window. To view the virtual environment, the user may virtually light portions of the virtual environment by turning on the light assembly 143 and positioning the client device 110 so that the light appears to propagate through the window into the virtual environment. Similarly, the user may orient a virtual object in the external environment to move through a window into the virtual environment by positioning the client device 110 so that the light from the light assembly appears to propagate through the window into the virtual environment.

[0082] In another example, if a user turns on the light assembly 143 and points a light at their hand by holding their hand in front of the client device 110, a virtual object may move on the user's hand. The user may then move their hand to move the virtual object in the external environment. Additionally, if a window is displayed and the user moves their hand close to an object that represents the window, the gaming module 135 may display the virtual object being placed in the virtual environment through the window.

[0083] <Process that displays a window to the virtual environment> FIG. 7 is a flow chart illustrating a general process 700 for displaying a window into a virtual environment, according to one embodiment. Process 700 may be completed by a client device (e.g., client device 110) having an object identifier module and a window module. The client device may be a mobile device, such as a smartphone, and may have a camera (e.g., camera assembly 125), a light assembly (e.g., light assembly 143), and a display. In some embodiments, the client device implements, at least in part, the parallel reality game described above in FIGS. 1-3. Although the following description is provided in the context of a client device, process 700 (or steps of the process) may be performed on other computing devices (e.g., a server). The steps of process 700 may be performed in a different order, and process 700 may include different or additional steps or fewer steps.

[0084] The client device captures (710) an image of the external environment with a camera assembly. The client device identifies (720) an object in the external environment based on the captured image. The client device determines (730) that the object in the external environment satisfies virtual environment criteria. The client device displays (740) a window into the virtual environment at or near the location of the object in the additional image of the external environment captured by the camera assembly. The client device may display the window in response to determining that the object satisfies the criteria.

[0085] In some embodiments, the client device displays virtual objects in the external environment moving through a window into the virtual environment.

[0086] The client device may receive information about the virtual environment from the server and may use the information about the virtual environment to determine virtual content to display through the window.

[0087] In some embodiments, the process 700 further includes a client device that identifies candidate objects in the external environment. The client device determines whether each of the candidate objects satisfies the virtual environment criteria. The client device selects one of the candidate objects that satisfies the virtual environment criteria as the aforementioned object.

[0088] Process 700 may further include the client device receiving an instruction to display the virtual environment and, in response to receiving the instruction, displaying the virtual environment in place of the image of the external environment.

[0089] Process 700 may further include the client device identifying a second object having a slope in the external environment based on the captured images and displaying a virtual object climbing the second object and sliding down the slope of the second object in additional images of the external environment captured by the camera assembly.

[0090] In some embodiments, determining that an object in the external environment satisfies the virtual environment criteria includes determining that the object is in a wall in the external environment.

[0091] <The process of moving virtual objects> FIG. 8 is a flow chart illustrating a general process 800 for moving a virtual object in an external environment, according to one embodiment. Process 800 may be completed by a client device (e.g., client device 110) having a flashlight module. The client device may be a mobile device, such as a smartphone, and may have a camera (e.g., camera assembly 125), a light assembly (e.g., light assembly 143), and a display. In some embodiments, the client device implements, at least in part, the parallel reality game described above in FIGS. 1-3. Although the following description is described in the context of a client device, process 800 (or steps of the process) may be performed on other computing devices (e.g., a server). The steps of process 800 may be performed in a different order, and process 800 may include different or additional steps or fewer steps.

[0092] The client device captures an image of an external environment with a camera assembly (810). The client device displays virtual objects in the image of the external environment on a display (820). The client device receives an aggregation instruction (e.g., from a user) (830). The client device displays virtual objects in an additional image of the external environment captured by the camera assembly moving to an area of ​​the external environment that corresponds to an area of ​​the display (840). The client device may display the virtual objects moving in response to receiving the aggregation instruction.

[0093] In some embodiments, the set instruction is an instruction to turn on a light assembly of the client device. The area of ​​the external environment may be an area illuminated by light from the light assembly. In some embodiments, the client device displays a visual effect in an area of ​​a display to indicate the area of ​​the external environment illuminated by light from the light assembly.

[0094] In some embodiments, the client device displays a plurality of virtual objects in an image of the external environment, each object being displayed at a different location in the external environment, and the client device may display each of the plurality of virtual objects moving from its location to an area of ​​the external environment.

[0095] In some embodiments, the client device determines that movement of the client device exceeds a threshold In response to determining that movement of the client device exceeds the threshold, the client device displays a virtual object moving from a region of the external environment to a second region of the external environment corresponding to a region of the display.

[0096] Process 800 may further include the client device identifying an object in the external environment having a slope based on the captured image and displaying a virtual object climbing the object and sliding down the slope of the object.

[0097] Although process 800 is described in the context of AR, a similar process may be implemented in the context of VR. For example, a client device may display an image of a VR environment, and in response to receiving a set instruction, the client device may display virtual objects in the VR environment that move to areas of the VR environment that correspond to areas of the display.

[0098] <The process of displaying virtual objects> FIG. 9 is a flow chart illustrating a general process 900 for displaying virtual objects in an external environment, according to one embodiment. Process 900 may be completed by a client device (e.g., client device 110) having a flashlight module. The client device may be a mobile device, such as a smartphone, and may have a camera (e.g., camera assembly 125), a light assembly (e.g., light assembly 143), and a display. In some embodiments, the client device implements the parallel reality game described above in FIGS. 1-3. Although the following description is provided in the context of a client device, process 900 (or steps of the process) may be performed on other computing devices (e.g., a server). The steps of process 900 may be performed in a different order, and process 900 may include different or additional steps or fewer steps.

[0099] A client device captures (910) an image of an external environment with a camera assembly. The client device identifies (920) objects in the external environment based on the captured image. The client device estimates (930) depth information for the objects in the captured image. The client device displays (940) a virtual object based on the identified object and the depth information.

[0100] In some embodiments, the virtual object is a virtual character, and displaying the virtual object includes displaying the virtual character interacting with the object.

[0101] In some embodiments, the object is a human hand.

[0102] Process 900 may further include identifying a slope of an object in the external environment based on the captured image and displaying a virtual object climbing and sliding down the slope of the object.

[0103] <Example computing system> FIG. 10 is an example architecture of a computing device according to one embodiment. Although FIG. 10 depicts a high-level block diagram showing the physical components of a computer that may be used as part or all of one or more of the entities described herein, according to one embodiment, the computer may have additional components, fewer components, or variations of those provided in FIG. 10. Although FIG. 10 depicts a computer 1000, the figure is intended as a functional description of various features that may be present in a computer system, rather than as a structural schematic of the implementations described herein. In fact, and as will be appreciated by those skilled in the art, items shown separately may be combined and some items may be separated.

[0104] 10 shows at least one processor 1002 coupled to a chipset 1004. Also coupled to the chipset 1004 are memory 1006, storage 1008, keyboard 1010, graphics adapter 1012, pointing device 1014, and network adapter 1016. A display 1018 is coupled to the graphics adapter 1012. In one embodiment, the functionality of the chipset 1004 is provided by a memory controller hub 1020 and an I / O hub 1022. In other embodiments, the memory 1006 is coupled directly to the processor 1002 instead of the chipset 1004. In some embodiments, the computer 1000 includes one or more communication buses for interconnecting these components. The one or more communication buses optionally include circuitry (sometimes referred to as a chipset) that interconnects and controls communication between the system components.

[0105] Storage 1008 is any non-transitory computer-readable storage medium, such as a hard drive, a compact disk read-only memory (CD-ROM), a DVD, or solid-state memory device or other optical storage, a magnetic cassette, a magnetic tape, a magnetic disk storage or other magnetic storage device, a magnetic disk storage device, an optical disk storage device, a flash memory device, or other non-volatile solid-state storage device. Such storage 1008 may also be referred to as persistent memory. Pointing device 1014 may be a mouse, a trackball, or other type of pointing device, and is used in combination with keyboard 1010 to input data into computer 1000. Graphics adapter 1012 displays images and other information on display 1018. Network adapter 1016 couples computer 1000 to a local or wide area network.

[0106] The memory 1006 holds instructions and data used by the processor 1002. The memory 1006 can be a non-persistent memory, examples of which include high-speed random access memory such as DRAM, SRAM, DDR RAM, ROM, EEPROM, flash memory, etc.

[0107] As is known in the art, computer 1000 may have other components than those shown in FIG. 10. Additionally, computer 1000 may omit certain components shown. In one embodiment, a computer 1000 acting as a server may omit keyboard 1010, pointing device 1014, graphics adapter 1012, or display 1018. Additionally, storage device 1008 may be local to computer 1000 or remote (e.g., implemented in a storage area network (SAN)).

[0108] As is well known in the art, the computer 1000 is adapted to execute computer program modules for providing the functionality described herein. As used herein, the term "module" refers to computer program logic utilized to provide a particular function. Thus, a module may be implemented in hardware, firmware, or software. In one embodiment, the program module is stored in the storage device 1008, loaded into the memory 1006, and executed by the processor 302.

[0109] <Additional considerations> Some of the above description describes embodiments in terms of algorithmic processes or operations. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. When these operations are described functionally, computationally, or logically, they will be understood to be implemented by computer programs that include instructions executed by a processor or equivalent electrical circuitry, or microcode, or the like. Further, it has been found that it is convenient in some cases, without loss of generality, to refer to these arrangements of functional operations as modules.

[0110] As used herein, any reference to "one embodiment" or "an embodiment" means that a particular element, feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. The appearances of the phrase "one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

[0111] Some embodiments may be described using the terms "coupled" and "connected," along with their derivatives. It should be understood that these terms are not intended as synonyms for each other. For example, some embodiments may be described using the term "connected" to indicate that two or more elements are in direct physical or electrical contact with each other. In other examples, some embodiments may be described using the term "coupled" to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other. These embodiments are not limited in this context.

[0112] As used herein, the terms "comprise," "comprises," "includes," "includes," "has," "having," or any other variation thereof are intended to cover non-exclusive inclusions. For example, a process, method, article, or apparatus comprising a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Furthermore, unless expressly stated to the contrary, "or" refers to an inclusive "or" and not an exclusive "or." For example, a condition A or B is satisfied by either: A being true (or present) and B being false (or absent), A being false (or absent) and B being true (or present), and both A and B being true (or present).

[0113] Furthermore, the use of indefinite articles is employed to describe elements and components of the embodiments. This is done for convenience only to convey the general meaning of the present disclosure. The description should be read to include one or at least one, and the singular also includes the plural unless the contrary meaning is clear. When values ​​are described as "about" or "approximately" (or variations thereof), such values ​​should be interpreted as being precise to ±10%, unless otherwise clear from the context. For example, "about 10" should be understood to mean "in the range of 9 to 11".

[0114] Those skilled in the art will, upon reading this disclosure, appreciate further alternative structural and functional designs for the systems and processes providing the AR functionality of the present disclosure. Thus, while specific embodiments and applications have been shown and described, it should be understood that the subject matter described is not limited to the precise structures and components disclosed herein, and that various modifications, changes, and variations to the arrangement, operation, and details of the methods and apparatus of the present disclosure will be apparent to those skilled in the art. The scope of protection should be limited only by the following claims.

Claims

1. 1. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing device, cause the computing device to perform an operation, the operation comprising: capturing an image of an external environment with a camera assembly; displaying, on a display, a virtual object within the image of the external environment; receiving an instruction via a user input to turn on a flashlight; responsive to receiving the instruction to activate the flashlight, identifying, for each of a plurality of additional images captured by the camera assembly, an area of ​​the display that corresponds to a portion of the external environment that is illuminated by the flashlight; and displaying the virtual object moving toward the area of ​​the display corresponding to the portion of the external environment overlaid on the plurality of additional images of the external environment. A non-transitory computer-readable storage medium.

2. The flashlight is a physical flashlight that illuminates the external environment. The non-transitory computer-readable storage medium of claim 1 .

3. The operation includes: identifying an object having a slope in the external environment based on the captured images; and displaying the virtual object climbing the object and sliding down the slope of the object. The non-transitory computer-readable storage medium of claim 1 .

4. Displaying the virtual object includes displaying a plurality of virtual objects within the image of the external environment, each virtual object being displayed at a different location in the external environment; displaying the virtual objects in the additional images moving towards the portion of the external environment corresponding to the region of the display includes displaying each of the virtual objects moving from a respective location into the region of the external environment. The non-transitory computer-readable storage medium of claim 1 .

5. The operation includes: determining that a movement of a client device associated with the display exceeds a threshold; and in response to determining that the movement of the client device has exceeded the threshold, displaying the virtual object moving from the region of the external environment to a second region of the external environment corresponding to the region of the display. The non-transitory computer-readable storage medium of claim 1 .

6. When a viewpoint of the camera assembly capturing the image moves, the portion of the external environment that is displayed is updated. The non-transitory computer-readable storage medium of claim 1 .

7. The flashlight is a virtual spotlight over the area of ​​the display. The non-transitory computer-readable storage medium of claim 1 .

8. Taking an image of an external environment with a camera assembly; displaying, on a display, a virtual object within the image of the external environment; receiving an instruction via a user input to turn on a flashlight; responsive to receiving the instruction to activate the flashlight, identifying, for each of a plurality of additional images captured by the camera assembly, an area of ​​the display that corresponds to a portion of the external environment that is illuminated by the flashlight; and displaying the virtual object moving toward the area of ​​the display corresponding to the portion of the external environment overlaid on the plurality of additional images of the external environment. method.

9. The flashlight is a physical flashlight that illuminates the external environment. The method according to claim 8.

10. Identifying an object having a slope in the external environment based on the captured image; and displaying the virtual object climbing the object and sliding down the slope of the object. The method according to claim 8.

11. The displaying of the virtual object includes displaying a plurality of virtual objects within the image of the external environment, each virtual object being displayed at a different location in the external environment; displaying the virtual objects in the additional images moving towards the portion of the external environment corresponding to the region of the display includes displaying each of the virtual objects moving from a respective location into the region of the external environment. The method according to claim 8.

12. Determining that movement of a client device associated with the display exceeds a threshold; and in response to determining that the movement of the client device has exceeded the threshold, displaying the virtual object moving from the region of the external environment to a second region of the external environment corresponding to the region of the display. The method according to claim 8.

13. The method of claim 12, wherein the portion of the external environment that is displayed is updated as the viewpoint of the camera assembly that captures the images moves. The method according to claim 8.

14. The flashlight is a virtual spotlight over the area of ​​the display. The method according to claim 8.

15. A computing device; and a computer-readable storage medium storing instructions that, when executed by the computing device, cause the computing device to perform an operation, The operation includes: capturing an image of an external environment with a camera assembly; displaying, on a display, a virtual object within the image of the external environment; receiving an instruction via a user input to turn on a flashlight; responsive to receiving the instruction to activate the flashlight, identifying, for each of a plurality of additional images captured by the camera assembly, an area of ​​the display that corresponds to a portion of the external environment that is illuminated by the flashlight; and displaying the virtual object moving toward the area of ​​the display corresponding to the portion of the external environment overlaid on the plurality of additional images of the external environment. system.

16. The flashlight is a physical flashlight that illuminates the external environment. The system of claim 15.

17. The operation includes: identifying an object having a slope in the external environment based on the captured images; and displaying the virtual object climbing the object and sliding down the slope of the object. The system of claim 15.

18. The displaying of the virtual object includes displaying a plurality of virtual objects within the image of the external environment, each virtual object being displayed at a different location in the external environment; displaying the virtual objects in the additional images moving towards the portion of the external environment corresponding to the region of the display includes displaying each of the virtual objects moving from a respective location into the region of the external environment. The system of claim 15.

19. The operation comprises: determining that a movement of a client device associated with the display exceeds a threshold; and in response to determining that the movement of the client device has exceeded the threshold, displaying the virtual object moving from the region of the external environment to a second region of the external environment corresponding to the region of the display. The system of claim 15.

20. The method of claim 1, wherein the portion of the external environment that is displayed is updated as a viewpoint of the camera assembly that captures the images moves. The system of claim 15.

Images