Interactable augmented and virtual reality experience

Abstract

To present information to a user through an immersive experience including augmented reality or virtual reality using a display device.SOLUTION: A client device for displaying various augmented reality and virtual reality images to a user, wherein the client device captures images of an external environment, identifies objects in the environment, and, if an object satisfies a virtual environment criterion, displays a window to a virtual environment at the location of the object in the environment. The client device also includes a light assembly capable of generating light, and when the user turns on the light assembly, virtual objects displayed in the environment gather to a region highlighted by the light from the light assembly. The client device further estimates depth information of an object in the environment, and, based on the depth information, displays a virtual object in the environment and displays the virtual object interacting with an object in the external environment.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The described subject matter relates generally to the display of interactive augmented reality (AR) and virtual reality (VR) images.

Background Art

[0002] A display device can be used to present information to a user through an immersive experience. This information can include not only conventional still images or videos, but also augmented reality (AR) or virtual reality (VR) images. AR and VR are useful in a variety of fields, including gaming, engineering, medicine, and aviation. However, AR and VR experiences are generally experienced separately from each other. Furthermore, a user's ability to interact with virtual objects (objects in either AR or VR) is typically limited to a few gestures or control commands.

Summary of the Invention

[0003] A client device can display various interactive augmented reality (AR) and virtual reality (VR) images to a user.

[0004] In some embodiments, the client device captures an image of an external environment and identifies real-world objects in the external environment. If an object meets the criteria of a virtual environment, the client device displays a window to the virtual environment at or near the location of the object in the external environment. The client device then displays a portion of the virtual environment through that window. Thus, the user can explore the virtual environment by moving the client device relative to the object.

[0005] In some embodiments, the client device includes a light assembly capable of generating light, and the client device displays a virtual character in an external environment. When the user turns on the light assembly, the virtual character will gather in the area within the external environment highlighted by the light from the light assembly. Thus, the user can control the movement of the virtual character within the external environment by turning on the light assembly and moving the client device.

[0006] In some embodiments, the client device estimates the depth information of an object within the external environment. Based on this depth information, the client device may display a virtual object in the external environment. The client device may also display a virtual object that interacts with the object in the external environment. Thus, the virtual object will appear more realistic or alive to the user.

Brief Description of the Drawings

[0007]

Figure 1

[0008]

Figure 2

[0009]

Figure 3

[0010]

Figure 4A

Figure 4B

Figure 4C

Figure 4D

[0011]

Figure 5A

Figure 5B

[0012]

Figure 6

[0013]

Figure 7

[0014]

Figure 8

[0015]

Figure 9

[0016]

Figure 10

[0017] The drawings and the following description are illustrative of specific embodiments only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods may be employed without departing from the described principles. Reference will now be made to several embodiments, illustrated in the accompanying drawings.

Embodiments for Carrying Out the Invention

[0018] <Exemplary Location-Based Parallel Reality Gaming System> In the context of a parallel reality game that includes augmented reality (AR) content in a virtual world geography that is parallel to at least a portion of the real-world geography, such that a player's movements and actions in the real world affect actions in the virtual world and vice versa, various embodiments will be described. Those skilled in the art will understand that the described subject matter is applicable in other situations where AR and virtual reality (VR) images are displayed, using the disclosure provided herein. Further, those skilled in the art will recognize, using the disclosure provided herein, that the configurations and underlying functions of a number of game interfaces will become apparent in light of this disclosure. The present disclosure is not intended to be limited to any one particular configuration.

[0019] Also, due to the inherent flexibility of computer-based systems, various configurations, combinations, and divisions of tasks and functions are possible among the components of the system. For example, the systems and methods according to aspects of the present disclosure can be implemented using a single computing device or across multiple computing devices (e.g., connected in a computer network).

[0020] FIG. 1 shows a networked computing environment 100 according to one or more embodiments. The networked computing environment 100 provides for player interaction in a virtual world having a geography parallel to the real world. In particular, real-world geographic regions can be directly linked or mapped to corresponding regions in the virtual world. Players can move around within the virtual world by moving to various geographic locations in the real world. For example, it can be used to track a player's position in the real world and update the player's position in the virtual world. Typically, the position of the player's client device 110 while interacting with the virtual world is discovered, and the position of the player in the real world is determined by assuming that the player is at the same (or substantially the same) location. For example, in various embodiments, if the position of the player in the real world is within a threshold distance (e.g., 10 meters, 20 meters, etc.) from the real-world position corresponding to the virtual position of a virtual element in the virtual world, the player can interact with the virtual element. For convenience, a player who is sufficiently close to such a virtual element to interact in this way is said to be at the real-world position corresponding to the virtual element. Also, although various embodiments are described with reference to "player position", those skilled in the art will recognize that such reference may also refer to the position of the player's client device 110.

[0021] Referring now to FIG. 2, it depicts a conceptual diagram of a virtual world 210 parallel to the real world 200 that can function as a game board for a player of a parallel reality game according to one embodiment. As shown, the virtual world 210 can include a geography parallel to the geography of the real world 200. In particular, the range of coordinates defining a geographic region or space in the real world 200 is mapped to a corresponding range of coordinates defining a virtual space in the virtual world 210. The range of coordinates in the real world 200 can be associated with a town, district, city, campus, region, country, continent, the world, or other geographic regions. Each geographic coordinate within the range of geographic coordinates is mapped to a corresponding coordinate in the virtual space of the virtual world.

[0022] The position of a player in the virtual world 210 corresponds to the position of the player in the real world 200. For example, player A located at position 212 in the real world 200 has a corresponding position 222 in the virtual world 210. Similarly, player B located at position 214 in the real world 200 has a corresponding position 224 in the virtual world. As the players move around within the geographical coordinate range of the real world, the players will also move around within the coordinate range that defines the virtual space of the virtual world 210. In particular, a positioning system (e.g., GPS system) associated with a portable computing device carried by the player can be used to track the player's position as the player moves within the geographical coordinate range in the real world. Using the data associated with the position of the player in the real world 200, the position of the player in the corresponding range of coordinates that define the virtual space of the virtual world 210 is updated. In this way, the players can move continuously with tracking within the range of coordinates that define the virtual space of the virtual world 210 simply by moving between the corresponding ranges of geographical coordinates in the real world 200 without the need to check in at a specific concrete position in the real world 200 or update the position information regularly.

[0023] A location-based game can include multiple game objectives that require a player to move to or interact with various virtual elements or virtual objects distributed at various virtual locations in a virtual world. As described herein, the virtual objects can be AR or VR objects displayed by the client device 110. The virtual objects include living objects and inanimate objects. The living objects may be referred to as virtual characters. The virtual characters can represent game characters such as non-player characters (NPCs). The player can move to these virtual locations by moving to the corresponding locations of the virtual elements or objects in the real world. For example, the positioning system can continuously track the player's position so that when the player continuously moves in the real world, the virtual world that the player continuously parallels also moves. And the player can interact with various virtual elements or objects at a specific location and achieve or implement one or more game objectives.

[0024] For example, the game objective is to interact with virtual elements 230 placed at various virtual positions in the virtual world 210. These virtual elements 230 can be linked to landmarks, geographical locations, or objects 240 in the real world 200. Real-world landmarks or objects 240 can be artworks, monuments, buildings, companies, libraries, museums, or other suitable real-world landmarks or objects. Interactions include acquiring some virtual item, claiming its ownership, or using it, consuming some virtual currency, etc. To acquire these virtual elements 230, the player moves to a landmark or geographical location 240 in the real world that is linked to the virtual element 230 and interacts with the virtual element 230 in the virtual world 210. For example, player A in FIG. 2 may have to move to landmark 240 in the real world 200 to interact with or acquire a virtual element 230 linked to a specific landmark 240. Interaction with the virtual element 230 can require actions in the real world such as taking a photo and / or verifying, obtaining, or acquiring other information about the landmark or object 240 associated with the virtual element 230. In other embodiments, different or additional mechanisms for acquiring virtual elements may be available. For example, in-game items may allow the player to interact remotely with virtual elements (e.g., from a real-world location that is not the location corresponding to the virtual element).

[0025] The game objective may require the player to use one or more virtual items collected by the player within the location-based game. For example, the player may search for virtual items (such as weapons, creatures, power-up items, or other items) that may be useful in completing the game objective and move through the virtual world 210. These virtual items can be discovered or collected by moving to different locations in the real world 200 or by completing various actions in either the virtual world 210 or the real world 200. In the example shown in FIG. 2, the player uses the virtual item 232 to acquire one or more virtual elements 230. In particular, the player can deploy the virtual item 232 at a location in the virtual world 210 that is proximate to or within the virtual element 230. By deploying one or more virtual items 232 in this way, as a result, the virtual element 230 can be acquired for a particular player, or for a particular player's team / clan.

[0026] In a particular implementation, the player may collect virtual energy as part of the parallel reality game. As depicted in FIG. 2, the virtual energy 250 can be distributed at different locations in the virtual world 210. The player can collect the virtual energy 250 by moving to the corresponding location of the virtual energy 250 in the real world 200. The virtual energy 250 can be used to power virtual items or to perform various game objectives within the game. A player who loses all of the virtual energy 250 may be disconnected from the game.

[0027] According to aspects of the present disclosure, a parallel reality game can be a large multiplayer location-based game where all participants in the game share the same virtual world. The players can be divided into different teams or factions and can cooperate to achieve one or more game objectives, such as acquiring or claiming ownership of virtual elements. Thus, a parallel reality game can essentially be a social game that promotes cooperation among the players of the game. Players of opposing teams can act against each other (or, in some cases, cooperate to achieve each other's goals) during the parallel reality game. A player may use virtual items to attack players of an opposing team or slow down progress. In some cases, players are encouraged to gather at a location in the real world for cooperative or interactive events in the parallel reality game. In such cases, the game server verifies that the player is physically present and not impersonating someone else.

[0028] A parallel reality game can have various features that improve and facilitate gameplay within the parallel reality game. For example, players can accumulate virtual currency or other virtual rewards (such as virtual tokens, virtual points, virtual material resources, etc.) that can be used throughout the game (e.g., purchasing in-game items, exchanging with other items, creating items, etc.). Players can progress to various levels by completing one or more game objectives and gaining experiences within the game. In some embodiments, players can communicate with each other through one or more communication interfaces provided within the game. Players can also obtain enhanced "powers" or virtual items that can be used to complete game objectives within the game. Those skilled in the art should understand that various other game features can be included in the parallel reality game without departing from the scope of the present disclosure as provided herein.

[0029] Referring again to FIG. 1, the networked computing environment 100 uses a client-server architecture, where a game server 120 communicates with client devices 110 through a network 105 to provide a parallel reality game to players of the client devices 110. The networked computing environment 100 may also include other external systems, such as a sponsor / advertiser system or a business system. Only one client device 110 is shown in FIG. 1, but any number of client devices 110 or other external systems may be connected to the game server 120 through the network 105. Further, the networked computing environment 100 may include different or additional elements, and the functionality may be distributed between the client device 110 and the server 120 in a manner different from the methods described below.

[0030] The client device 110 can be any portable computing device usable by a player to interface with the game server 120. For example, the client device 110 can be a wireless device, a personal digital assistant (PDA), a portable gaming device, a mobile phone, a smartphone, a tablet, a navigation system, a handheld GPS system, a wearable computing device, a display having one or more processors, or other similar devices. As another example, the client device 110 includes conventional computer systems such as desktop or laptop computers. Further, the client device 110 may be a vehicle equipped with a computing device. In short, the client device 110 can be any computer device or system capable of enabling a player to interact with the game server 120. As a computing device, the client device 110 can include one or more processors and one or more computer-readable storage media. The computer-readable storage media can store instructions for causing the processor to perform operations. Preferably, the client device 110 is a portable computing device that can be easily carried or transported by a player, such as a smartphone or a tablet.

[0031] The client device 110 communicates with the game server 120 and provides sensory data of the physical environment to the game server 120. The client device 110 includes a camera assembly 125 that captures two-dimensional image data of a scene in the physical environment where the client device 110 is located. In the embodiment shown in FIG. 1, each client device 110 includes additional software components such as a gaming module 135, a positioning module 140, and a light assembly 143. The client device 110 may include various other input / output devices for receiving information from the player or providing information to the player. Examples of input / output devices include a display screen, a touch screen, a touch pad, data input keys, speakers, and a microphone suitable for voice recognition. The client device 110 may include various other sensors for recording data from the client device 110, including but not limited to motion sensors, accelerometers, gyroscopes, other inertial measurement units (IMUs), barometers, positioning systems, thermometers, light sensors, and the like. The client device 110 may further include a network interface that provides communication through the network 105. The network interface may include any suitable components for interfacing with one or more networks, including, for example, a transmitter, a receiver, a port, a controller, an antenna, or other suitable components.

[0032] The camera assembly 125 captures image data of a scene in the environment where the client device 110 is located. The camera assembly 125 may utilize various variable photo sensors with a variable capture rate and a variable color capture range. The camera assembly 125 may include a wide-angle lens or a telephoto lens. The camera assembly 125 may be configured to capture a single image or a video as the image data. The camera assembly 125 captures the image data and shares this image data with the computing device on the client device 110. Metadata that describes other details of the image data, such as sensory data (e.g., temperature, ambient brightness of the environment), or capture data (e.g., exposure, warmth, shutter speed, focal length, capture time, etc.), can be attached to the image data. The camera assembly 125 can include one or more cameras capable of capturing image data. In one example, the camera assembly 125 includes one camera and is configured to capture monocular image data. In another example, the camera assembly 125 includes two cameras and is configured to capture stereo image data. In various other embodiments, the camera assembly 125 each includes a plurality of cameras configured to capture image data.

[0033] The gaming module 135 provides an interface for a player to participate in a parallel reality game. The game server 120 transmits game data used by the gaming module 135 in the client device 110 to the client device 110 via the network 105, and provides a local version of the game to a player located at a position away from the game server 120. The game server 120 can include a network interface for providing communication through the network 105. The network interface can include any suitable components for interfacing with one or more networks, including, for example, a transmitter, a receiver, a port, a controller, an antenna, or other suitable components.

[0034] The gaming module 135 executed by the client device 110 provides an interface between the player and the parallel reality game. The gaming module 135 can present a user interface on a display device associated with the client device 110 that displays a virtual world associated with the game (e.g., renders a depiction of the virtual world), enabling the player to interact in the virtual world and perform various game objectives. In some other embodiments, the gaming module 135 presents image data of the real world (e.g., captured by the camera assembly 125) augmented with virtual elements or objects of the parallel reality game. In these embodiments, the gaming module 135 may generate or adjust virtual content according to other information received from other components of the client device. For example, the gaming module 135 may adjust virtual objects displayed on the user interface according to the depth information of the scene captured in the image data.

[0035] The gaming module 135 can also control various other outputs so that players can interact with the game without looking at the display screen. For example, the gaming module 135 can control various sounds, vibrations, or other notifications so that players can play the game without looking at the display screen. The gaming module 135 can access the game data received from the game server 120 to provide players with an accurate representation of the game. The gaming module 135 can receive and process player inputs and provide updates to the game server 120 through the network 105. The gaming module 135 may also generate or adjust game content displayed by the client device 110. For example, the gaming module 135 may generate virtual elements based on information describing the passable space in a scene (e.g., determined by the passable space estimation model 130). For example, the gaming module 135 can determine a path for virtual elements in the scene over the passable space.

[0036] The positioning module 140 can be a device or circuit for monitoring the position of the client device 110. For example, the positioning module 140 can use triangulation, proximity to a mobile relay tower or WiFi hotspot, or other suitable position determination techniques to determine the actual or relative position based on the IP address using a satellite navigation positioning system (e.g., GPS system, Galileo positioning system, Global Navigation Satellite System (GLONASS), BeiDou satellite navigation positioning system), an inertial navigation system, or a dead reckoning system. The positioning module 140 may further include various other sensors that can assist in accurately positioning the client device 110.

[0037] When the player moves around in the real world with the client device 110, the positioning module 140 tracks the player's position and provides the player's position information to the gaming module 135. The gaming module 135 updates the player's position in the virtual world associated with the game based on the player's actual position in the real world. In this way, the player can interact with the virtual world just by carrying or transporting the client device 110 in the real world. In particular, the position of the player in the virtual world can correspond to the position of the player in the real world. The gaming module 135 can provide the player's position information to the game server 120 through the network 105. In response, the game server 120 can establish various techniques to verify the position of the client device 110 in order to prevent cheaters from faking the position of the client device 110. It should be understood that the position information associated with the player is used only when the player is informed that the position information is accessed and how the position information is used in the context of the game (for example, updating the player's position in the virtual world), and only when permission is given. Also, any position information associated with the player is stored and maintained in a way that protects the player's privacy.

[0038] The game server 120 can be any computing device and can include one or more processors and one or more computer-readable storage media. The computer-readable storage media can store instructions that cause the processor to perform operations. The game server 120 can include, or communicate with, a game database 115. The game database 115 stores game data used in the parallel reality game that is supplied or provided to the client 110 through the network 105.

[0039] The game data stored in the game database 115 may include: (1) data associated with the virtual world in the parallel reality game (e.g., rendering data used to render the virtual world on the display device, geographical coordinates of positions in the virtual world, etc.); (2) data associated with the players of the parallel reality game (e.g., player profiles including, but not limited to, player information, player experience levels, player currency, the current position of the player in the virtual / real world, player energy levels, player preferences, team information, faction information, etc.); (3) data associated with the game objectives (e.g., data associated with the current game objective, the status of the game objective, past game objectives, future game objectives, desired game objectives, etc.); (4) data associated with virtual elements in the virtual world (e.g., the position of the virtual element, the type of the virtual element, the game objective associated with the virtual element, the corresponding real-world position information for the virtual element, the behavior of the virtual element, the relevance of the virtual element, etc.); (5) data associated with real-world objects, landmarks, positions linked to the elements of the virtual world (e.g., the position of the real-world object / landmark, the description of the real-world object / landmark, the relevance of the virtual element linked to the real-world object, etc.); (6) the game status (e.g., the current number of players, the current status of the game objective, the player leaderboard, etc.); (7) data associated with the actions / inputs of the players (e.g., the current position of the player, the past position of the player, the movement of the player, the input of the player, the query of the player, the communication of the player, etc.); and (8) any other arbitrary data used, related to, or obtained during the implementation of the parallel reality. The game data stored in the game database 115 can be added either offline or in real time by the system administrator or by data received from the users / players of the system 100, such as the client device 110, through the network 105.

[0040] The game server 120 can be configured to receive requests for game data from the client device 110 (e.g., via remote procedure calls (RPCs)) and respond to these requests via the network 105. For example, the game server 120 can encode game data into one or more data files and provide this data file to the client device 110. Also, the game server 120 can be configured to receive game data (e.g., player position, player actions, player input, etc.) from the client device 110 via the network 105. For example, the client device 110 can be configured to periodically send player input and other updates to the game server 120, and the game server 120 can use this to update the game data in the game database 115 to reflect any and all changes in the game situation.

[0041] In the illustrated embodiment, the server 120 includes a universal game module 145, a commercial game module 150, a data collection module 155, an event module 160, and a passable space estimation learning system 170. As described above, the game server 120 interacts with a game database 115 that may be part of the game server 120 or remotely accessible (e.g., the game database 115 may be a distributed database accessed via the network 105). In other embodiments, the game server 120 includes different or additional elements. Also, the functionality may be distributed among the elements in a manner different from that described. For example, the game database 115 can be integrated within the game server 120.

[0042] The Universal Game Module 145 hosts a parallel reality game for all players and serves as a reliable source of information on the current status of all players in the parallel reality game. As the host, the Universal Game Module 145 generates game content for presentation to players, for example, via each client device 110. When hosting a parallel reality game, the Universal Game Module 145 may access the game database 115 to retrieve or store game data. The Universal Game Module 145 also receives game data (e.g., insider information, player input, player position, player actions, landmark information, etc.) from the client devices 110 and incorporates the received game data into the entire parallel reality game for all players. The Universal Game Module 145 can also manage the transmission of game data to the client devices 110 through the network 105. The Universal Game Module 145 may also control aspects of the security of the client devices 110, including, but not limited to, ensuring the connection between the client devices 110 and the game server 120, establishing connections between various client devices 110, and verifying the locations of the various client devices 110.

[0043] The Commercial Game Module 150 can be separated from or be part of the Universal Game Module 145 in the embodiments in which it is included. The Commercial Game Module 150 can manage the inclusion of various game features within a parallel reality game linked to commercial activities in the real world. For example, the Commercial Game Module 150 can receive requests to include game features linked to commercial activities in the parallel reality game from external systems such as sponsors / advertisers, businesses, or other entities via the network 105 (through the network interface). The Commercial Game Module 150 can then arrange for the inclusion of these game features in the parallel reality game.

[0044] The game server 120 can further include a data collection module 155. In the embodiments in which it is included, the data collection module 155 can be separated from the universal game module 145 or can be a part thereof. The data collection module 155 can manage the inclusion of various game features into the parallel reality game linked to the data collection activities in the real world. For example, the data collection module 155 can modify the game data stored in the database 115 so as to include game features linked to the data collection activities in the parallel reality game. The data collection module 155 can also analyze the data collected by the player according to the data collection activities and provide the data for access by various platforms.

[0045] The event module 160 manages the player's access to events in the parallel reality game. It should be understood that although the term "event" is used for convenience, this term does not necessarily refer to a specific event at a specific location or time. Rather, it may refer to the provision of access-controlled game content that uses one or more access criteria to determine whether a player may access the content. Such content may be part of a larger parallel reality game that includes game content with little or no access control, or may be a stand-alone access-controlled parallel reality game.

[0046] Network 105 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof. This network can also include a direct connection between client device 110 and game server 120. Usually, communication between game server 120 and client device 110 can be carried out via a network interface using any type of wired or wireless connection and various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encoding or formatting (e.g., HTML, XML, JSON), or protection methods (e.g., VPN, secure HTTP, SSL).

[0047] The technology considered in this specification refers to the actions taken and the information sent between such systems, along with servers, databases, software applications, and other computer-based systems. One of ordinary skill in the art will recognize that the flexibility inherent in computer-based systems allows for various configurations, combinations, and divisions of tasks among components. For example, the server processing considered in this specification may be implemented using a single server or multiple servers operating in combination. The database and application may be implemented on a single system or distributed across multiple systems. The distributed components may operate sequentially or in parallel.

[0048] Also, in the systems and methods contemplated herein, in situations where the system and methods access and analyze personal information about a user (e.g., a player), such as location information, or use personal information, the user may be given the opportunity to control whether a program or function collects information, whether to receive content from the system or other applications, or how to receive it. Such information or data shall not be collected or used until the user is provided with a meaningful notice as to what information is being collected and how that information will be used. This information shall not be collected or used without the user's consent, and this consent may also be revoked or changed by the user at any time. Accordingly, the user can control how information about the user is collected and how it is used by the application or system. Also, certain information or data can be handled in one or more ways before being stored or used so as to exclude personally identifiable information. For example, the identification of the user may be handled such that personally identifiable information cannot be determined for the user.

[0049] <Exemplary Game Interface> FIG. 3 depicts an embodiment of a game interface 300 that can be presented on the display of client 110 as part of the interface between a player and virtual world 210. The game interface 300 includes a display window 310 that can be used to display the virtual world 210 and various other aspects of the game, such as the position 222 of the player in the virtual world 210, the positions of virtual elements 230, virtual items 232, and virtual energy 250. The user interface 300 can also display other information, such as game data information, game communications, player information, client location assessment instructions, and other information associated with the game. For example, the user interface can display player information 315, such as the player's name, experience level, and other information. The user interface 300 can include a menu 320 for accessing various game settings and other information associated with the game. The user interface 300 can also include a communication interface 330 that enables communication between the game system and the player and between one or more players of the parallel reality game.

[0050] According to an aspect of the present disclosure, a player can interact with a parallel reality game simply by carrying the client device 110 in the real world. For example, a player can play the game by simply accessing an application associated with the parallel reality game on a smartphone and walking around the real world with the smartphone. In this regard, the player does not need to continuously view the visual representation of the virtual world on the display screen to play a location-based game. As a result, the user interface 300 can include a plurality of non-visual elements so that the user can interact with the game. For example, the game interface can provide an auditory notification to the player when the player is approaching a virtual element or object in the game or when an important event occurs in the parallel reality game. The player can control such an auditory notification with the audio control 340. Different types of auditory notifications can be provided to the user according to the type of virtual element or event. The auditory notification can increase or decrease in frequency or volume according to the proximity of the player to the virtual element or object. Other non-visual notifications and signals, such as vibration notifications or other suitable notifications or signals, can be provided to the user.

[0051] Referring again to FIG. 1, in some embodiments, the gaming module 135 includes an object identifier module 175, a window module 180, a flashlight module 185, an object interaction module 190, and a depth estimation module 195. These modules are stored in the client device 110, but one or more of these modules may be stored in the game server 120. For example, an image captured by the camera assembly 125 is sent to a module on the game server 120, and this module later provides a display command to the client device 110 via the network 105. Further, these modules are part of the gaming module 135 but are not limited to the gaming context. Modules 175, 180, 185, 190, and 195 will be further described below.

[0052] <Display of Window in Virtual Environment> In some embodiments, the image data is presented by the client device 110 to provide a combined or interchangeable AR / VR experience for a user (e.g., a game player). The combined AR / VR experience includes displaying AR image data to the user and, in certain situations, VR image data is displayed to the user. For example, while displaying an image of an external environment (also referred to as the real world), the gaming module 135 displays a window. In the context of this description, a window is a virtual object that shows a virtual environment (referred to as the window) that was not previously displayed to the user. The window is typically displayed on an object in the external environment. For example, the window is displayed on a photo frame on a wall. Thus, the user can move the photo frame to view portions of the virtual environment that appear to be behind the photo frame. Such a combined AR / VR experience is further described below with respect to the object identifier module 175 and the window module 180.

[0053] The object identifier module 175 receives and analyzes the images captured by the camera assembly 125 and identifies the objects within the images. The object identifier module 175 can identify inanimate objects and living objects (e.g., animals and humans), or parts of living objects (e.g., hands or feet). Identifying an object means recognizing that a group of pixels in the image represents a certain object and determining the object type of that object. The object type describes what the object is (e.g., a person, a dog, a chair, a wall, a table, or a lamp). For example, the object identifier module 175 determines a group of pixels representing a chair. The object identifier module 175 may be configured to identify common objects in a room, such as walls, tables, chairs, pictures, and lights. In some embodiments, the object identifier module 175 uses a machine learning classifier capable of identifying one or more objects in the input image (or a time series of images). This classifier is trained using training images that include object type labels for groups of pixels (e.g., manually labeled). Alternatively, a rule-based approach may be used, where rules are applied to the input image to determine whether an object of a given type is depicted.

[0054] The object identifier module 175 may determine whether the identified object meets one or more virtual environment criteria. The virtual environment criteria describe an object or characteristics of an object that are suitable for representing a window into the virtual environment. Suitable objects typically have a flat surface facing the external environment, where the surface area of the surface is large enough (e.g., larger than 4 square inches) to overlay a portion of the virtual environment. Suitable objects may be mounted on a wall. For example, the object may be thin, rectangular, and desirably mounted on a wall. Suitable objects may also be easily accessible by a user (e.g., the object is visible and there are no obstacles preventing the user from approaching the object, etc.). Thus, the object identifier module 175 may determine whether the object meets one or more of these virtual environment criteria.

[0055] The virtual environment criteria may be a list of object types suitable for representing a window. For example, the virtual environment criteria identify that a power outlet, a light switch, a photo frame, and a computer monitor are suitable objects. Thus, in these embodiments, an object may meet the virtual environment criteria if it is assigned to one of the object types in the list (by the object identifier module 175).

[0056] In some embodiments, instead of identifying the entire object, the object identifier module 175 identifies the surface of the object in the external environment and determines whether any of the identified surfaces are suitable for representing a window (e.g., whether the surface is flat or has a surface area exceeding a threshold).

[0057] The window module 180 provides instructions for displaying a window into the virtual environment to an object that meets one or more virtual environment criteria (e.g., for display by the client device 110). If multiple identified objects meet the virtual environment criteria, the window module 180 may select the object that best meets the criteria. Alternatively, the window module 180 may allow the user to select one of the multiple objects as the object representing the window. In some embodiments, the window may not be displayed until the window module 180 receives instructions from the user to view the window, or until the client device 110 comes within a threshold distance of the identified object.

[0058] The window is displayed at the location of the selected object (e.g., the face facing outside the object) or near the location of the selected object (e.g., within 12 inches of the object). The face of the window may be substantially parallel (e.g., within 5 degrees) to the face facing outside the object. The shape and size of the window may correspond to the shape and size of the object (e.g., match or be substantially similar). For example, the boundary of the window matches the contour of the object.

[0059] The window shows a user (using the client device 110) a portion of the virtual environment through the window. In other words, a portion of the virtual environment is viewed from the perspective of someone looking through the window. For example, to view other parts of the virtual environment, the user needs to move the client device 110 relative to the object. In some cases, the window includes a door that the user must open (e.g., by selecting the door) to view a portion of the virtual environment.

[0060] Virtual environments typically include objects of interest to a user playing a game (e.g., a parallel reality game). For example, a virtual environment includes virtual characters related to the game. In some cases, virtual characters in the virtual environment interact with virtual or real objects in the external environment (e.g., virtual characters in the external environment). For example, a virtual character in the virtual environment talks to the user through a window. Information related to the virtual environment, such as the size and shape of the virtual environment and objects in that virtual environment, may be received from the game server 120. For example, the game server 120 displays a virtual room including four walls, a floor, and a ceiling, and provides instructions to display specific virtual objects within this room.

[0061] In some embodiments, the user can "enter" the virtual environment, for example, by selecting a window or providing an instruction to enter the window. In such embodiments, since the user is shown to have "entered" the virtual environment (in other words, the client device displays a VR image corresponding to the virtual environment), the image data of the virtual environment may be displayed instead of the image data of the external environment.

[0062] In some embodiments, prior to displaying a window on an object, the window module 180 displays an AR visual effect (e.g., an animation effect) on the selected object or around the selected object. The visual effect is intended to draw the user's line of sight to the object and inform the user that the window will be displayed on that object. The visual effect may move or change color to attract the user's attention. For example, the visual effect changes the color of the object. Other examples of visual effects include smoke, explosions, fireworks, and stars.

[0063] Figures 4A - 4D are images as a series of examples showing the display of a window in a virtual environment according to some embodiments. Figure 4A depicts a display 400 of a client device 405 that displays image data of an external environment 410 according to one embodiment. In the example of Figure 4A, the external environment 410 is part of a room that includes two walls, a floor, a table, and a power outlet 415 mounted on the wall. Although omitted in Figure 4A, the image data of the external environment 410 may be extended with virtual objects.

[0064] In the examples of Figures 4A - 4D, the wall outlet 415 is selected to represent a window to the virtual environment. To communicate this to the user, the window module 180 displays a visual effect 420 through the wall outlet 420 (see Figure 4B). In this example, the visual effect 420 includes stars around the outlet 415.

[0065] Figures 4C and 4D show a window 425 to a virtual environment 430 displayed on top of the outlet 415. The size and shape of the window 425 match the size and shape of the outlet 415. In Figure 4D, the client device 110 is in a position closer to the outlet 415 so that details of the virtual environment 430 can be seen. The virtual environment 430 includes two virtual characters 430 and a virtual table 440 in the room. This room appears behind the wall on which the outlet 415 is mounted.

[0066] <Interaction by Light> As described with reference to FIG. 1, the client device 110 may include a light assembly 143 capable of generating light to illuminate the external environment. For example, the light assembly generates sufficient light to assist the user in viewing portions of the external environment in a dim lighting situation. The light assembly 143 may be used by the camera assembly 125 to illuminate the external environment when taking an image. The light assembly includes one or more light sources, such as light emitting diodes (LEDs), to generate light. The following description assumes that the light sources of the light assembly 143 are arranged such that the area of the external environment illuminated by the light assembly 143 is included in the image captured by the camera assembly 125. For example, the light assembly 143 is oriented in the same direction as the camera assembly 125.

[0067] When the light assembly 143 is generating light, it may be referred to as being "on". When the light assembly 143 is not generating any light or is generating a small amount of light, it may be referred to as being "off". The light assembly 143 may provide a user interface for the user to turn the light assembly 143 on and off. The gaming module 135 may also include a user interface for controlling the light assembly 143 by the user (e.g., so that the user can control the light assembly 143 without leaving the game).

[0068] In some gaming situations, or in gaming mode, the user may interact with virtual objects displayed in the external environment (or virtual environment) using the light from the light assembly 143. For example, when the light assembly 143 is on, the gaming module 143 displays virtual objects that were not previously displayed. In other examples, after the light assembly 143 is turned on, virtual characters in the external environment gather in the area of the external environment illuminated by the light assembly 143. Depending on the game, when the light assembly 143 is turned off, the virtual objects may remain in that area or may start moving again through the external environment. Thus, by turning on the light assembly 143, the user may control the position of virtual objects in the external environment by moving the client device 110 (and thereby changing the portion of the environment illuminated by the light assembly 143). In this example, the instruction to turn on the light assembly 143 may be referred to as a gathering instruction. In some embodiments, the user can instruct the virtual object to move to a position in the external environment by selecting an area of the display (e.g., representing a position).

[0069] The flashlight module 185 is communicatively coupled to the light assembly 143 and displays the movement of the virtual object (e.g., provides an indication to the display for the client device 110). Specifically, in response to the light assembly 143 being turned on, the flashlight module 185 displays the virtual object gathering from its current position in the external environment to the area of the external environment illuminated by the light assembly 143.

[0070] Since the light assembly 143 is typically fixed relative to the camera assembly, the flash light module 185 need not identify the light from the light assembly 143 in the image data of the external environment. Instead, the flash light module 185 may move virtual objects to the area of the external environment being displayed by the client device 110 (referred to as the area of the external environment corresponding to the area of the display). In some cases, the virtual object is moved to a specific area of the display, such as the central area. When the client device 110 is moved when the light assembly 143 is on, different areas of the environment are illuminated. When the movement exceeds a threshold (determined, for example, by analyzing signals from the positioning module 140), the flash light module 185 may display the movement of the virtual object to a new area that is receiving light from the previous position.

[0071] In some embodiments, the flash light module 185 displays a visual effect (e.g., an animated effect) indicating the area illuminated by the light assembly 143. This may assist the user in identifying the area. Example visual effects include a virtual spotlight or highlight on the area.

[0072] Figures 5A and 5B are a series of images as examples showing how virtual objects gather in some embodiments. Figure 5A depicts a client device 110 displaying image data of three virtual characters 505 in an external environment 510 according to one embodiment. The virtual characters 505 have a human form and are arranged throughout the external environment 510. The virtual characters 505 may be arranged throughout the external environment 510 for any number of reasons. For example, the game module 135 has generated virtual characters 505 at random locations in the external environment 510, and the game module 135 displays the virtual characters 505 walking around the external environment.

[0073] While playing the game, it may be desirable for the user to gather three virtual characters 505 in a certain area of the external environment 510. To gather the virtual characters 505, the user may turn on the light assembly 143 as described above. FIG. 5B depicts an image data in which the client device 110 displays three virtual characters 505 gathered in an area of the external environment 515 illuminated by the light assembly 143 according to an embodiment. In the example of FIG. 5B, the contour of the illuminated area 515 is virtually highlighted by a visual effect that helps the user identify the area 515.

[0074] <Interaction with Virtual Objects> Referring to FIG. 1, the object interaction module 190 displays how a virtual object interacts with an object in the external environment (e.g., provides instructions for display by the client device 110). As a result, the virtual object appears more realistic and alive to the user. For this purpose, the object interaction module 190 receives the object identified from the object interaction module 190. By analyzing characteristics such as the shape and surface of the identified object, the object interaction module 190 can display how the virtual object directly or indirectly interacts with the identified object. Examples of indirect interactions include a virtual character walking around or looking at an object. Examples of direct interactions include a virtual character touching, approaching, standing up, or climbing an object. For example, the object interaction module 190 displays how a virtual character climbs (e.g., goes up and down) an object such as a stick-shaped object or an object with a protrusion or handle. In another example, the object interaction module 190 displays how a virtual character slides down an inclined plane having an angle exceeding a threshold amount (e.g., more than 5 degrees and less than 80 degrees) with respect to what the object interaction module 190 determines to be the ground (e.g., a plane orthogonal to the average direction of the visible ground in gravity or the camera's field of view).

[0075] In some embodiments, the object interaction module 190 displays how the virtual object interacts with the user. For example, the user can "pick up" the virtual object and move it to different positions in the environment. In other examples, the object interaction module 190 gives instructions such that the virtual object "follows" the user in the external environment. In this example, the object interaction module 190 can identify the object as a person. The object interaction module 190 then displays how the virtual character moves (e.g., walks) behind the person in the environment. The components of the person's body (e.g., feet, torso, and arms) may be identified such that the virtual character is displayed as walking in step behind the person.

[0076] The depth estimation module 195 estimates the distance from the client device 110 to the object (identified by the object identifier module 175). Based on the estimated distance (also referred to as depth information), the gaming module 135 displays the virtual object in the external environment (e.g., provides instructions for the client device 110 to display). For example, when the virtual object is displayed near an object, the size of the virtual object may be adjusted based on the estimated distance to that object. In some embodiments, the depth information can be used to partially display the virtual object behind an object. For example, if the depth information indicates that a real-world object is closer than the position of the virtual object to be displayed, the virtual object may be displayed such that the real-world object appears to cover part of the virtual object. Thus, by using depth information, the virtual object appears more realistic.

[0077] In some embodiments, the gaming module 135 includes a virtual camera module (not shown) that causes the user to take a photo showing a virtual object in the external environment. Also in these embodiments, the virtual object may be displayed based on depth information.

[0078] The depth estimation module 195 may be constructed by machine learning. For example, the depth estimation module is trained using training images that include object distance labels (e.g., manually labeled) for pixel groups in the game. If the client device includes two cameras, this distance may be estimated based on stereo image data. Alternatively, a rule-based approach may be used that applies rules for estimating the distance to an object in the input image to the input image.

[0079] FIG. 6 depicts a client device 110 that displays image data of two virtual characters 605 within an external environment 610 according to one embodiment. Character 605A is shown as standing on a table in the background of the image, and character 605B is shown as standing on the ground in the foreground of the image. The virtual characters 605 have the same size, but are displayed in different sizes in the image so that the characters appear to be in the external environment 610. In particular, since character 605A is displayed in the background and character 605B is displayed in the foreground, character 605A is smaller than character 605B and can provide a sense of depth. The size of the characters 605 may be based on distance information from the depth estimation module 195. For example, the depth estimation module 195 estimates the distance between the client device and the table and determines the heights of characters 605A and 605B based on this distance.

[0080] <Interaction of Modules> The window module 180, the flashlight module 185, the object interaction module 190, and the depth estimation module 195 have been described as independent, but they may all be used together to improve the user's gaming experience. For example, each virtual object may be displayed based on depth information from the depth estimation module 195.

[0081] In other examples, in some gaming situations, the virtual environment may be darkened so that the user cannot see a portion of the virtual environment through the window. The user may virtually illuminate a portion of the virtual environment by turning on the light assembly 143 and positioning the client device 110 so that light appears to propagate through the window into the virtual environment. Similarly, the user may orient a virtual object in the external environment so that it moves through the window into the virtual environment by positioning the client device 110 so that light from the light assembly appears to propagate through the window into the virtual environment.

[0082] In other examples, when the user turns on the light assembly 143 and holds a hand in front of the client device 110 so that the light is directed at the hand, the virtual object may move on the user's hand. The user may then move the virtual object in the external environment by moving the hand. Further, when the window is displayed and the user brings a hand close to the object representing the window, the gaming module 135 may display the virtual object as being disposed in the virtual environment through the window.

[0083] <Process of displaying a window into a virtual environment> Figure 7 is a flowchart illustrating a general process 700 for displaying a window into a virtual environment, according to one embodiment. Process 700 may be accomplished by a client device (e.g., client device 110) having an object identifier module and a window module. The client device may be a mobile device such as a smartphone, and may have a camera (e.g., camera assembly 125), a light assembly (e.g., light assembly 143), and a display. In some embodiments, the client device implements at least in part the parallel reality game described in FIGS. 1-3 above. The following description is in the context of a client device, but process 700 (or steps of the process) may be implemented on other computing devices (e.g., a server). The steps of process 700 may be performed in a different order, process 700 may include different or additional steps, or may include fewer steps.

[0084] The client device captures an image of the external environment with the camera assembly (710). The client device identifies an object in the external environment based on the captured image (720). The client device determines whether the object in the external environment meets the virtual environment criteria (730). The client device displays a window into the virtual environment at or near the position of the object in an additional image of the external environment captured by the camera assembly (740). The client device may display the window in response to determining that the object meets the criteria.

[0085] In some embodiments, the client device displays how a virtual object in the external environment moves through the window into the virtual environment.

[0086] The client device may receive information about the virtual environment from a server and may use the information about the virtual environment to determine virtual content to display through the window.

[0087] In some embodiments, process 700 further includes a client device that identifies candidate objects in an external environment. This client device determines whether each of the candidate objects meets virtual environment criteria. The client device selects one of the candidate objects that meets the virtual environment criteria as the object described above.

[0088] Process 700 may further include a client device that receives an instruction to display a virtual environment and, in response to receiving this instruction, displays the virtual environment instead of an image of the external environment.

[0089] Process 700 may further include a client device that identifies a second object having an inclined surface in the external environment based on a captured image, climbs the second object in an additional image of the external environment captured by the camera assembly, and displays a virtual object that slides down the inclined surface of the second object.

[0090] In some embodiments, determining that an object in the external environment meets virtual environment criteria includes determining that the object is on a wall in the external environment.

[0091] <Process of Moving a Virtual Object> FIG. 8 is a flowchart illustrating a general process 800 for moving virtual objects in an external environment according to an embodiment. The process 800 may be accomplished by a client device (e.g., client device 110) having a flashlight module. The client device may be a mobile device such as a smartphone and may have a camera (e.g., camera assembly 125), a light assembly (e.g., light assembly 143), and a display. In some embodiments, the client device implements at least in part the parallel reality game described in FIGS. 1-3 above. The following description is in the context of a client device, but the process 800 (or steps of the process) may be implemented on other computing devices (e.g., a server). The steps of the process 800 may be performed in a different order, the process 800 may include different or additional steps, or may include fewer steps.

[0092] The client device captures an image of the external environment with the camera assembly (810). The client device displays a virtual object in the image of the external environment on the display (820). The client device receives an aggregation command (e.g., from a user) (830). The client device displays the movement of the virtual object in the additional image of the external environment captured by the camera assembly to a region of the external environment corresponding to the region of the display (840). The client device may display the movement of the virtual object in response to receiving the aggregation command.

[0093] In some embodiments, the aggregation command is a command to turn on the light assembly of the client device. The region of the external environment may be a region illuminated by light from the light assembly. In some embodiments, the client device displays a visual effect in the region of the display to indicate the region of the external environment illuminated by light from the light assembly.

[0094] In some embodiments, the client displays a plurality of virtual objects in an image of the external environment, and each object is displayed at a different position in the external environment. The client device may display how each of the plurality of virtual objects moves from its position to an area of the external environment.

[0095] In some embodiments, the client device determines that the movement of the client device exceeds a threshold. In response to determining that the movement of the client device exceeds the threshold, the client device displays how a virtual object moves from an area of the external environment to a second area of the external environment corresponding to the area of the display.

[0096] Process 800 may further include a client device that, based on the captured image, identifies an object having an inclined surface in the external environment, climbs the object, and displays a virtual object that slides down the inclined surface of the object.

[0097] Process 800 is described in the context of AR, but a similar process may be implemented in the context of VR. For example, the client device displays an image of a VR environment, and in response to receiving a set instruction, the client device displays a virtual object of the VR environment that moves to an area of the VR environment corresponding to the area of the display.

[0098] <Process of Displaying Virtual Object> FIG. 9 is a flowchart illustrating a general process 900 for displaying virtual objects in an external environment according to an embodiment. The process 900 may be accomplished by a client device having a flashlight module (e.g., client device 110). The client device may be a mobile device such as a smartphone and may have a camera (e.g., camera assembly 125), a light assembly (e.g., light assembly 143), and a display. In some embodiments, the client device implements the parallel reality game described in FIGS. 1-3 above. The following description is in the context of a client device, but the process 900 (or steps of the process) may be implemented on other computing devices (e.g., a server). The steps of the process 900 may be performed in a different order, the process 900 may include different or additional steps, or may include fewer steps.

[0099] The client device captures an image of the external environment with the camera assembly (910). The client device 920 identifies an object in the external environment based on the captured image (920). The client device estimates the depth information of the object in the captured image (930). The client device displays a virtual object based on the identified object and the depth information (940).

[0100] In some embodiments, the virtual object is a virtual character, and displaying the virtual object includes displaying the virtual character interacting with the object.

[0101] In some embodiments, the object is a human hand.

[0102] Process 900 may further include identifying an inclined surface of an object in the external environment based on the captured image, climbing the object, and displaying a virtual object that slides down the inclined surface of the object.

[0103] <Example arithmetic system> FIG. 10 is an architecture as an example of an arithmetic device according to an embodiment. FIG. 10 depicts a high-level block diagram showing the physical components of a computer used as part or all of one or more entities described herein. However, according to one embodiment, the computer may have additional components, fewer components, or variations thereof than those provided in FIG. 10. In FIG. 10, although a computer 1000 is depicted, the figure is intended as a functional description of various features that may exist in a computer system, rather than a structural schematic of the implementation described herein. In fact, as is recognized by those skilled in the art, the separately shown items can be combined and some items can be separated.

[0104] FIG. 10 shows at least one processor 1002 connected to a chipset 1004. Also connected to the chipset 1004 are a memory 1006, a storage device 1008, a keyboard 1010, a graphics adapter 1012, a pointing device 1014, and a network adapter 1016. A display 1018 is connected to the graphics adapter 1012. In one embodiment, the functions of the chipset 1004 are provided by a memory controller hub 1020 and an I / O hub 1022. In other embodiments, the memory 1006 is directly connected to the processor 1002 instead of the chipset 1004. In some embodiments, the computer 1000 includes one or more communication buses for interconnecting these components. The one or more communication buses optionally include circuitry (sometimes referred to as a chipset) for interconnecting and controlling communication between system components.

[0105] The memory device 1008 is any non-transitory computer-readable storage medium such as a hard drive, a compact disk read-only memory (CD-ROM), a DVD, or a solid-state memory device or other optical storage, a magnetic cassette, a magnetic tape, a magnetic disk storage or other magnetic storage device, a magnetic disk storage device, an optical disk storage device, a flash memory device, or other non-volatile solid-state storage device, etc. Such a memory device 1008 can also be referred to as persistent memory. The pointing device 1014 may be a mouse, a trackball, or other types of pointing devices, and is used in combination with the keyboard 1010 to input data to the computer 1000. The graphics adapter 1012 displays images and other information on the display 1018. The network adapter 1016 connects the computer 1000 to a local area network or a wide area network.

[0106] The memory 1006 holds instructions and data used by the processor 1002. The memory 1006 can be non-persistent memory, examples of which include high-speed random access memory such as DRAM, SRAM, DDR RAM, ROM, EEPROM, flash memory, etc.

[0107] As is well known in the art, the computer 1000 can have other components different from those shown in FIG. 10. Further, the computer 1000 can omit the specific components shown in the figure. In one embodiment, the computer 1000 acting as a server may omit the keyboard 1010, the pointing device 1014, the graphics adapter 1012, or the display 1018. Further, the memory device 1008 can be present locally or remotely to the computer 1000 (for example, implemented within a storage area network (SAN)).

[0108] As is well known in the art, computer 1000 is adapted to execute computer program modules for providing the functions described herein. As used herein, the term "module" refers to computer program logic utilized to provide a particular function. Thus, a module can be implemented in hardware, firmware, or software. In one embodiment, the program modules are stored in storage device 1008, loaded into memory 1006, and executed by processor 302.

[0109] <Additional Considerations> Some of the above description describes embodiments related to algorithmic processes or operations. These algorithmic descriptions and representations are commonly used by those skilled in the data processing field to effectively convey the content of their work to other skilled artisans. It is understood that these operations, when described functionally, computationally, or logically, are implemented by a computer program that includes instructions executed by a processor or equivalent electrical circuit, or microcode, etc. Further, in some cases, it is also apparent that it is convenient to refer to the arrangement of these functional operations as modules without loss of generality.

[0110] Any reference herein to "one embodiment" or "an embodiment" means that a particular element, feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. Although the phrase "one embodiment" appears in various places in the specification, these do not necessarily all refer to the same embodiment.

[0111] Some embodiments may be described using the terms "coupled" and "connected" along with their derivatives. It should be understood that these terms are not intended to be synonyms of each other. For example, some embodiments may be described using the term "connected" to indicate that two or more elements are in direct physical or electrical contact with each other. In other examples, some embodiments may be described using the term "coupled" to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" may also mean that two or more elements do not contact each other directly, but cooperate or interact with each other. These embodiments are not limited to this context.

[0112] As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having" or any other variation thereof are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, "or" refers to an inclusive "or" and not an exclusive "or". For example, a condition A or B is satisfied by any of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).

[0113] Furthermore, the use of the indefinite article is employed to describe elements and components of the embodiments. This is done solely for convenience to convey the general meaning of the present disclosure. This description must be read to include one, or at least one, and the singular form also includes the plural unless the contrary meaning is apparent. When a value is described as "about" or "substantially" (or derivatives thereof), such a value must be construed as being precisely ±10% unless another meaning is apparent from the context. For example, "about 10" must be understood to mean "in the range of 9 to 11".

[0114] Those skilled in the art will, by reading the present disclosure, understand further alternative structural and functional designs for the systems and processes that provide the AR functionality of the present disclosure. Accordingly, while specific embodiments and applications have been illustrated and described, the subject matter of the description is not limited to the precise structures and components disclosed herein, and it must be understood that various modifications, changes, and variations to the arrangement, operation, and details of the methods and apparatuses of the present disclosure will be apparent to those skilled in the art. The scope of protection must be limited only by the following claims.

Claims

1. A non - transitory computer - readable storage medium that stores instructions for causing an arithmetic device to perform operations during execution by the arithmetic device, the operations including: capturing an image of an external environment by a camera assembly; identifying an object in the external environment based on the captured image; determining whether the object in the external environment meets virtual environment criteria; displaying a window to the virtual environment at or near the position of the object within an additional image of the external environment captured by the camera assembly in response to determining that the object meets the virtual environment criteria; and a non - transitory computer - readable storage medium.

2. The operations further include displaying, through the window, the movement of a virtual object in the external environment into the virtual environment. The non - transitory computer - readable storage medium according to Claim 1.

3. The operations further include: receiving information about the virtual environment from a server; determining virtual content to be displayed through the window using the information about the virtual environment. The non - transitory computer - readable storage medium according to Claim 1.

4. The operations further include: identifying candidate objects in the external environment; determining whether each of the candidate objects meets the virtual environment criteria; selecting one of the candidate objects that meets the virtual environment criteria as the object. The non - transitory computer - readable storage medium according to Claim 1.

5. The operations further include: receiving an instruction to display the virtual environment; displaying the virtual environment in place of the image of the external environment in response to receiving the instruction. The non - transitory computer - readable storage medium according to Claim 1.

6. The shape and size of the window correspond to the shape and size of the object. The non - transitory computer - readable storage medium according to Claim 1.

7. The plane of the window is substantially parallel to the surface of the object. The non - transitory computer - readable storage medium according to Claim 1.

8. The virtual environment is a virtual room having one or more virtual objects. The non - transitory computer - readable storage medium according to Claim 1.

9. Determining that the object in the external environment meets the virtual environment standard includes determining that the object is on the wall of the external environment. The non-transitory computer-readable storage medium according to claim 1.

10. The operation is displaying a virtual object in the external environment on a display; receiving a set of instructions including an instruction to turn on a light assembly of a mobile device associated with the display via user input; and further including, in response to receiving the set of instructions, displaying a state in which the virtual object moves to an area of the external environment corresponding to an area of the display. The non-transitory computer-readable storage medium according to claim 1.

11. The operation is identifying a second object having an inclined surface in the external environment based on the captured image; and further including, on the additional image of the external environment captured by the camera assembly, displaying a virtual object that climbs the second object and slides down the inclined surface of the second object. The non-transitory computer-readable storage medium according to claim 1.

12. A non-transitory computer-readable storage medium that stores instructions for causing an arithmetic device to perform an operation when executed by the arithmetic device, wherein the operation is capturing an image of an external environment by a camera assembly; displaying a virtual object in the image of the external environment on a display; receiving a set of instructions; and including, in response to receiving the set of instructions, displaying a state in which the virtual object in an additional image of the external environment captured by the camera assembly moves to an area of the external environment corresponding to an area of the display. Non-transitory computer-readable storage medium.

13. The set of instructions includes an instruction to turn on a light assembly of a client device associated with the display. The non-transitory computer-readable storage medium according to claim 12.

14. The area of the external environment is an area illuminated by light from the light assembly. The non-transitory computer-readable storage medium according to claim 13.

15. The operation is identifying an object having an inclined surface in the external environment based on the captured image; further including displaying that the virtual object climbs the object and slides down the slope of the object; The non-transitory computer-readable storage medium according to claim 12.

16. Displaying the virtual object includes displaying a plurality of virtual objects respectively displayed at different positions of the external environment within the image of the external environment. Displaying the movement of the virtual object in the additional image to the area of the external environment corresponding to the area of the display includes displaying the movement of each of the plurality of virtual objects from their respective positions to the area of the external environment. The non-transitory computer-readable storage medium according to claim 12.

17. The operation is determining that the movement of the client device associated with the display has exceeded a threshold; further including, in response to the determination that the movement of the client device has exceeded the threshold, displaying the movement of the virtual object from the area of the external environment to a second area of the external environment corresponding to the area of the display. The non-transitory computer-readable storage medium according to claim 12.

18. A non-transitory computer-readable storage medium storing instructions that cause an arithmetic device to perform an operation when executed by the arithmetic device, the operation including capturing an image of an external environment by a camera assembly; identifying an object in the external environment based on the captured image; estimating the depth information of the object in the captured image; displaying a virtual object based on the identified object and the depth information. Non-transitory computer-readable storage medium.

19. The virtual object is a virtual character, and displaying the virtual object includes displaying the interaction between the virtual character and the object. The non-transitory computer-readable storage medium according to claim 18.

20. The operation is identifying the slope of the object in the external environment based on the captured image; further including displaying that the virtual object climbs the object and slides down the slope of the object. The non-transitory computer-readable storage medium according to claim 18.

Images