Mapping traversal spaces in scene using three-dimensional mesh

Abstract

The parallel reality game uses a virtual game board having tiles placed on an identified traversal space that corresponds to a flat area of the scene. A game board generation module receives one or more images of a scene captured by a camera of a mobile device. The game board generation module obtains a topographic mesh of the scene based on the received one or more images. The game board generation module then identifies a traversal space within the scene based on the obtained topographic mesh. A game board generation module determines a location of each polygon tile in the set of polygon tiles in the identified traversal space. The game board generation module also allows queries to identify portions of the game boards that meet the one or more provided criteria.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 316,325 filed on March 3, 2022, and U.S. Provisional Patent Application No. 63 / 345,420 filed on May 24, 2022, the entire contents of which are incorporated herein by reference. Background Art 1. Technical Field

[0003] The described subject matter relates generally to augmented reality and, more particularly, to building a game board that corresponds to real-world geography.

[0004] 2. Problem

[0005] Location-based games use the real world as their geographic environment. Parallel reality games are a type of location-based games that use a virtual world that is parallel to the real world's geographic environment. The parallel virtual world can span the entire real world, or it can span a bounded area around a mobile device that is used to play the game. Parallel reality games played in a bounded area around a mobile device can set up bounded areas by requiring the player to manually define the bounded area. For example, the game can require the player to walk around the desired boundaries of the game to allow the game to identify where the edge of the play area is placed. However, it would be advantageous to enable parallel reality games to automatically detect and set up the game area. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A networked computing environment is shown in accordance with one or more embodiments.

[0007] Figure 2 Depicted are representations of a virtual world having a geographic environment that parallels the real world in accordance with one or more embodiments.

[0008] Figure 3 A block diagram of a game module 135 is shown in accordance with one or more embodiments.

[0009] Figure 4 Depicted is an exemplary game interface for a parallel reality game that may be presented on a display of a client device as part of an interface between a player and a virtual world or a parallel reality world in accordance with one or more embodiments.

[0010] Figure 5 is a flow chart describing a process for generating an augmented reality game board in accordance with one or more embodiments.

[0011] FIG. 6A to FIG. 6C A process for placing tiles within traversable space 610 is shown in accordance with one or more embodiments.

[0012] Fig. 7A Depicted are exemplary game interfaces for parallel reality games in accordance with one or more embodiments.

[0013] Figure 7B Depicted is an exemplary game interface for a parallel reality game having a game board with optimized tile placement in accordance with one or more embodiments.

[0014] Figure 7C Depicted is an exemplary game interface for a parallel reality game having a game board with tiles having different surface properties in accordance with one or more embodiments.

[0015] Figure 8 Depicted is an example game interface for a parallel reality game with a procedurally generated game board in accordance with one or more embodiments.

[0016] Fig. 9 is a flow chart describing a method for dynamically generating an augmented reality game board according to one or more embodiments.

[0017] Fig.10 An example computer system suitable for use in generating an augmented reality game board is shown in accordance with one or more embodiments.

[0018] The accompanying drawings and the following description describe certain embodiments by way of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of structures and methods may be employed without departing from the principles described. Reference will now be made to several embodiments, examples of which are shown in the accompanying drawings. DETAILED DESCRIPTION

[0019] Exemplary Location-Based Parallel Reality Gaming System

[0020] Various embodiments are described in the context of a parallel reality game that includes augmented reality content in a virtual world geographic environment that is parallel to at least a portion of a real world geographic environment so that player motion and actions in the real world affect actions in the virtual world, and vice versa. Those of ordinary skill in the art using the disclosure provided herein will appreciate that the described subject matter is applicable to other situations where it is necessary to determine a substantially flat and usable surface area within an image. In addition, the inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and divisions of tasks and functions between and among the components of the system. For example, systems and methods according to various aspects of the present disclosure can be implemented using a single computing device or across multiple computing devices (e.g., connected in a computer network).

[0021] Figure 1 A networked computing environment 100 according to one or more embodiments is shown. The networked computing environment 100 provides players with interactions in a virtual world with a geographic environment parallel to the real world. Specifically, geographic areas in the real world can be directly linked or mapped to corresponding areas in the virtual world. Players can move around in the virtual world by moving to various geographic locations in the real world. For example, the player's location in the real world can be tracked and used to update the player's location in the virtual world. Typically, the player's location in the real world is determined by finding the location of a client device 110, through which the player interacts with the virtual world, and it is assumed that the player is at the same (or approximately the same) location. For example, in various embodiments, if the player's location in the real world is within a threshold distance of the real world location (e.g., ten meters, twenty meters, etc.), the player can interact with a virtual element, and the real world location corresponds to the virtual location of the virtual element in the virtual world. For convenience, various embodiments are described with reference to "player's location", but one of those skilled in the art will understand that these references can refer to the location of the player's client device 110.

[0022] Reference now Figure 2 , which depicts a conceptual diagram of a virtual world 210 according to one or more embodiments, which is parallel to the real world 200, and the real world 200 can serve as a game board for players of a parallel reality game. As shown, the virtual world 210 can include a geographic environment that is parallel to the geographic environment of the real world 200. Specifically, a coordinate range that defines a geographic area or space in the real world 200 is mapped to a corresponding coordinate range that defines a virtual space in the virtual world 210. The coordinate range in the real world 200 can be associated with a town, community, city, campus, region, country, continent, the entire earth, or other geographic area. Each geographic coordinate in the geographic coordinate range is mapped to a corresponding coordinate in the virtual space in the virtual world.

[0023] The position of the player in the virtual world 210 corresponds to the position of the player in the real world 200. For example, player A at position 212 in the real world 200 has a corresponding position 222 in the virtual world 210. Similarly, player B at position 214 in the real world has a corresponding position 224 in the virtual world. When the player moves around in the geographic coordinate range in the real world, the player also moves around in the coordinate range that defines the virtual space in the virtual world 210. Specifically, when the player navigates the geographic coordinate range in the real world, a positioning system (e.g., a GPS system) associated with a mobile computing device carried by the player can be used to track the player's position. Data associated with the player's position in the real world 200 is used to update the player's position in the corresponding coordinate range that defines the virtual space in the virtual world 210. In this way, the player can navigate along a continuous trajectory in the coordinate range that defines the virtual space in the virtual world 210 by simply traveling between the corresponding geographic coordinate ranges in the real world 200, without registering or regularly updating position information at designated discrete locations in the real world 200.

[0024] Location-based games can include multiple game objectives that require players to travel to and / or interact with various virtual elements and / or virtual objects at various virtual locations dispersed in the virtual world. Players can travel to these virtual locations by traveling to corresponding locations of virtual elements or objects in the real world. For example, a positioning system can continuously track the player's location so that when the player continuously navigates the real world, the player also continuously navigates a parallel virtual world. The player can then interact with the various virtual elements and / or objects at the specified locations to achieve or execute one or more game objectives.

[0025] For example, the game objective is for the player to interact with virtual elements 230 located at various virtual locations in the virtual world 210. These virtual elements 230 may be linked to landmarks, geographic locations, or objects 240 in the real world 200. The real world landmarks or objects 240 may be works of art, monuments, buildings, businesses, libraries, museums, or other suitable real world landmarks or objects. Interactions include capturing, claiming ownership, using some virtual items, spending some virtual currency, etc. In order to capture these virtual elements 230, the player must travel to the landmarks or geographic locations 240 in the real world that are linked to the virtual elements 230, and must perform any necessary interactions with the virtual elements 230 in the virtual world 210. For example, Figure 2Player A may have to travel to a landmark 240 in the real world 200 to interact with or capture a virtual element 230 that is linked to a particular landmark 240. Interaction with a virtual element 230 may require action in the real world, such as taking a picture and / or verifying, acquiring, or capturing other information about a landmark or object 240 associated with the virtual element 230.

[0026] The game objective may require the player to use one or more virtual items collected by the player in the location-based game. For example, the player may travel in the virtual world 210, looking for virtual items (e.g., weapons, creatures, power-ups, or other items) that may be useful in completing the game objective. These virtual items may be found or collected by traveling to different locations in the real world 200 or by completing various actions in the virtual world 210 or the real world 200. Figure 2 In the example shown in , a player uses virtual item 232 to capture one or more virtual elements 230. Specifically, the player may deploy virtual item 232 at a location in virtual world 210 near virtual element 230 or within virtual element 230. Deploying one or more virtual items 232 in this manner may result in the capture of virtual element 230 for a particular player or for a particular player's team / faction.

[0027] In one implementation, players may have to collect virtual energy as part of a parallel reality game. Figure 2 As depicted in , virtual energy 250 may be dispersed at different locations in the virtual world 210. Players may collect virtual energy 250 by traveling to corresponding locations of virtual energy 250 in the real world 200. Virtual energy 250 may be used to power virtual items and / or perform various game objectives in the game. A player who loses all virtual energy 250 may be disconnected from the game.

[0028] According to various aspects of the present disclosure, a parallel reality game can be a location-based game of large-scale multiplayer, in which each participant in the game shares the same virtual world. Players can be divided into separate teams or factions, and can work together to achieve one or more game goals, such as capturing or claiming ownership of virtual elements. In this way, a parallel reality game can essentially be a social game that encourages cooperation between players in the game. During a parallel reality game, players from opposing teams can confront each other (or sometimes cooperate to achieve a common goal). Players can use virtual items to attack or hinder the advancement of players of the opposing team. In some cases, players are encouraged to gather at real-world locations for cooperation or interactive events in parallel reality games. In these cases, the game server attempts to ensure that the player does exist physically, rather than deceiving (location).

[0029] Parallel reality games can have various features to enhance and encourage playing games in parallel reality games. For example, players can accumulate virtual currency or another virtual reward (e.g., virtual tokens, virtual points, virtual material resources, etc.), which can be used in the entire game (e.g., to purchase in-game items, exchange other items, make items, etc.). When the player completes one or more game goals and gains experience in the game, the player can advance at various levels. In some embodiments, players can communicate with each other through one or more communication interfaces provided in the game. Players can also obtain enhanced "power" or virtual items that can be used to complete game goals in the game. It should be understood by those skilled in the art of ordinary skill in the art using the disclosure provided herein that, without departing from the scope of the present disclosure, parallel reality games can include various other game features.

[0030] Return to reference Figure 1 , the networked computing environment 100 uses a client-server architecture, where a game server 120 communicates with a client device 110 via a network 105 to provide a parallel reality game to a player at the client device 110. The networked computing environment 100 may also include other external systems, such as sponsor / advertiser systems or business systems. Figure 1 Only one client device 110 is shown, but any number of clients 110 or other external systems may be connected to the game server 120 via the network 105. Furthermore, the networked computing environment 100 may include different or additional elements, and functionality may be distributed between the client devices 110 and the server 120 in a manner different from that described below.

[0031] The client device 110 can be any portable computing device that can be used by a player to interface with the game server 120. For example, the client device 110 can be a wireless device, a personal digital assistant (PDA), a portable gaming device, a cellular phone, a smart phone, a tablet computer, a navigation system, a handheld GPS system, a wearable computing device, a display with one or more processors, or other such devices. In another example, the client device 110 includes a traditional computer system, such as a desktop computer or a notebook computer. Still, the client device 110 can be a vehicle with a computing device. In short, the client device 110 can be any computer device or system that enables a player to interact with the game server 120. As a computing device, the client device 110 can include one or more processors and one or more computer-readable storage media. The computer-readable storage medium can store instructions that cause the processor to perform operations. The client device 110 is preferably a portable computing device that can be easily carried or otherwise transported with the player, such as a smart phone or a tablet computer.

[0032] The client device 110 communicates with the game server 120 to provide the game server 120 with sensory data of the physical environment. The client device 110 includes a camera component 125 that captures two-dimensional image data of the scene in the physical environment where the client device 110 is located. Figure 1 In the embodiment shown in , each client device 110 includes software components, such as a game module 135 and a positioning module 140. The client device 110 may include various other input / output devices for receiving information from a player and / or providing information to a player. Example input / output devices include display screens, touch screens, touch pads, data entry keys, speakers, and microphones suitable for speech recognition. The client device 110 may also include other various sensors for recording data from the client device 110, including but not limited to motion sensors, accelerometers, gyroscopes, other inertial measurement units (IMUs), barometers, positioning systems, thermometers, light sensors, etc. The client device 110 may also include a network interface for providing communication through the network 105. The network interface may include any suitable components for interfacing with one or more networks, including, for example, transmitters, receivers, ports, controllers, antennas, or other suitable components.

[0033] The camera assembly 125 captures image data of the scene of the environment in which the client device 110 is located. The camera assembly 125 can utilize a variety of different light sensors, which have different color capture ranges at different capture rates. The camera assembly 125 may include a wide-angle lens or a telephoto lens. The camera assembly 125 may be configured to capture a single image or video as image data. In addition, the direction of the camera assembly 125 may be parallel to the ground, while the camera assembly 125 is aimed at the horizon. The camera assembly 125 captures image data and shares the image data with the computing device on the client device 110. The image data may be attached with metadata describing other details of the image data, including perception data (e.g., temperature, ambient brightness) or capture data (e.g., exposure, temperature, shutter speed, focal length, capture time, etc.). The camera assembly 125 may include one or more cameras that can capture image data. In one instance, the camera assembly 125 includes one camera, and the camera is configured to capture monocular image data. In another instance, the camera assembly 125 includes two cameras, and the two cameras are configured to capture stereo image data. In various other implementations, the camera assembly 125 includes a plurality of cameras, each camera being configured to capture image data.

[0034] The game module 135 provides an interface for players to participate in the parallel reality game. The game server 120 sends game data to the client device 110 via the network 105 for use by the game module 135 at the client device 110 to provide a local version of the game to players at a location remote from the game server 120. The game server 120 may include a network interface for providing communication via the network 105. The network interface may include any suitable components for interfacing with one or more networks, including, for example, a transmitter, a receiver, a port, a controller, an antenna, or other suitable components.

[0035] The game module 135 executed by the client device 110 provides an interface between the player and the parallel reality game. The game module 135 can present a user interface on a display device associated with the client device 110, which displays a virtual world associated with the game (e.g., an image of a rendered virtual world), and allows the user to interact in the virtual world to perform various game goals. In some other embodiments, the game module 135 presents image data from the real world (e.g., captured by the camera component 125), which is enhanced with virtual elements from the parallel reality game. In these embodiments, the game module 135 can generate virtual content and / or adjust virtual content based on other information received from other components of the client device 110. For example, the game module 135 can adjust the virtual objects to be displayed on the user interface based on the depth map of the scene captured in the image data. In other embodiments, such as when using headphones, the game module 135 can only cause virtual elements to be displayed on a transparent or translucent display so that users perceive virtual elements as being covered on their view of the real world.

[0036] The game module 135 can also control various other outputs to allow the player to interact with the game without the player watching the display screen. For example, the game module 135 can control various audio, vibration, or other notifications, which allows the player to play the game without watching the display screen. The game module 135 can access the game data received from the game server 120 to provide an accurate representation of the game to the user. The game module 135 can receive and process player inputs and provide updates to the game server 120 via the network 105. The game module 135 can also generate and / or adjust the game content to be displayed by the client device 110. For example, the game module 135 can generate virtual elements based on the comparison between one or more terrain grids representing the real world environment around the client device 110 with depth information.

[0037] The positioning module 140 can be any device or circuit device for monitoring the location of the client device 110. For example, the positioning module 140 can determine the actual or relative location by using a satellite navigation positioning system (e.g., GPS system, Galileo positioning system, Global Navigation Satellite System (GLONASS), Beidou Satellite Navigation and Positioning System), an inertial navigation system, a dead reckoning system, based on an IP address, by using triangulation and / or proximity to a cellular tower or Wi-Fi hotspot, and / or other suitable techniques for determining location. The positioning module 140 can also include various other sensors that can help accurately locate the location of the client device 110.

[0038] When the player moves around with the client device 110 in the real world, the positioning module 140 tracks the player's location and provides the player location information to the game module 135. The game module 135 updates the player's location in the virtual world associated with the game based on the actual location of the player in the real world. Therefore, the player can interact with the virtual world simply by carrying or transporting the client device 110 in the real world. Specifically, the player's location in the virtual world can correspond to the player's location in the real world. The game module 135 can provide the player's location information to the game server 120 through the network 105. In response, the game server 120 can verify the location of the client device 110 through various technologies to prevent cheaters from spoofing the location of the client device 110. It should be understood that the location information associated with the player is only used if the player has been notified of the player's location information to be accessed and how to use the location information in the context of the game (for example, in order to update the player's location in the virtual world) after granting permission. In addition, any location information associated with the player will be stored and maintained in a way that protects the player's privacy.

[0039] The game server 120 may be any computing device and may include one or more processors and one or more computer-readable storage media. The computer-readable storage medium may store instructions that cause the processor to perform operations. The game server 120 may include a game database 115 or may communicate with a game database 115. The game database 115 stores game data used in the parallel reality game, which will be served or provided to (multiple) clients 120 via the network 105.

[0040] The game data stored in the game database 115 may include: (1) data associated with the virtual world in the parallel reality game (e.g., image data used to render the virtual world on a display device, geographic coordinates of locations in the virtual world, etc.); (2) data associated with players of the parallel reality game (e.g., player profiles, including but not limited to player information, player experience level, player currency, current player position in the virtual world / real world, player energy level, player preferences, team information, faction information, etc.); (3) data associated with game objectives (e.g., data associated with the current game objective, the state of the game objective, past game objectives, future game objectives, expected game objectives, etc.); (4) data associated with virtual elements in the virtual world (e.g., the position of virtual elements, The game database 115 may include data associated with the game and / or its associated virtual elements (e.g., the type of virtual element, the game objective associated with the virtual element; the corresponding real-world location information for the virtual element; the behavior of the virtual element, the relevance of the virtual element, etc.); (5) data associated with real-world objects, landmarks, and locations linked to virtual world elements (e.g., the location of the real-world object / landmark, the description of the real-world object / landmark, the relevance of the virtual element linked to the real-world object, etc.); (6) game status (e.g., the current number of players, the current state of the game objective, player leaderboards, etc.); (7) data associated with player actions / inputs (e.g., current player location, past player location, player movement, player input, player queries, player communications, etc.); and (8) any other data used, related, or obtained during the implementation of the parallel reality game. The game data stored in the game database 115 may be composed offline or in real time by a system administrator and / or by data received from users / players of the system 100 (e.g., from client devices 110 via network 105).

[0041] The game server 120 may be configured to receive requests for game data from the client devices 110 (e.g., via remote procedure calls (RPCs)) and to respond to these requests via the network 105. For example, the game server 120 may encode the game data in one or more data files and provide the data files to the client devices 110. In addition, the game server 120 may be configured to receive game data (e.g., player positions, player actions, player inputs, etc.) from the client devices 110 via the network 105. For example, the client devices 110 may be configured to periodically send player inputs and other updates to the game server 120, which uses these updates to update the game data in the game database 115 to reflect any and all changed conditions for the game.

[0042] In the embodiment shown, game server 120 includes universal game module 145, commercial game module 150, data collection module 155, event module 160. As mentioned above, game server 120 interacts with game database 115, which can be part of game server 120 or accessed remotely (for example, game database 115 can be a distributed database accessed via network 105). In other embodiments, game server 120 includes different and / or additional elements. For example, game server 120 can include terrain grid management module 320, semantic segmentation module 340, and / or game board generation module 350. In addition, energy supply can be distributed between elements in a different manner from that described. For example, game database 115 can be integrated into game server 120.

[0043] In an embodiment including a universal game module, universal game module 145 hosts parallel reality games for all players, and serves as an authoritative source for the current state of parallel reality games for all players. As a hosting party, universal game module 145 generates game content for presentation to players, for example, via its corresponding client device 110. Universal game module 145 can access game database 115 to retrieve and / or store game data when hosting parallel reality games. Universal game module 145 also receives game data (for example, depth information, player input, player positioning, player action, landmark information, etc.) from client device 110, and merges the received game data into the overall parallel reality game for all players of parallel reality games. Universal game module 145 can also manage the delivery of game data to client device 110 through network 105. Universal game module 145 can also manage the security aspects of client device 110, including but not limited to protecting the connection between client device 110 and game server 120, establishing connection between various client devices 110, and verifying the location of various client devices 110.

[0044] In an embodiment including a commercial game module, the commercial game module 150 may be separate from the universal game module 145 or as part of the universal game module 145. The commercial game module 150 may manage various game features included in the parallel reality game that are linked to commercial activities in the real world. For example, the commercial game module 150 may receive a request from an external system (such as a sponsor / advertiser, a business, or other entity) through the network 105 (via a network interface) to include game features linked to commercial activities in the parallel reality game. The commercial game module 150 may then arrange for these game features to be included in the parallel reality game.

[0045] The game server 120 may also include a data collection module 155. In embodiments that include one data collection module, the data collection module 155 may be separate from or part of the general game module 145. The data collection module 155 may manage various game features included within the parallel reality game that are linked to data collection activities in the real world. For example, the data collection module 155 may modify game data stored in the game database 115 to include game features linked to data collection activities in the parallel reality game. The data collection module 155 may also analyze the data collected by the players in accordance with the data collection activities and provide the data for access by various platforms.

[0046] The event module 160 manages player access to events in the parallel reality game. Although the term "event" is used for convenience, it should be understood that this term does not need to refer to a specified event at a specified location or time. Instead, it can refer to any provision of access-controlled game content, where one or more access criteria are used to determine whether a player can access the content. Such content can be part of a larger parallel reality game that includes game content with less or no access control, or can be an independent, access-controlled parallel reality game.

[0047] The network 105 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof. The network can also include a direct connection between the client device 110 and the game server 120. In general, the communication between the game server 120 and the client device 110 can use any type of wired and / or wireless connection and be carried via a network interface using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML, JSON), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0048] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions and information sent to such systems. A technician in the art will recognize that the inherent flexibility of computer-based systems allows various possible configurations, combinations, and divisions of tasks and functions between and among components. For example, the server process discussed herein can be implemented using a single server or multiple servers working in combination. Databases and applications can be implemented on a single system, or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0049] Figure 3One embodiment of a game module 135 is shown. In the embodiment shown, the game module 135 includes a terrain grid management module 320, a local terrain grid storage 325, a semantic segmentation module 340, and a game board generation module 350. In other embodiments, the game module 135 includes additional or different components. Furthermore, the described functionality may be distributed differently between the components. For example, Figure 3 One or more components of the game module 135 shown in FIG. 1 may alternatively be included in the game server 120 .

[0050] The terrain mesh management module 320 manages the terrain mesh for the game module 135. The terrain mesh is a three-dimensional representation of the environment in the real world, and may include a combination of geometry (e.g., a polygonal mesh), color (e.g., RGB data), texture (e.g., a texture map, a bump map, etc.), other material properties (e.g., reflectivity, friction, density, etc.), and other information describing the real world environment. Alternatively, the terrain mesh may be purely geometric, and other properties (e.g., segmentation data) may be stored separately and mapped to the polygons of the terrain mesh.

[0051] Depending on the embodiment, the terrain mesh may be generated (e.g., by the client device 110 or the game server 120) using processes that require varying degrees of computational complexity and that respectively produce terrain meshes that may be displayed with varying degrees of visual accuracy (e.g., similarity to a real-world environment). For example, the geometry of the terrain mesh may be represented using high-density polygonal meshes (e.g., high polygon counts) that correspond one-to-one or nearly one-to-one with the real-world environment. Furthermore, the terrain mesh used by the client device 110 may include photorealistic or near-photorealistic geometry and textures that represent the real-world environment.

[0052] In some embodiments, terrain mesh management module 320 generates a terrain mesh using scan information describing the real-world environment surrounding client device 110 (e.g., provided by camera assembly 125 or sensor module), or retrieves a previously generated terrain mesh for the real-world environment using the location of client device 110. In embodiments where terrain mesh management module 320 generates the mesh, terrain mesh management module 320 uses scan information captured from a scan of the real-world environment, which scan information was captured using one or more sensors managed by sensor module 310. In embodiments where terrain mesh management module 320 retrieves a previously generated terrain mesh, terrain management module 310 uses the geolocation of client device 110, the scan information, or other information to identify and retrieve a previously generated terrain mesh (e.g., from a local terrain mesh database 325, or from a game server 120).

[0053] The terrain grid management module 320 may store the terrain grid generated or otherwise obtained by the client device 110 locally in the local terrain grid storage 325. The client device 110 may use the terrain grid obtained by the terrain grid management module 320 for relevant processes of the client device 110. For example, the terrain grid management module 320 may determine a terrain grid for a real-world environment that was previously stored locally based on a scan of the real-world environment or the geographic location of the client device 110. In some embodiments, the terrain grid management module 320 may provide the terrain grid to the game server 120 for storage. In some embodiments, the terrain grid management module 320 may obtain a terrain grid or other information describing the real-world environment from the game server 120, such as a terrain grid stored at the game server 120 corresponding to the real-world environment where the client device 110 is located.

[0054] In some embodiments, the terrain grid management module 320 coordinates the collection of scan information for the real-world environment by a player associated with the client device 110 using the client device 110. For example, the terrain grid management module 320 may provide a user interface for display on the client device 110 that displays scan-related information (i.e., a scan interface), such as an image of the real-world environment captured by the camera assembly 125. The scan interface may include one or more interactable objects (e.g., virtual buttons) configured to control the state of the scanning process, such as an interactable object that starts the scan, pauses the scan, ends the scan (e.g., in order to generate a terrain grid using the collected scan-related information). The scan interface may also include various visualizations of the scan-related information, such as visualizations of portions of the terrain grid generated during the scan (e.g., the geometry of the terrain grid) or depth information. In addition, the scan interface may guide the player through the scanning process, such as by displaying a message or visual indicator describing the portion of the real-world environment to be scanned or how much of the real-world environment to be scanned. In some embodiments, multiple client devices 110 simultaneously coordinate the collection of relevant terrain grids by multiple corresponding users. For example, the game server 120 can instruct multiple client devices to collect terrain meshes. The terrain meshes collected by multiple client devices 110 can be combined into a single terrain mesh. In this way, multiple client devices 110 can generate corresponding terrain meshes to jointly plan the real world environment.

[0055] In some embodiments, the terrain mesh management module 320 generates a terrain mesh using one or more mesh generation techniques. Specifically, the terrain mesh generation module 320 can use the scan information to perform one or more mesh generation techniques. The scan information can be obtained directly by the client device 110 (e.g., one or more images captured by the camera assembly 125), or retrieved from another device or remote system (e.g., the game server 120). The terrain mesh management module 320 can use various computer vision techniques to obtain scan information or generate a terrain mesh, such as geometric computer vision techniques or machine learning-based computer vision techniques. For example, using computer vision techniques, the terrain mesh management module 320 can determine depth information (e.g., a point cloud or a dense depth image) that describes a real-world environment. In addition, the terrain mesh management module 320 can use various mesh generation techniques (such as Delaunay triangulation, Ruppert algorithm, frontier algorithm, Poisson reconstruction, etc.) to process the scan information.

[0056] In some embodiments, the terrain mesh generation process used by terrain mesh management module 320 generates a terrain mesh by combining new scan information (e.g., scan information collected by client device 110) and previously collected scan information (e.g., scan information stored on client device 110 or game server 120). In some embodiments, terrain mesh management module 320 generates the terrain mesh in real time or near real time. For example, terrain mesh management module 320 may generate a terrain mesh corresponding to a portion of the real-world environment within milliseconds or seconds after client device 110 captures or receives scan information for the real-world environment.

[0057] In some embodiments, the terrain grid management module 320 determines location information for the real-world environment in which the client device 110 is located. For example, the terrain grid management module 320 may determine the geolocation of the real-world environment based on the geolocation of the client device 110. Alternatively or additionally, the terrain grid management module 320 may use data captured by one or more sensors to determine the location of the client device 110 by comparing the captured data with previously captured sensor data for the real-world environment. For example, the terrain grid management module 320 may use various image recognition techniques to predict the location of the client device 110 from one or more images captured by the device's camera.

[0058] In an embodiment, the terrain mesh management module 320 uses the location information describing the real-world environment to determine whether a terrain mesh previously generated for the real-world environment is stored locally or on the game server 120. For example, a terrain mesh stored in the local terrain mesh storage 325 or on the game server 120 may be stored in association with geolocation and other location-related metadata (e.g., a description of the location, such as an address or location name). In some embodiments, the terrain mesh management module 320 uses the location information describing the real-world environment to retrieve other data associated with the real-world location, such as scan information stored by the client device 110 or the game server 120.

[0059] If the terrain grid management module 320 requests a terrain grid from the game server 120 based on the location information, the game server 120 may determine whether the client device 110 is authorized to access some or all of the stored terrain grids associated with the location information. For example, some terrain grids stored by the game server 120 representing public environments (e.g., parks, entertainment venues, etc.) may be publicly accessible to any of the client devices 110. Other terrain grids stored on the game server 120 may be accessible only to authorized client devices 110, such as terrain grids representing private environments (e.g., a player's home), or terrain grids that are designated as private by the client device 110 when providing the terrain grid to the game server 120.

[0060] In some embodiments, terrain mesh management module 320 dynamically combines some or all of the multiple terrain meshes for the relevant process of client device 110. In these embodiments, terrain mesh management module 320 may combine terrain meshes obtained from one or more sources, such as terrain meshes generated by client device 110, terrain meshes retrieved from local terrain mesh storage 325, or terrain meshes retrieved from game server 120. For example, terrain mesh management module 320 may generate a terrain mesh representing an indoor space (such as a house) where client device 110 is located by retrieving one or more previously generated terrain meshes representing a first portion of the indoor space (e.g., a living room, a dining room, etc.), and generating a new terrain mesh for a second portion of the indoor space to generate one or more other terrain meshes representing portions of the indoor space. The terrain meshes retrieved by terrain mesh management module 320 (e.g., for combination) may be generated by client device 110 or other client devices 110 simultaneously or over a period of time.

[0061] The terrain grid management module 320 may combine (e.g., stitch) one or more retrieved or generated terrain grids and provide the combined terrain grid to other components of the game module 135 for displaying AR content. The combined one or more terrain grids may be generated entirely by the client device 110, or crowdsourced from multiple client devices 110, such as via the network 130. In some embodiments, the terrain grid management module 320 generates or retrieves a terrain grid when the location of the client device 110 changes. For example, after determining that the client device 110 has entered a room, the terrain grid management module 320 may retrieve or generate a terrain grid of a room representing an indoor space.

[0062] The local terrain grid storage 325 includes one or more computer-readable media configured to store information describing a terrain grid. The information stored by the local terrain grid storage 325 may be retrieved or generated by the terrain grid management module 320, or otherwise obtained or determined by the client device 110. The local terrain grid data storage 325 may additionally or alternatively store other information describing the real-world environment, such as scan or location information associated with the real-world environment and obtained by the client device 110.

[0063] The semantic segmentation module 340 uses the semantic segmentation model 345 to match regions of the image with objects or semantics that the semantic segmentation model 340 is trained to recognize. When trained, the semantic segmentation model 345 is configured to predict characteristics associated with an object represented by one or more pixels of the image. For example, the semantic segmentation model 345 is a machine learning model that is trained based on a training data set of images of various surfaces with different characteristics to recognize the type of surface for the object represented in the image. In some embodiments, the semantic segmentation model 345 is trained to recognize materials associated with the surface. For example, the semantic segmentation model 345 is trained to recognize whether the surface is made of concrete, wood, grass, water, snow, etc.

[0064] In some embodiments, the semantic segmentation module 340 receives the terrain mesh from the terrain mesh management module 320 and predicts one or more properties for each polygonal unit in the terrain mesh. For example, the semantic segmentation module 340 predicts a material associated with each polygonal unit of the received terrain mesh. The semantic segmentation module 340 may use an image used to generate the terrain mesh to predict the properties associated with each polygonal unit. For example, the semantic segmentation module 340 may additionally receive one or more images used by the terrain mesh management module 320 to generate the terrain mesh and predict the properties associated with each polygonal unit in the terrain mesh.

[0065] The game board generation module 350 receives a terrain grid and determines placement for a set of tiles within the terrain grid. In some embodiments, the game board generation module 350 places polygonal tiles within the received terrain grid. The polygonal tiles placed by the game board generation module 350 may all have the same shape and size. For example, the game board generation module 350 places square or hexagonal tiles of a specified size within the received terrain grid.

[0066] In some embodiments, the game board generation module 350 identifies traversable spaces within the received grid and determines the placement of polygonal tiles within the identified traversable spaces. Figure 5 Provides a detailed description of an example process for determining placement of polygon tiles within a traversable space.

[0067] The game module 135 can generate AR objects that interact with (e.g., are positioned by or moved according to) the terrain grid of the real-world environment around the client device 110 and / or the tiles placed by the game board generation module 350. In some embodiments, the display module 360 ​​simulates a customized view of the real-world environment by providing an AR interface for display by the client device 110. The AR interface can include virtual objects in a three-dimensional space, for which the position of the virtual objects (e.g., AR objects) in the three-dimensional space is determined using the generated terrain grid of the three-dimensional space. For example, the AR object can be displayed to interact with real-world objects (e.g., tables, chairs, floors, ceilings, walls, etc.) represented by one or more terrain grids. The display module 360 ​​can display the AR object placed on the image captured from the camera assembly 125, or display the AR object alone.

[0068] In some embodiments, the AR interface is associated with a parallel reality game hosted by the game server 120. In this case, the displayed AR object can be an interactive game object (e.g., a game button, a game item, etc.), a game board, a game character (e.g., a game character controllable by a player), a game effect, or any other AR content of the parallel reality game. In other embodiments, virtual objects can be displayed with some transparency or selectivity so that the user can view the real world through the virtual object. In one embodiment, the display module 360 ​​can display a representation of a terrain grid to allow the user to view a game board generated for a location different from the user's current location.

[0069] In addition, where the systems and methods discussed herein access and analyze personal information about a user, or utilize personal information (such as location information), the user may be provided with an opportunity to control whether a program or function collects information, and whether and / or how content is received from the system or other applications. No such information or data is collected or used until a user is provided with meaningful notice of what information is to be collected and how the information is to be used. Information is not collected or used unless the user provides permission, which permission may be revoked or modified by the user at any time. Thus, the user may control how information about the user is collected by an application or system and how that information is used by it. In addition, certain information or data may be processed in one or more ways before being stored or used to remove personally identifiable information. For example, the identity of the user may be processed so that personally identifiable information about the user cannot be determined.

[0070] Example Game Interface

[0071] Figure 4 An exemplary game interface 400 for a parallel reality game according to one or more embodiments is depicted, which can be used as a means for a player to interact with a virtual world such as Figure 2 The portion of the interface between the virtual world 210) or the parallel real world is presented on the display of the client device 110. The game interface includes a display window 410, which can be used to display various aspects of the virtual world and the game, such as the location of virtual items. For example, the display window 410 displays a game board 420 with a plurality of tiles 425. Figure 4 In the example of , the tiles are polygonal tiles. Specifically, the tiles 425 of the game board 420 are square tiles. However, tiles of any other shapes (such as triangular tiles or hexagonal tiles) can be used. In some embodiments, only shapes that can tile planes can be used to generate the game board 420. In addition, the display window 410 displays one or more virtual objects or virtual characters 430. The virtual objects or virtual characters 430 can be placed in one or more tiles 425 of the game board 420.

[0072] The user interface 400 may also display other information, such as game data information, game communication, player information, client position verification instructions and other information associated with the game. For example, the user interface 400 may display player information 415, such as player name, experience level and other information. The user interface 400 may include a menu 450 for accessing various game settings and other information associated with the game. The user interface 400 may also include a communication interface that enables communication between the game system and the player and between one or more players of the parallel reality game.

[0073] According to various aspects of the present disclosure, players can interact with parallel reality games by simply carrying client devices 110 around in the real world. For example, players can play games by simply accessing applications associated with parallel reality games on smart phones and moving around in the real world using smart phones. In this regard, players do not need to continuously watch the visual representation of the virtual world on the display screen to play location-based games. Therefore, the user interface 400 may include multiple non-visual elements that allow users / players to interact with the game. For example, when a player approaches a virtual element or object in the game, or when an important event occurs in the parallel reality game, the game interface may provide audible notifications to the player. The player may use audio control 440 to control these audible notifications. Depending on the type of virtual element or event, different types of audible notifications may be provided to the user / player. The frequency or volume of the audible notification may increase or decrease depending on the proximity of the player to the virtual element or object. Other non-visual notifications and signals may be provided to the user / player, such as vibration notifications or other suitable notifications or signals.

[0074] Those of ordinary skill in the art using the disclosure provided herein will appreciate that many game interface configurations and underlying functionality will be apparent in light of this disclosure. This disclosure is not intended to be limited to any one particular configuration.

[0075] Exemplary Methods

[0076] Figure 5 is a flow chart describing a method 500 for generating an augmented reality game board according to one or more embodiments. The method 500 produces a game board that can be overlaid on an image or video of a scene to implement an augmented reality game. Figure 5 The steps of method 500 may be performed by game module 135 of client device 110. Alternatively, one or more steps of method 500 may be performed by game server 120 or by other components of client device 110. Furthermore, some embodiments may perform steps in parallel, in a different order, or perform different steps.

[0077] In some embodiments, the method 500 begins with the game module 135 of the client device 110 receiving 510 one or more images of a scene. For example, the game module 135 of the client device 110 receives one or more images (such as a series of images from a video) that are captured using the camera component 125 of the client device 110.

[0078] The game module 135 receives 520 a grid (such as a terrain grid) based on one or more images. In some embodiments, the grid is generated by the terrain grid management module 320 based on the received one or more images. Alternatively, the grid is retrieved from the local terrain grid storage 325. In some embodiments, the game module 135 sends one or more images to the game server 120 and receives from the game server 120 a grid for the scene depicted in the one or more images. In other words, the game server can generate or retrieve a pre-generated grid based on one or more images received from the game module 135 of the client device 110 via the network 105.

[0079] In some embodiments, the grid is retrieved based on the location of the client device 110 as determined by the positioning module 140. The terrain grid management module 320 or the game server 120 may identify a grid corresponding to the scene at the current location of the client device and provide the identified grid to the game module 135.

[0080] The game board generation module 350 identifies the traversable space in the scene based on the received grid. In one embodiment, whether the grid unit is traversable is semantic information determined by applying a classifier to the data defining the grid unit (e.g., by the semantic segmentation module 340). In other embodiments, the game board generation module 350 identifies the traversable space by identifying the grid unit parallel to the horizontal plane (e.g., the floor plane). The game board generation module 350 determines the angle between each grid unit and the horizontal plane in the grid unit of the received grid. The horizontal plane can be identified by various techniques, such as using a classifier to identify the floor plane or using a force sensor of the client device 110 to measure the gravity vector. If the angle between the grid unit and the horizontal plane is less than a threshold angle, the game board generation module 350 classifies the grid unit as traversable. In some embodiments, if the standard deviation of the normal vector of the grid unit in the group is less than a threshold, the game board generation module 350 can evaluate multiple grid units connected to each other and regard them as a single grid unit. This can prevent small anomalies in the grid that cause the game board generation module 350 to designate portions of the grid as non-traversable when they are actually traversable for the avatar. The game board generation module 350 can then add the grids that are classified as traversable to the traversable space.

[0081] In some embodiments, in addition to or instead of comparing the angle between the grid cell and the horizontal plane with a threshold angle, the game board generation module 350 determines other metrics, such as the average or median distance between the grid cell and the horizontal plane, and compares the calculated one or more metrics with corresponding thresholds. In some embodiments, the game board generation module 350 determines the average distance between the plane defined by the grid cell and the horizontal plane, and compares the average distance with the threshold distance. Alternatively, the game board generation module 350 determines whether any part of the grid cell exceeds the threshold distance with the distance of the specified horizontal plane. In other embodiments, the game board generation module 350 can determine the distance between the plane defined by the grid cell and the horizontal plane by methods such as random forests, support vector machines (SVM), or temperature-sensitive methods. If the angle between the grid cell and the horizontal plane is less than the threshold angle and the distance between the grid cell and the horizontal plane is less than the threshold distance, the game board generation module 350 classifies the grid cell as traversable. In other embodiments, the distance between grid cells and other planes may be used, such that if the game board is to be placed on a wall, a vertical plane corresponding to the wall may be used, or if the ground in the environment depicted in one or more images is determined to be tilted, a plane angled relative to the horizontal may be used.

[0082] The game board generation module 350 places 540 the polygonal tiles in the identified traversable space. In some embodiments, the game board generation module 350 places tiles of preset shapes and sizes. For example, the game board generation module 350 places square tiles with a perimeter of 1 foot (30 cm) by 1 foot (30 cm). In some embodiments, the shape and size of the tiles are selected based on the game played by the user of the client device 110. In addition, the shape of the tiles can be limited to shapes that can tile a plane. In some embodiments, the game board generation module 350 places tiles of various shapes or sizes in a preset pattern.

[0083] In some embodiments, the game board generation module 350 may also use the avatar configuration to identify traversable surfaces. As described above, the game board generation module 350 divides the terrain grid into tiles. The game board generation module 350 determines the height measurement of each tile (e.g., the average or median height of the portion of the topological grid on which the tile is placed), and uses a proxy or sliding window to identify adjacent groups of tiles. The proxy can be a avatar selected by the player, or can be a avatar used to determine connected tiles.

[0084] The game board generation module 350 can use the characteristics of the virtual character to identify the surface that can be traversed by the virtual character selected by the player. The surface is a collection of adjacent tiles, for which the virtual character can reach all tiles by traveling between adjacent tiles, and the height difference of the adjacent tiles is not much greater than the step or jump height for the virtual character. For example, a virtual character with a larger step height (e.g., height change) may be able to climb the steps of a given step height, while a virtual character with a smaller step height may not be able to climb the steps of a given step height. In another example, the virtual character configuration (e.g., jump height / distance) can allow the virtual character to jump over the gap between disconnected surfaces. In some embodiments, the virtual character configuration can determine the size of the tile. In addition, the game board generation module 350 can identify a multi-level surface. For example, a first surface (e.g., a floor surface) and a second surface (e.g., a desktop surface) located above the first surface.

[0085] FIG. 6A to FIG. 6C 6 shows a process for placing tiles within traversable space 610 according to one or more embodiments. Fig. 6A , to place the polygonal tile, the game board generation module 350 determines the placement of the initial tile 620. The game board generation module 350 can determine the location of the initial tile by identifying the edge or boundary of the traversable space 610, and placing the initial tile 620 at a set distance from the boundary of the traversable space 610.

[0086] The game board generation module 350 identifies the locations of the additional tiles based on the placement of the initial tiles 620. For example, Figure 6B As shown in , for each edge of the initial tile 620, the game board generation module 350 determines whether the additional tile placed adjacent to the initial tile along the edge of the initial tile overlaps the non-traversable space 615. If the additional tile does not overlap the non-traversable space 615, the game board generation module 350 adds the additional tile to the game board. Conversely, if the additional tile overlaps the non-traversable space 615, the game board generation module 350 excludes the additional tile from the game board. For example, in Figure 6B In the diagram of FIG. 6 , the additional tile 630 does not overlap the non-traversable space 615 (i.e., it is completely contained within the traversable space 610), and the additional tile 635 overlaps the non-traversable space. As such, the game board generation module 350 adds the additional tile 630 that does not overlap the non-traversable space 615 to the game board. In other embodiments, the game board generation module 350 adds tiles that overlap the non-traversable area by less than a threshold amount (e.g., less than 5%).

[0087] The game board generation module 350 recursively adds more additional tiles by repeating this process for each additional tile added to the game board. Figure 6C As shown in the figure of , the game board generation module 350 determines whether the additional tile 640 adjacent to the tile 630 overlaps with the non-traversable space 615, and if the additional tile 640 does not overlap with the non-traversable space 615 (or the amount of overlap is less than the threshold amount), the additional tile 640 is added. Similarly, for each tile in the tiles 640 added to the game board, the game board generation module 350 determines whether the additional tile 650 overlaps with the non-traversable space 615, and if the additional tile 650 does not overlap with the non-traversable space 615 (or the amount of overlap is less than the threshold amount), the additional tile 650 is added. Repeat this process until no more additional tiles can be added to the game board. In some embodiments, the game board generation module 350 establishes the game board during the initialization phase of the parallel reality game. Alternatively or additionally, the game board generation module 350 can continuously update the game board as the user / player moves. The game board generation module 350 can obtain an updated terrain grid (e.g., a terrain grid of a new scene visited by a player as the player moves around the game area) and expand the game board by adding additional tiles to the game board to cover the traversable space of the new scene visited by the player. In other words, the game board generation module 350 identifies the traversable area of ​​the new scene and appends tiles to the edge of the board game to expand the board game in the direction of the new scene.

[0088] In some embodiments, the game board generation module 350 uses other algorithms to determine the positions of polygonal tiles within the traversable space. For example, the game board generation module 350 may overlap a tile grid over the traversable space and add tiles to the game board from a grid that is completely enclosed within the traversable space. Alternatively, the game board generation module 350 may generate a game board that completely covers the traversable space and removes tiles that overlap the non-traversable space from the game board.

[0089] In some embodiments, the efficiency of tile placement or the efficiency of the game board (i.e., the ratio between the area covered by the game board and the area of ​​traversable space) depends on the initial tile placement. In some embodiments, when the process for generating a game board is initiated, the initial tile placement is determined based on the initial viewpoint. This may result in non-optimized tile placement.

[0090] Fig. 7A An exemplary game interface 700A for a parallel reality game with a game board having non-optimized tile placement is depicted in accordance with one or more embodiments. Fig. 7AAs shown in the game interface 700A of FIG. 4 , the tiles 425 of the game board 420A are angled relative to the boundaries of the traversable area 610. Thus, the efficiency of the game board 420A is reduced.

[0091] The game board generation module 350 can optimize the placement of the tiles 425 of the game board 420 to increase the efficiency of the game board. For example, the game board generation module 350 determines the angle and placement of the initial tiles 620 to increase the efficiency of the game board 420. In some embodiments, the game board generation module 350 generates a plurality of game boards, each of which has initial tiles with different angles and positions. Then, the game board generation module 350 calculates the efficiency of each game board in the generated game board, and selects the game board with maximum efficiency. In other embodiments, tile positioning and size, tile angle, distance and restrictions (e.g., characteristics and overlapping surfaces) can be modeled as discrete optimization problems (e.g., mixed integer problems). Algorithms such as branch and bound algorithms can be used to solve this discrete optimization problem.

[0092] Figure 7B Depicted is an exemplary game interface 700B for a parallel reality game with a game board having optimized tile placement according to one or more embodiments. Figure 7B The game interface 700B shown in FIG. Fig. 7A The portion of the traversable space 610 covered by the game board 420a is compared to that covered by Figure 7B The portion of the traversable space 610 covered by the game board 420B is increased. For example, in order to increase the efficiency of the game board 420B, the game board generation module 350 rotates the game board tiles 425. By rotating the game board tiles 425, Figure 4 The tiles of B's ​​game board 420B are aligned with one or more boundaries of the traversable space 610 .

[0093] Return to reference Figure 5 , the game board generation module 350 may assign 550 characteristics to each polygonal tile of the game board. For example, the game board generation module 350 may determine the characteristics for the tile based on the surface material at the location corresponding to the tile. The game board generation module 350 may assign one or more characteristics to the tile for the grid cells corresponding to the location associated with the tile based on the output of the semantic segmentation module 340.

[0094] Figure 7C Depicted is an exemplary game interface 700C for a parallel reality game having a game board with tiles having different surface characteristics in accordance with one or more embodiments. Figure 7CThe game interface 700C of FIG. 700 shows a game board 710 having a first portion and a second portion, the first portion including a first tile set 725 having a first characteristic, and the second portion including a second tile set 735 having a second characteristic. The first tile set 725 is arranged on a first surface 720 made of a first material, and the second tile set 735 is arranged on a second surface 730 made of a second material. Specifically, in Figure 7C In the example of , the first tile set 725 is assigned a first property corresponding to the first material of the first surface 720, and the second tile set 735 is assigned a second property corresponding to the second material of the second surface 730. Additionally or alternatively, potential tiles with certain properties may not be placed if they are placed. For example, potential tiles that overlap with the first material (e.g., water) may not be placed so that these parts of the image (and the physical environment) are not available for placement of AR characters or other objects.

[0095] For example, first surface 720 may be a concrete surface, while second surface 730 may be a grass surface. Semantic segmentation module 340 may analyze an image depicting first surface 720, and may assign a property indicating that the grid cell corresponds to a concrete surface to grid cells that overlap with first surface 720. Similarly, semantic segmentation module 340 may analyze an image depicting second surface 730, and may assign a property indicating that the grid cell corresponds to a grass surface to grid cells that overlap with second surface 730.

[0096] The game board generation module 350 then assigns tile characteristics to each of the tiles of the game board based on the grid cells that overlap with the tile. In some embodiments, the tile characteristics are determined based on the game played by the user of the client device 110. For example, the game played by the user of the client device may configure the game board generation module 350 to assign tile characteristics corresponding to ice to tiles that overlap with concrete surfaces, and to assign tile characteristics corresponding to water to tiles that overlap with grass surfaces. Thus, in a parallel reality game using the generated game board, tiles that overlap with the first surface 720 appear to be made of ice, and tiles that overlap with the second surface 730 appear to be made of water.

[0097] In some embodiments, one or more tiles overlap with multiple surfaces, each surface being associated with different characteristics. In this case, the game board generation module 350 can assign characteristics to the tiles based on the amount of overlap between the tiles and each of the surfaces. For example, the game board generation module 350 determines the amount of overlap between the tiles and each of the surfaces, and assigns characteristics associated with the surface, which has the maximum amount of overlap with the tiles. For example, the game board 710 includes a tile 750, which has a first portion overlapping with the first surface 720, and a second portion overlapping with the second surface 730. The game board generation module 350 determines the area of ​​the first part and the area of ​​the second part, and compares the area of ​​the first part with the area of ​​the second part. If the area of ​​the first part is greater than the area of ​​the second part, the game board generation module 350 assigns the characteristics corresponding to the first surface 720 to the tile 750. Alternatively, if the area of ​​the second part is greater than the area of ​​the third part, the game board generation module 350 assigns the characteristics corresponding to the second surface 730 to the tile 750.

[0098] exist Figure 7C In the example of , the second portion of the tile 750 overlapping the second surface 730 is larger than the first portion of the tile 750 overlapping the first surface 720. As such, the tile 750 is assigned a characteristic corresponding to the second surface 730. In addition, the game board 710 includes a tile 755 having a first portion overlapping the first surface 720 and a second portion overlapping the second surface 730. Here, the first portion of the tile 755 overlapping the first surface 720 is larger than the second portion of the tile 755 overlapping the second surface 730. As such, the tile 755 is assigned a characteristic corresponding to the first surface 720.

[0099] Procedurally generated augmented reality game board

[0100] Figure 8 An exemplary game interface 800 for a parallel reality game according to one or more embodiments is depicted, which can be used as a means for players to interact with a virtual world such as Figure 2 The portion of the interface between the virtual world 210) or the parallel real world is presented on the display of the client device 110. Figure 8 The game interface 800 uses a procedurally generated game board or a dynamic game board that expands when a player performs certain actions, when a player moves around the game board, or when events occur in the virtual world.

[0101] In some embodiments, the game board generation module 350 generates a game board around the area 810 near the player. The portion of the game board around the area 810 can be generated in response to a request for a game board (e.g., generated by a player requesting to perform a certain action or activate a specified mode within the game), or can be automatically generated (e.g., on a client device 110 in a location for which a game board has not yet been generated). In one embodiment, the game board generated around the area 810 near the player is generated based on one or more images captured by using the camera assembly 125 of the player's client device 110. For example, based on one or more images captured using the camera assembly 125 of the player's client device 110, a height field or grid is generated, and a traversable area in the area 810 near the player is identified. Based on the identified traversable area, the game board generation module generates a game board. Alternatively, location data (e.g., GPS data) for the client device 110 can be used to retrieve a pre-generated height field or terrain grid for the adjacent client device (e.g., from the game server 120).

[0102] The game board generation module 350 can extend the game board to include an area 820 that is away from (e.g., beyond a threshold distance) the user's location. In some embodiments, the area 820 includes an area visible from the user's location. However, the area 820 may be located at a distance that causes the accuracy or confidence level in the identification of traversable areas using the height field and terrain mesh generated from the camera data to decrease below a certain threshold. The generation of the game board in the area 820 can be performed based on a combination of information collected from the player's client device 110 (such as one or more images captured by the camera assembly 125 of the client device 110, depth information captured by the depth sensor of the client device, etc.), as well as information received from external sources (such as 3D map information received from the game server 120).

[0103] The game board generation module 350 may also expand the game board to include areas 830 that are not visible to the client device 110. For example, the areas 830 may include areas that are blocked by buildings or other large objects. Figure 8 In the example of , area 830 includes a road behind a group of buildings. In one embodiment, the game board generation module 350 generates a game board for area 830 based on 2D map data (e.g., retrieved from the game server 120).

[0104] Fig. 9 9 is a flow chart describing a method 900 for dynamically generating an augmented reality game board in accordance with one or more embodiments. The method 900 produces a game board that can be used to place virtual objects or characters in a virtual world of an augmented reality game. Fig. 9The steps of method 900 may be performed by the game module 135 of the client device 110. Alternatively, one or more steps of method 900 may be performed by the game server 120, or by other components of the client device 110. In addition, some embodiments may perform steps in parallel, in a different order, or perform different steps.

[0105] In some embodiments, the method 900 begins with the game module 135 of the client device 110 identifying the location of the player. In some embodiments, the game module 135 determines the location of the player by determining the location of the player's client device. For example, the game module 135 can determine the location of the player's client device 110 using global positioning system (GPS) signals, using video positioning system (VPS) analysis, or a combination thereof.

[0106] The game module 135 determines 920 the topology around the player and generates 930 the game board based on the determined topology around the player. The game module 135 may determine the topology around the player and use Figure 5 For example, the game module 135 may generate a grid or determine a height field based on one or more images (or videos) captured using the camera assembly 125 of the player's client device 110, and generate a game board based on the grid or height field.

[0107] The game module 135 may identify 940 one or more areas outside of the existing game board in which to extend the game board beyond the boundaries of the current game board. For example, the game logic of the parallel reality game may choose to spawn an object or character at a location outside the current extent of the game board. In some embodiments, the game module 135 receives an indication of a location for extending the game board.

[0108] Game module 135 receives 950 information about the identified area outside the current game board. Specifically, game module 135 receives information about the area that will be used to expand the current game board. In some embodiments, game module 135 receives information about the topological structure of the area outside the current game board from game server 120. Alternatively or additionally, game module 135 receives information about the topological structure of the area outside the current game board from a third-party system. Game module 135 can receive different information or additional information about the area outside the game board, such as semantic characteristics. In some embodiments, game module 135 sends a request to identify one or more areas, and receives information about these areas. In other words, game module 135 sends a request to identify the area that will be used to expand the current game board, and receives information about these areas (e.g., topological structure and other geographic information).

[0109] In some embodiments, topological structure information comprises the height information of the point in one or more regions identified. In other embodiments, topological structure information comprises the grid or 3D map information about one or more regions identified. In some embodiments, game module 135 receives different topological structure information for different regions. For example, when the player uses parallel reality game to explore the world, game server 120 analyzes the topological structure of the position of the player using parallel reality game, and stores the topological structure information of these positions. When the game module 135 of the client device of the player requests the topological structure information of the region previously analyzed, the game server 120 sends the stored topological structure information for the region generated based on the previous analysis to the game module 135 of the client device. However, if the game module 135 of the client device of the player requests the topological structure information of the region not previously analyzed (or not analyzed within the threshold time amount) by the game server 120, the game server 120 sends the less accurate topological structure information for the requested region to the game module 135 of the client device 120. For example, the game module 135 may send 3D map information or satellite image information (or information derived therefrom) about the requested area to the game module 135 of the player's client device.

[0110] Based on the received topology information, the game module 135 expands 960 the game board. For example, the game module 135 identifies traversable regions within the identified area for expanding the game board, and appends the identified traversable regions to the traversable regions of the current game board.

[0111] In some embodiments, characteristics of traversable areas within the identified area for extending the game board are determined based on the topological information received by the game module 135. In addition, the characteristics of the traversable areas are additionally determined based on the distance between the identified area for extending the game board and the location of the client device. Specifically, the algorithm for determining the characteristics of the traversable areas can depend on the distance between the identified area for extending the game board and the location of the client device. Alternatively, the characteristics of the traversable areas are determined based on the confidence level of the accuracy of the received topological information. For example, the algorithm for determining the characteristics of the traversable areas depends on the confidence level of the accuracy of the received topological information.

[0112] In some embodiments, characteristics for a traversable area (such as height values ​​of points within the traversable area and semantics of points within the traversable area) are determined based on a weighted average or weighted combination of multiple sources. Each of the sources can be weighted based on the confidence level of the corresponding source. In some embodiments, the confidence level of topological information for an area received from certain sources depends on the distance between the player's client device and the area. For example, topological information for an area determined based on an image captured by a camera component 125 of a client device 110 or sensor data captured by a client device 110 depends on the distance between the client device 100 and the area. As the distance between the area and the client device 110 increases, the confidence level of the topological information determined based on the image captured by the camera component 125 of the client device 110 or the sensor data captured by the client device 110 decreases.

[0113] In other embodiments, the game module 135 periodically updates the topology information of the traversable area, and updates the characteristics of the traversable area based on the updated topology information. For example, as the player moves around the world, the game module 135 of the player's client device 110 captures images or sensor data of the client device's surroundings. When the player moves to a second location, the client device 110 captures images or sensor data of the second location's surroundings. Based on the captured images or sensor data of the second location's surroundings, the game module 135 updates the topology information of the area around the second location. Based on the determined topology information, the game module 135 updates the characteristics of the game board corresponding to the area around the second location.

[0114] In this way, the game module 135 is able to extend the game board for the player beyond the player's surroundings by using topology information from a source that is less accurate than the topology information determined based on image or sensor data captured by the client device. As the player approaches areas corresponding to locations where the game board was generated using a less accurate source, the characteristics of the game board at those locations can be updated based on information newly captured by the camera assembly or sensor of the client device 110.

[0115] In some embodiments, this additionally allows the game module 135 to extend the game board to locations that are not visible by the camera assembly or sensor of the client device. For example, the game module 135 is able to extend the game board to a location behind a building or other large object. The game module 135 can use topological information from less accurate sources to determine the characteristics of the game board at locations that are not visible by the client device, and when the player moves to a second location that is not visible by the client device and becomes visible, the characteristics of the game board at these locations can be updated.

[0116] In various embodiments, the game module 135 determines the game board at multiple distances with corresponding levels of detail / granularity. The example embodiment described above has three game board areas with corresponding levels of detail, but it should be understood that any number of areas can be used. Each area is determined from corresponding geographic data, such as camera images, 3D map data, 2D map data, and other geolocation data (such as weather reports, population density, user counts in the previous time period, etc.). Image segmentation can also be used to apply features to the game board based on images of corresponding parts of the real-world environment. For example, based on a segmentation classifier applied to an image of a building captured by the client device 110, a label such as "building", "concrete", "office", etc. can be assigned to a portion of the terrain grid corresponding to the building.

[0117] Parallel reality games can dynamically place virtual elements (e.g., objects or roles) in programmatically generated game boards according to the semantic characteristics of game boards. The game can specify one or more characteristics of the desired position for the virtual element, and query the game board for the position that meets the desired characteristics. In some embodiments, multiple possible positions with matching scores can be returned, and the matching score indicates how close the identified position is to the confidence level of the desired characteristics and / or position suitability. The search for a suitable position can start from the area around the client device, and the sensor of the client device can be used to generate a semantic topological grid (e.g., extending to a distance of about five meters from the client device) for the area. If a suitable position is not found, the search can be extended to the next area, which is farther from the client device but still visible to the camera of the device. The area can be modeled using an image captured by the client device, a previous image captured by other devices, 3D map data, 2D map data, and / or a combination of any other available information. If the second area does not have a suitable position either, the client device can be searched for invisible and / or far enough to be effectively invisible in the third area. The suitability of the position in the third area can be determined from the 2D map data and other geographic information available to the client device.

[0118] To give a specific example, a virtual bird may be intended to be placed in a tree. In order to find a location for the virtual bird, the client device may first search a segmentation map generated from a camera image captured by the client device to identify potential trees. If a suitable tree is found, the bird may be placed in the tree. Conversely, if no tree is found, the search may be extended to a further area visible to the camera of the client device. The client device may identify portions of the image corresponding to the forest in the background, for which individual trees cannot be accurately segmented, and place the bird in the forest. As the player moves toward the forest, a new terrain map with semantic information may be generated that can resolve individual trees and place the bird in a suitable tree. Finally, if the tree is not visible to the camera at all, the client device may search for a forested area on a 2D map and place the bird in the forested area. The client device may present an arrow or other indicator to point the player to the bird. As before, as the player approaches the bird, the game board may be updated more accurately, and the bird is eventually placed in the designated tree.

[0119] In one embodiment, virtual items can be located by searching for locations with specified objects, materials, or other properties. Examples include placing virtual objects on physical objects (e.g., tables, benches, chairs, beds, etc.), placing objects on surfaces of specific materials (e.g., water, grass, concrete, sand, etc.), or placing objects in certain types of locations (e.g., indoor public spaces, parks, roads, fields, etc.). More complex properties for desired locations can also be specified. For example, the game can query for objects that are large enough for the virtual character to hide behind. If the virtual character is a rabbit, a nearby small bush in the topology map generated by the client device may be a suitable location. Because the location and shape of the bush can be determined relatively accurately from the camera image, a realistic animation of the virtual rabbit running behind the bush or even peering out from behind can be provided. Conversely, if the virtual character is The location may then be behind a building in the background of the image captured by the device camera. In this case, because the building is far away, its location does not need to be known with high precision. By scaling the virtual character based on a relatively inaccurate estimate of the distance to the building, a believable experience can be generated.

[0120] To give another example, when a player approaches a skittish avatar, it may attempt to run away and hide. The query may be for suitable hiding locations, which may be evaluated using a series of factors, such as the percentage of directions in which the location is visible from a predetermined distance (e.g., five meters), the lighting level of the location, the number of other players or avatars within a threshold distance of a potential hiding spot (e.g., ten meters), and the material the location is made of.

[0121] Exemplary Computing System

[0122] Fig.10 is an example architecture of a computing device according to an embodiment. Fig.10 Depicted is a high-level block diagram showing the physical components of a computer that may be used as part or all of one or more of the entities described herein, but according to embodiments, the computer may have Fig.10 Additional components, fewer components, or variations of components provided in. Although Fig.10 A computer 1000 is depicted, but the diagram is intended as a functional description of various features that may be present in a computer system, rather than as a schematic diagram of the architecture of the implementations described herein. In practice, and as recognized by those of ordinary skill in the art, items shown individually may be combined, and some items may be separated.

[0123] Fig.10 1004. Memory 1006, storage device 1008, keyboard 1010, graphics adapter 1012, pointing device 1014, and network adapter 1016 are also coupled to chipset 1004. Display 1018 is coupled to graphics adapter 1012. In one embodiment, the functionality of chipset 1004 is provided by memory controller hub 1020 and I / O hub 1022. In another embodiment, memory 1006 is coupled directly to processor 1002, rather than chipset 1004. In some embodiments, computer 1000 includes one or more communication buses for interconnecting these components. One or more communication buses optionally include circuitry (sometimes referred to as a chipset) that interconnects and controls communications between system components.

[0124] The storage device 1008 is any non-transitory computer-readable storage medium, such as a hard drive, a compact disk read-only memory (CD-ROM), a DVD, or a solid-state storage device or other optical storage device, a magnetic cassette, a magnetic tape, a magnetic disk storage device or other magnetic storage device, a magnetic disk storage device, an optical disk storage device, a flash memory device, or other non-volatile solid-state storage device. Such storage devices 1008 may also be referred to as persistent memory. Pointing device 1014 may be a mouse, a trackball, or other type of pointing device and is used in combination with keyboard 1010 to input data into computer 1000. Graphics adapter 1012 displays images and other information on display 1018. Network adapter 1016 couples computer 1000 to a local area network or a wide area network.

[0125] The memory 1006 holds instructions and data used by the processor 1002. The memory 1006 may be a non-persistent memory, examples of which include high-speed random access memory such as DRAM, SRAM, DDR RAM, ROM, EEPROM, flash memory.

[0126] As is known in the art, the computer 1000 may have Fig.10 1000 and / or other components. In addition, the computer 1000 may lack some of the components shown. In one embodiment, the computer 1000 acting as a server may lack the keyboard 1010, the pointing device 1014, the graphics adapter 1012, and / or the display 1018. In addition, the storage device 1008 may be local and / or remote from the computer 1000 (such as contained in a storage area network (SAN)).

[0127] As known in the art, computer 1000 is suitable for executing computer program modules for providing the functions described herein. As used herein, the term "module" refers to computer program logic that is used to provide a specified function. Therefore, a module can be implemented in hardware, firmware, and / or software. In one embodiment, the program module is stored on storage device 1008, loaded into memory 1006, and executed by processor 1002.

[0128] Additional Notes

[0129] Some parts of the above description describe embodiments according to algorithmic processes or operations. These algorithmic descriptions and representations are usually used by those skilled in the art of data processing to effectively convey the essence of their work to other skilled in the art. Although these operations are described in terms of function, calculation, or logic, these operations are understood to be implemented by a computer program, which includes instructions for being executed by a processor or equivalent circuit, microcode, etc. In addition, without losing generality, it is sometimes convenient to refer to the arrangement of these functional operations as a module.

[0130] As used herein, any reference to "one embodiment" or "an embodiment" means that a particular element, feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment. The phrase "in one embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment.

[0131] Some embodiments may be described using the expressions "coupled" and "connected" and their derivatives. It should be understood that these terms are not intended to be synonymous with each other. For example, some embodiments may be described using the term "connected" to indicate that two or more elements are in direct physical or circuit contact with each other. In another example, some embodiments may be described using the term "coupled" to indicate that two or more elements are in direct physical or circuit contact with each other. However, the term "coupled" may also mean that two or more elements are not in direct contact with each other, but still work together or interact with each other. Embodiments are not limited in this context.

[0132] As used herein, the terms "comprises," "comprising," "including," "consisting of," "having," "having," or any other variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. In addition, unless expressly stated to the contrary, "or" refers to an inclusive or, not an exclusive or. For example, a condition A or B is satisfied by any of the following: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and both A and B are true (or both exist).

[0133] In addition, the use of "a" or "an" is adopted to describe elements and components of the embodiments. This is merely for convenience and to give a general sense of the present disclosure. The description should be understood to include one or at least one, and the singular also includes the plural unless it is obvious that it is otherwise meant.

[0134] Upon reading this disclosure, those skilled in the art will appreciate additional alternative structural and functional designs for systems and processes for determining or using repeatability of points of interest. Thus, while specific embodiments and applications have been shown and described, it will be understood that the described subject matter is not limited to the precise construction and components disclosed herein, and that various modifications, changes, and variations may be made to the arrangement, operation, and details of the disclosed methods and apparatus that are apparent to those skilled in the art.

Claims

1. A computer-implemented method comprising: receiving one or more images of a scene from a camera of a mobile device; obtaining a terrain grid of the scene based on the one or more images, the terrain grid comprising a plurality of cells; identifying traversable space within the scene from the terrain mesh; as well as A position of each polygonal tile of a plurality of polygonal tiles in the identified traversable space is determined, wherein the plurality of polygonal tiles form a game board.

2. The computer-implemented method of claim 1 , wherein identifying the traversable space comprises, for each cell of the terrain grid: Determines the angle between the cell and a specified plane, comparing the angle between the cell and the specified plane to a threshold angle, and In response to the angle between the cell and the designated plane being less than the threshold angle, allocating the cell to the traversable space.

3. The computer-implemented method of claim 2, wherein identifying the traversable space further comprises, for each cell of the terrain grid: determining the distance between the cell and the specified plane, comparing the distance between the cell and the specified plane to a threshold distance, and In response to the angle between the cell and the designated plane being less than the threshold angle and the distance between the cell and the designated plane being less than the threshold distance, allocating the cell to the traversable space.

4. The computer-implemented method of claim 1 , wherein identifying the traversable space further comprises: A traversable space for the avatar is identified based on the avatar configuration.

5. The computer-implemented method of claim 1 , wherein determining a position of each polygonal tile in a plurality of polygonal tiles in the identified traversable space comprises, for each tile in a grid of tiles: determining whether the tile is contained within the traversable space, and In response to determining that the tile is contained within the traversable space, the tile is added to the game board.

6. The computer-implemented method of claim 1 , wherein determining a position of each polygonal tile of a plurality of polygonal tiles in the identified traversable space comprises: Identifying the location of the initial tile within the traversable space; determining whether a second tile adjacent to the initial tile overlaps with a non-traversable space surrounding the traversable space; as well as In response to determining that the second tile does not overlap the non-traversable space, the second tile is added to the game board. The computer-implemented method of claim 6 , wherein the second tile has the same shape and size as the initial tile.

8. The computer-implemented method of claim 1 , further comprising, for each polygonal tile placed on the game board: determining a property associated with each cell that overlaps the polygonal tile, and A characteristic to be assigned to the polygonal tile is determined based on the determined characteristic associated with each cell that overlaps the polygonal tile.

9. The computer-implemented method of claim 1 , further comprising: A gaming interface is presented to a user that overlays a representation of the game board including the plurality of polygonal tiles with a second set of images of the scene captured by the camera of the mobile device.

10. The computer-implemented method of claim 1 , wherein obtaining a terrain mesh of the scene comprises: The terrain mesh of the scene is generated based on the one or more images of the scene received from a camera of a mobile device.

11. The computer-implemented method of claim 1 , wherein obtaining a terrain mesh of the scene comprises: A pre-generated terrain mesh is retrieved based on the location of the client device.

12. A non-transitory computer-readable storage medium configured to store instructions that, when executed by a processor, cause the processor to: receiving one or more images of a scene from a camera of a mobile device; obtaining a terrain grid of the scene based on the one or more images, the terrain grid comprising a plurality of cells; identifying traversable space within the scene from the terrain mesh; as well as A position of each polygonal tile of a plurality of polygonal tiles in the identified traversable space is determined, wherein the plurality of polygonal tiles form a game board.

13. The non-transitory computer-readable storage medium of claim 12, wherein the instructions for identifying a traversable space cause the processor to, for each cell of the terrain grid: Determines the angle between the cell and a specified plane, comparing the angle between the cell and the specified plane to a threshold, and In response to the angle between the cell and the designated plane being less than the threshold, allocating the cell to the traversable space.

14. The non-transitory computer-readable storage medium of claim 13, wherein the instructions for identifying a traversable space further cause the processor to, for each cell of the terrain grid: determining the distance between the cell and the specified plane, comparing the distance between the cell and the specified plane to a threshold distance, and In response to the angle between the cell and the designated plane being less than the threshold angle and the distance between the cell and the designated plane being less than the threshold distance, allocating the cell to the traversable space.

15. The non-transitory computer-readable storage medium of claim 13, wherein the instructions for determining a position of each polygonal tile in a plurality of polygonal tiles in the identified traversable space cause the processor to, for each tile in a grid of tiles: determining whether the tile is contained within the traversable space, and In response to determining that the tile is contained within the traversable space, the tile is added to the game board.

16. The non-transitory computer-readable storage medium of claim 13, wherein the instructions for determining a position of each polygon tile of a plurality of polygon tiles in the identified traversable space cause the processor to: Identifying the location of the initial tile within the traversable space; determining whether a second tile adjacent to the initial tile overlaps with a non-traversable space surrounding the traversable space; and In response to determining that the second tile does not overlap the non-traversable space, the second tile is added to the game board.

17. The non-transitory computer-readable storage medium of claim 13, wherein the instructions further cause the processor to, for each polygon tile placed on the game board: determining a property associated with each cell that overlaps the polygonal tile, and A characteristic to be assigned to the polygonal tile is determined based on the determined characteristic associated with each cell that overlaps the polygonal tile.

18. The non-transitory computer-readable storage medium of claim 13, wherein the instructions further cause the processor to: A gaming interface is presented to a user that overlays a representation of the game board including the plurality of polygonal tiles with a second set of images of the scene captured by the camera of the mobile device.

19. A computer-implemented method comprising: Determining a request to identify an area outside of the game board for extending a game board of a parallel reality game; receiving topology information for the area outside of the game board from a plurality of sources; determining a confidence level of the received topology information from each of the plurality of sources; as well as The game board is expanded based on a combination of the received topology information, wherein the topology information is combined based on the determined confidence level for each of the plurality of sources.

20. A computer-implemented method comprising: determining characteristics associated with locations on a game board of a parallel reality game; receiving a query for a location on the game board, the query specifying desired criteria; identifying, in response to the query, candidate locations on the game board by performing a search of the locations on the game board for locations that satisfy the desired criteria specified by the query; For each candidate position among the candidate positions, calculating a suitability score; as well as A location among the candidate locations having a suitability score above a threshold is selected.

Images