Maintaining Object Alignment in 3D Map Segments

Abstract

A method for determining the position of a virtual object will be described. The location of the client device is determined, and based on the determined position, a set of map segments is obtained. The virtual object to be displayed on the client device is determined. A relationship vector between the virtual object and each map segment of the obtained set of map segments is obtained. Each relationship vector is weighted based on the object parameters of the virtual object. The position at which to display the virtual object is determined based on the weighted relationship vectors. The virtual object is provided to be displayed on the client device at the determined position.

Description

Technical Field

[0001] The present disclosure generally relates to augmented reality (AR), and more particularly to maintaining spatial relationships between different sets of three-dimensional (3D) maps in AR applications.

Background Art

[0002] Parallel reality applications have a virtual geography that mirrors at least a portion of the real-world geography. To create and maintain a virtual geography similar to the real-world geography, new images of the real world are frequently captured by an AR system to update the virtual geography. To process multiple images into a three-dimensional representation, some AR systems use visual anchors, such as small fragments of one image, and orient other images in three-dimensional space based on their relative coordinates to the visual anchor. However, this method can make it difficult to place virtual objects in the real-world representation. Especially in large-scale AR environments.

Summary of the Invention

[0003] A method for positioning virtual objects to maintain alignment of the virtual objects within a map segment of a three-dimensional (3D) representation of a portion of the real world is described. The method creates a map of a large area of the real world composed of map segments. Each map segment can be a point cloud of image data captured by a mobile device or other 3D or a 3D representation of other physical environments. Drift and other errors in the map segments are corrected by recording the positions of the virtual objects relative to multiple map segments and determining where to display the virtual objects using combinations (optionally weighted) of the positions of the virtual objects in at least some of the map segments.

[0004] In some embodiments, the method includes determining a location of a client device in the real world and obtaining a set of map segments based on the location of the client device. For each of the obtained map segments, a relationship vector is obtained. The relationship vector is weighted based on object parameters associated with the virtual object. Based on the weighted relationship vector, a position to display the virtual object is determined. The virtual object is provided for display at the determined position.

[0005] These and other features, aspects, and advantages can be better understood with reference to the following description and the appended claims. The following description describes various embodiments in which the AR application is a parallel reality game, but it should be understood that the same or similar techniques are applicable to many ARs. The accompanying drawings illustrate specific embodiments and, together with the description, serve to explain various principles. However, the drawings should not be considered limiting. Rather, the scope of protection should be determined from the claims.

Brief Description of the Drawings

[0006]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6

[0007] Here, various embodiments are referred to in detail, and one or more examples are shown in the drawings. Each example is provided as an illustration of the described embodiments, not as a limitation of the claims. Indeed, those skilled in the art will understand that various modifications and variations can be made without departing from the principles described. For example, features illustrated or described as part of one embodiment can be used in conjunction with another embodiment to yield further embodiments. Accordingly, the present disclosure is intended to embrace such modifications and variations that come within the scope of the appended claims and their equivalents.

[0008] (Overview) Generally, the present disclosure relates to parallel reality games that occur in a virtual world mapped to real-world locations. The virtual world has experiences related to real-world actions, such experiences incorporating virtual objects, virtual items, virtual energy, virtual characters, and other virtual elements that can be used or collected by players of a parallel reality game having a virtual world parallel to at least a portion of the real world. In particular, experiences in the virtual world are determined based on data associated with one or more real-world actions. In this way, virtual experiences can correspond to actions in the real world that make the gameplay more immersive. Further, the actions and conditions for finding virtual experiences in the virtual world based on data located in the real world improve the link between the parallel virtual world and the real world, further enhancing the illusion that it is another dimension of the real world through which players can perceive and interact with the virtual world through the parallel reality game.

[0009] The game server may host a location-based parallel reality game having a player game area that includes a virtual environment having a geography that parallels at least a portion of the geography of the real world. A player may navigate a virtual space in the virtual world by navigating the corresponding geographical space in the real world. In particular, a player may navigate a range of coordinates that define a virtual space in the virtual world by navigating a range of geographical coordinates in the real world.

[0010] In one aspect, a player's position may be monitored or tracked using a positioning system (e.g., a GPS system) associated with the player's mobile computing device (e.g., a cellular phone, smartphone, game device, AR headset, or other device). The position and orientation of the player's mobile computing device (collectively referred to as the "pose" of the device) may be determined by a positioning process that is more accurate than a positioning system determined from sensor data (e.g., an image captured by the device's camera). As the player moves in the real world, the player's position information may be provided to a game server that hosts the parallel reality game via a network. The game server may update the position of the player in the parallel virtual world to correspond to the player's position in the real world.

[0011] The parallel reality game may include one or more virtual elements with which players can interact during the course of the parallel reality game. To interact with a virtual element, a player may move to a location corresponding to the virtual element in the real world and may need to perform any necessary interactions in the parallel reality game. According to aspects of the present disclosure, a virtual experience can be generated in the virtual world based on data associated with real-world actions. The experience in the virtual world may be determined by analyzing data associated with real-world actions. For example, a real-world action may result in an experience in the virtual world that is determined by the real-world action.

[0012] Linking virtual experiences to real-world actions enables a more engaging experience for players. In this way, the subject matter of the present disclosure can have a technical effect of providing an improved computer-based implementation of a parallel reality game that provides for the generation of virtual experiences in a parallel reality game in a manner that improves the link between the real world and the parallel virtual world.

[0013] In one embodiment, a game server associated with a parallel reality game can access data associated with the location of an individual in the real world. The data associated with the location of an individual in the real world can be obtained or derived from any suitable source. The data associated with the location of an individual in the real world can include the location of a mobile device user in the real world. In particular, a user of a mobile device, such as a smartphone, can optionally provide location information regarding the geographical location of the real world in order to enhance certain location-based features and other features. The information optionally provided by the user of the mobile device can be provided under anonymous conditions in order to protect the privacy of the user who optionally provides the location information.

[0014] The data associated with the location of an individual in the real world can also include data associated with the location of a player of the parallel reality game. In particular, the game server can receive location information from each of a plurality of players during play of the parallel reality game, and as a result, the game server can update the location of the players in the parallel virtual world associated with the parallel reality game.

[0015] The game server can analyze data associated with the location of an individual in the real world and generate a virtual experience based on such data. For example, the game server can place virtual elements within the virtual world of a user (or another different user) that are collected when the user (or another different user) moves to a specific location in the real world. In certain aspects, the virtual elements can be used to enhance the experience in the real world. For example, the virtual elements can be exchanged or presented for one or more goods or services in the real world. Generating a virtual experience in the virtual world based on real-world actions can give the player a reason to move to a specific location in the real world.

[0016] In certain implementations, specific real-world actions can be mapped directly and / or indirectly to experiences in the virtual world. For example, weather data from the real world can be mapped directly to virtual weather in the virtual world. Similarly, real-world weather data can be mapped indirectly to the virtual world such that, for example, it may be more difficult to find certain virtual elements when the real-world weather conditions indicate rain. As described herein, such mapping can include any real-world action and can be mapped directly or indirectly to one or more experiences in the virtual world regardless of whether such experiences are related to real-world actions. As another example, a solar eclipse in the real world can be mapped indirectly to the virtual world, resulting in a virtual experience where the virtual energy of all players in the virtual world increases. Alternatively, or in combination with the previous example, the solar eclipse can be mapped directly to the virtual world, causing a virtual solar eclipse to be visible in the virtual world. In this way, the game server can generate virtual experiences in the virtual world from real-world actions.

[0017] The game server can generate virtual experiences in a parallel virtual world based on other data associated with real-world actions. For example, the game server can generate virtual experiences based on items of cultural, entertainment, or commercial value, map data, hazard data, weather data, event calendar data, and other related real-world actions. As an example, the game server can include virtual experiences in the virtual world based on actions related to real-world items corresponding to locations of public, educational, commercial, or entertainment value, such as public works, tourist attractions, landscapes, libraries, hiking courses, etc.

[0018] (Exemplary Location-Based Parallel Reality Game System) An exemplary computer-implemented location-based game system according to an exemplary embodiment of the present disclosure will now be described. The subject matter of the present invention is described with reference to parallel reality games. A parallel reality game is a location-based game having a virtual world geography that is parallel to at least a portion of the real-world geography such that the movement and actions of a player in the real world affect the actions in the virtual world and vice versa. Those skilled in the art who utilize the disclosure provided herein should understand that the subject matter of the present disclosure can be applied to other game systems. Additionally, the computer-based system-specific flexibility allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among the components of the system. For example, the systems and methods for modifying or validating game data according to the aspects of this specification can be implemented using a single computing device or across multiple computing devices.

[0019] FIG. 1 shows an exemplary computer-implemented location-based game system 100 configured according to one embodiment. The location-based game system 100 provides for the interaction of multiple players in a virtual world having a geography parallel to the real world. In particular, the geographical areas of the real world can be directly linked or mapped to corresponding areas of the virtual world. Players can move through the virtual world by moving to various geographical locations in the real world. For example, the system 100 can track the position of a player in the real world and update the position of the player in the virtual world based on the player's current position in the real world.

[0020] FIG. 2 shows a conceptual diagram of a virtual world 210 parallel to the real world 200 that can function as a game board for players of a location-based game according to an exemplary embodiment of the present disclosure. As illustrated, the virtual world 210 can include a geography parallel to the geography of the real world 200. In particular, the range of coordinates that define a geographical area or space in the real world 200 is mapped to a corresponding range of coordinates that define a virtual space in the virtual world 210. The range of coordinates in the real world 200 can be associated with a town, neighborhood, city, campus, locale, country, continent, the entire earth, or other geographical areas. Each geographical coordinate within the range of geographical coordinates of the real world 200 is mapped to a corresponding coordinate within the virtual space of the virtual world 210.

[0021] The position of the player in the virtual world 210 corresponds to the position of the player in the real world 200. For example, player A located at position 212 in the real world 200 has a corresponding position 222 in the virtual world 210. Similarly, player B located at position 214 within the real world has a corresponding position 224 within the virtual world. When a player moves within the geographical coordinate range in the real world, the player also moves within the range of coordinates that define the virtual space within the virtual world 210. In particular, a positioning system (e.g., a GPS system) associated with the mobile device carried by the player can be used to track the player's position when the player moves within the geographical coordinate range in the real world 200. Data associated with the position of the player in the real world 200 is used to update the position of the player within the corresponding range that defines the virtual space within the virtual world 210. Thus, the player does not necessarily need to check in or periodically update position information at a specific individual location within the real world 200, and can move in a continuous track within the range of coordinates that define the virtual space within the virtual world 210 just by moving within the corresponding geographical coordinate range in the real world 200.

[0022] A location-based game may include a plurality of game objectives that require the player to move to and / or interact with various virtual elements and / or virtual objects scattered at various virtual positions within the virtual world. The player can move to these virtual positions by moving to the corresponding positions of the virtual elements or virtual objects in the real world. For example, the positioning system can continuously track the player's position such that when the player continuously moves in the real world, the player also continuously moves in the parallel virtual world. The player can then interact with various virtual elements and objects at a specific location to achieve or execute one or more game objectives.

[0023] For example, the objectives of the game may include interacting with the player or, alternatively, claiming ownership of virtual elements 230 located at various virtual positions within the virtual world 210. These virtual elements 230 may be linked to landmarks, geographical locations, or objects 240 within the real world 200. The real-world landmarks or objects 240 may be artworks, monuments, buildings, businesses, libraries, museums, or other suitable real-world landmarks or objects. To capture these virtual elements 230, the player moves to the landmark or geographical location 240 in the real world that is linked to the virtual element 230 in the virtual world 210 and interacts with the virtual element 230 within the virtual world 210. For example, player A in FIG. 2 may move to the landmark 240 in the real world 200 to interact with the virtual element 230 linked to that particular landmark 240. The interaction with the virtual element 230 may be associated with actions in the real world such as capturing the landmark or object 240 associated with the virtual element 230 and / or verifying, obtaining, or capturing other information. In another example, the player may send a virtual character to a landmark 240 at a specific remote virtual location so that the virtual character can interact with the virtual element 230, object 240, or another player 214 where the virtual character is located.

[0024] The objective of the game may include the player using one or more virtual items, which are collected by the player within a location-based game. For example, the player may move through the virtual world searching for virtual items (such as weapons, food, medical supplies, soldiers, creatures, or other items) that may be useful in achieving the game objectives. These virtual items may be found or collected by moving to different locations in the real world or by completing various actions either in the virtual world or the real world. For example, the player may interact with virtual element 230 within the virtual world to obtain virtual elements. In the embodiment shown in FIG. 2, the player may use virtual item 232 to capture one or more virtual elements 230. In particular, the player may deploy virtual item 232 at a location within virtual world 210 that is proximate to or within virtual element 230. Deploying one or more virtual items 232 proximate to virtual element 230 may result in the capture of virtual element 230 for a particular player or the player's team and / or faction.

[0025] In one particular implementation, the player may collect partial virtual energy of a location-based game. As shown in FIG. 2, virtual energy 250 may be scattered at different locations within virtual world 210. The player may collect virtual energy 250 by moving to the corresponding location of virtual energy 250 within real world 200. Virtual energy 250 may be used to power virtual items or to perform various game objectives within the game. A player who loses all of the virtual energy 250 of the game may be temporarily disconnected from the game.

[0026] According to aspects of the present disclosure, a location-based game may be a large-scale multiplayer location-based game in which all participants in the game share the same virtual world. Virtual players may be divided into separate teams or factions and may cooperate to achieve one or more game objectives, such as capturing or claiming ownership of virtual elements. For convenience, all such groups of players are referred to herein as teams. In this way, a location-based game can essentially be a social game that promotes cooperation among players within the game. Players on opposing teams may battle each other during the location-based game. Players may use virtual items to attack or interfere with the progress of players on the opposing team. In some examples, players on different teams may cooperate in a specific shared virtual experience (e.g., a boss battle) to achieve a common goal.

[0027] A location-based game may have various features for enhancing and rewarding gameplay within the location-based game. For example, players may accumulate virtual currency or other virtual rewards that can be used throughout the game. Players may advance through various levels as they achieve one or more game objectives and gain experience within the game. In some embodiments, players may communicate with each other via one or more communication interfaces provided within the game. Players may also acquire enhanced "powers" or virtual items that can be used to complete game objectives within the game. Using the disclosure provided herein, one of ordinary skill in the art should understand that various other game features may be included in the parallel reality game without departing from the scope of the present disclosure.

[0028] Referring to FIG. 1, the illustrated computer-implemented location-based game system 100 includes a client-server architecture, where a game server 110 communicates with one or more clients 120 via a network 130. Two clients 120 are shown in FIG. 1, but any number of clients 120 can be connected to the game server 110 via the network 130. The server 110 may host a universal game module 112 that controls aspects of the location-based game for the player and receives and processes inputs from players within the location-based game. On the client side, each client 120 may include a game module 125 that operates as a game application to provide the user with an interface to the system 100. The game server 110 transmits game data to the clients 120 via the network 130, and the game module 125 of the client 120 uses this to provide the player, who is located away from the game server 110, with a local version of the game (e.g., a portion of the virtual world specific to the player's location).

[0029] It will be understood that the term "module" refers to computer logic utilized to provide a desired functionality. Thus, a module can be implemented in hardware, firmware, and / or software that controls a general-purpose processor. In one embodiment, a module is a program code file stored in a storage device, loaded into memory, and executed by a processor, or can be provided from a computer program product such as computer-executable instructions stored on a tangible computer-readable storage medium such as a RAM hard disk, or an optical or magnetic medium.

[0030] The game server 110 can be any computing device and may include a processor and a memory. The memory may store instructions that cause the processor to execute operations. The game server 110 may include or communicate with a game database 115. The game database 115 stores game data that is provided to the client(s) 120 via the network 130 or used in a location-based game that is provided.

[0031] The game data stored in the game database 115 includes: (1) data associated with the virtual world within the location-based game (e.g., image data used to represent the virtual world on a display device, geographical coordinates of locations within the virtual world, etc.); (2) data associated with the players of the location-based game (e.g., player information, player's experience level, player currency, player inventory, current player position in the virtual / real world, player energy level, player preferences, team information, etc.); (3) data related to game objectives (e.g., data related to the current game objective, the purpose of the game, the purposes of past games, the purposes of future games, desired game purposes, etc.); (4) data associated with virtual elements in the virtual world (e.g., the position of the virtual element, the type of the virtual element, the game objective associated with the virtual element, the corresponding real-world location information of the virtual element, the actions of the virtual element, the relevance of the virtual element, etc.); (5) data objects / landmarks associated with the positions (e.g., real-world positions) linked to real-world objects, landmarks, and elements of the virtual world (e.g., descriptions of real-world objects / landmarks, the relevance of virtual elements linked to real-world objects, etc.); (6) game status (e.g., the current number of players, the current status of the game objective, player leaderboards, etc.); (7) data actions / inputs associated with the player (e.g., the current player position, past player positions, player movements, player inputs, player queries, player communications, etc.); and (8) may include other data used, related to, or obtained during the implementation of the location information game. The game data stored in the game database 115 is set offline or in real-time by the system administrator and / or is set by data received from users / players of the system 100, such as one or more clients 120 via the network 130.

[0032] As will be described in further detail below, the game server 110 may include a real-world state database 117 and may communicate with the real-world state database 117. The real-world state database 117 may be part of the game database 115, may be integral with the game database 115, or may be separate from the game database 115. The real-world state database 117 stores data associated with the state of the real world, such as the individual and / or aggregated positions of players in the real world, actions associated with positions of cultural or commercial value, map data providing the positions of roads, highways, and waterways, the current and past positions of individual players, hazard data, weather data, event calendar data, and other appropriate data. The data stored in the real-world state database 117 may be collected or obtained from any suitable source. For example, in one aspect, the real-world condition database 117 may be coupled to, included in, or be part of a map database that stores map information, such as one or more map databases accessed by a mapping service. According to another exemplary aspect, the real-world state database 117 may obtain or access data associated with the past and current positions of players, for example, from the game database 115. According to yet another exemplary aspect, the real-world state database 117 may be coupled to one or more external data sources or services that periodically provide population data, hazard data, weather data, event calendar data, or other data to the real-world state database 117.

[0033] The game server 110 may be configured to receive requests for game data from one or more clients 120 (e.g., via remote procedure calls (RPCs)) and respond to those requests via their network 130. For example, the game server 110 may encode game data in one or more data files and provide the data files to the client 120. Additionally, the game server 110 may be configured to receive game data (e.g., player position, player action, player input, etc.) from one or more clients 120 via the network 130. For example, the client device 120 may be configured to periodically send player input and other updates to the game server 110, and the game server 110 may update the game data in the game database 115 and use it to reflect any and all changed conditions of the game.

[0034] As shown, the game server 110 may include a universal game module 112. The universal game module 112 hosts location-based games for all players and functions as the authoritative source of the current status of the location-based games for all players. The universal game module 112 incorporates game data (e.g., player input, player position, player action, player status, landmark information, etc.) from the client 120 and the game data received for the overall location-based game for all players of the location-based game. The universal game module 112 may also manage the distribution of game data to the client 120 via the network 130.

[0035] In the embodiment shown in FIG. 1, the game server 110 also includes a locator module 114. The locator module 114 can be part of the universal game or separate from the universal game module 112. The locator module 114 is configured to access data associated with real-world actions, analyze the data, and determine a virtual experience in the virtual world based on the data associated with real-world actions. For example, the locator module 114 can modify the game data stored in the game database 115 to locate a virtual experience in the virtual world based on the data associated with real-world actions.

[0036] The mapping module 116 creates and updates a map of the real-world geography. For example, the mapping module 116 receives real-world images from one or more client devices 120 and combines those images into a three-dimensional representation of the real world. The game server 110 can use this three-dimensional representation to update the geography of the virtual world and more closely mirror the real world. In addition to the real-world images, the mapping module 116 further receives location data from the positioning device 128 of the client device 120 to assist the mapping module 116 in placing the image data in the correct location within the three-dimensional representation.

[0037] The mapping module 116 creates a three-dimensional representation of the real world by creating map segments, each of which describes a part of the real world and is tagged with a real-world location (e.g., GPS coordinates). In one embodiment, each map segment is a point cloud or mesh having three-dimensional geometry. The map segments are generated, for example, by a user of the client device 120 that captures video of the local area around the client. Image data (e.g., video) communicates with the mapping module 116 of the game server 110 that generates map segments from the image data. The mapping module 116 can determine the relative positions (e.g., movement vectors between map segment coordinate spaces) between each map segment and nearby map segments (such as adjacent map segments), and create a three-dimensional representation of the real world composed of interrelated map segments.

[0038] Other modules can be used with the game server 110. Any number of modules can be programmed or configured to perform the described server-side functions. As further described herein, the various server-side components can be rearranged. For example, the game database 115 can be integrated into the game server 120. Other configurations are apparent in light of the present disclosure, and the present disclosure is not intended to be limited to any particular configuration.

[0039] The client 120 can be any computing device that a player can use to interact with the game system 100. For example, the client 120 can be a wireless device, a personal digital assistant (PDA), a portable game device, a cellular phone, a smartphone, a tablet, a navigation system, a handheld GPS system, or other such device. In short, the client 120 can be any computer device or system that executes the game module 125 and allows the player to interact with the virtual world.

[0040] The game module 125 executed by the client 120 provides an interface between the player and the location-based game. The game module 125 can present a user interface on a display device associated with the client 120 that displays a virtual world associated with the game and enables the user to interact in the virtual world to perform various game objectives. The game module 125 can also control various other outputs to enable the player to interact with the game without the player having to view the display screen. For example, the game module 125 can provide various audio, vibration, or other notifications that enable the player to play the game without looking at the display screen. The game module 125 can access game data received from the game server 110 to provide an accurate representation of the game to the player. The game module 125 receives and processes player input and updates the game server 110 via the network 130.

[0041] Since the game system 100 is for location-based games, the client 120 is preferably a portable computing device such as a smartphone or other portable device that can be easily carried with the player or can be carried otherwise. The player can interact with the virtual world in the real world simply by carrying or transporting the client 120. The client 120 may include a positioning device 128 that monitors the player's position during gameplay. The positioning device 128 may be any device or circuit for monitoring the position of the client 120. For example, the positioning device 128 may use a satellite navigation positioning system (e.g., GPS system, Galileo positioning system, Global Navigation Satellite System (GLONASS), Beidou satellite navigation and positioning system), an inertial navigation system, a dead reckoning system, based on an IP address, triangulation and / or proximity to a cellular phone base station or a WiFi hotspot, and / or other suitable techniques for determining the actual or relative position to determine the actual position or relative position.

[0042] As the player moves around in the real world with the client 120, the positioning device 128 tracks the player's position and provides player position information to the game module 125. The game module 125 updates the player's position within the virtual system based on the player's actual position in the real world. In particular, the player's position in the virtual world may correspond to the player's position in the real world. The game module 125 may provide player position information to the game server 110 via the network 130 such that the universal game module 112 tracks the player's position throughout the game.

[0043] Network 130 may be any type of communication network such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof. The network may also include a direct connection between client device 110 and game server 120. Generally, communication between game server 110 and client 120 may be carried over a network that uses any type of wired and / or wireless connection, using various communication protocols (such as TCP / IP, HTTP, S1v1TP, FTP, etc.), encodings or formats (such as HTML, JSON, XML, etc.), and / or protection schemes (such as VPN, secure HTTP, SSL, etc.).

[0044] The technology discussed herein refers to servers, databases, software applications, and other computer-based systems, as well as actions performed between such systems and information transmitted between such systems. One of ordinary skill in the art will appreciate that the inherent flexibility of computer-based systems enables various configurations, combinations, and divisions of tasks and functions between components. For example, the server processes discussed herein may be implemented using a single server, or multiple servers may work together. Databases and applications may be implemented on a single system, or may be distributed across multiple systems. Distributed components may be performed sequentially or in parallel.

[0045] In addition, in situations where the systems and methods described herein access and analyze personal information about a user or use personal information such as location information, the user may be provided with the opportunity to control whether a program or function collects information, whether to receive content from the system or other applications, and / or how to receive content. Such information or data is not collected or used until the user is provided with a meaningful notice as to what information is being collected and how that information will be used. The information is not collected or used without the user's consent, and the user may revoke or change the consent at any time. Thus, the user can control how information about the user is collected and used by the application or system. Further, certain information or data may be processed in one or more ways such that information that can identify an individual is removed before it is stored or used. For example, the user's identity may be processed such that it cannot be used to determine personally identifiable information about the user.

[0046] (Exemplary Mapping Module) FIG. 3 is a block diagram of a mapping module 116 according to one or more embodiments. Embodiment mapping module 116 includes a map segmentation module 310, an object positioning module 320, and a data store 330. These modules 310-330 enable the mapping module 116 to capture image data from the client device 120 and create or update a three-dimensional representation of the real world.

[0047] The map segmentation module 310 receives image data or map data of a three-dimensional representation of the real world and generates discrete map segments. A map segment can be composed of one instance of the received image data. For example, a user of a client device may capture image data (e.g., an image or a video) of an area surrounding the client device, and the video is communicated to the map segmentation module 310 and the image data is converted into map segments. A single map segment may include image data received from one client device within a pre-set period (e.g., a 30-second video received from a client device). In some embodiments, a map segment may combine image data from multiple client devices captured at the same location and time (e.g., within a pre-set time, within a threshold distance). In another embodiment, the map segmentation module 310 sets a fixed size for each map segment and captures image data until the data meets the fixed size. For example, a map segment may represent a cubic volume of the real world with each side having a fixed length (e.g., 3 meters). The map segmentation module 310 determines, based on the position data received from the client device, whether the image data belongs to a specific cubic map segment, and adds the received image data to that segment until all the data necessary to complete that map segment is received. Map segments are arranged to generate a three-dimensional representation of a local area. Each map segment can be rotated or transformed independently of other map segments to create a real-world representation that aligns with the current sensor data being captured by the client device 120.

[0048] When the map segmentation module 310 creates a map segment, the map segment is stored in the data store 330. The map segmentation module 310 can determine the placement of the map segment relative to other map segments, and the data store 330 further stores that relationship. As more image data is received and more map segments are created, previous map segments can be shifted and / or rotated to improve the accuracy of the three-dimensional representation of the real world. When a map segment is shifted, the data store 330 is updated to include the updated relationship between the map segments.

[0049] The map segmentation module 310 further receives location data of client devices associated with the game server 110 to assist the mapping module 116 in placing virtual objects. The map segmentation module 310 uses the location data (e.g., GPS coordinates in the real world) to obtain a set of map segments in the data store 330 corresponding to the location of the client device. For example, if the location data of the client device indicates that the client device is near a landmark, the map segmentation module 310 obtains all map segments associated with the landmark from the data store 330. In some embodiments, the map segment module 310 obtains a set number or set of map segments associated with locations within a threshold distance of the received location data. For example, the map segment module 310 may obtain the eight map segments closest to the map segment containing the location of the client device. In other embodiments, the map segmentation module 310 obtains a set of map segments that include locations within a 100-foot radius of the location data. This set of map segments is used to place virtual objects for display on the client device.

[0050] When additional image data is received by the map segmentation module 310, the object positioning module 320 places virtual objects within the map segments and updates the positions of the virtual objects. The virtual objects may be created by an administrator of the game server 110 together with a set of object parameters. The object parameters specify rules for placing the virtual objects. For example, a virtual object that is paint may include object parameters indicating that the paint is placed on a flat vertical surface such as a wall. A virtual object that is a fence may include object parameters indicating that the fence is straight and continuous.

[0051] For each virtual object, the object positioning module 320 generates a relationship vector between the virtual object and a set of map segments. The set of map segments can be the map segments closest to the virtual object. In some embodiments, the set of map segments is selected based on the object parameters. The relationship vector indicates the position of the virtual object with respect to each map segment in the set of map segments. The relationship vector can be from any point within the map segment, such as the center or an edge, to the virtual object. An exemplary relationship vector is shown in FIG. 4. In some embodiments, the object positioning module 320 may further detect real-world surfaces or objects within the set of map segments and generate a relationship vector between the virtual object and the real-world surface or object.

[0052] In some embodiments, the object positioning module 320 may have a lower and upper threshold on the number of relationship vectors beforehand. A larger number of relationship vectors results in higher accuracy of map segment alignment but requires a large amount of computing power for the solution. The number of relationship vectors used is tuned for accuracy while maintaining low computational power and thus low latency.

[0053] The object positioning module 320 further determines a relative weighting of the relationship vectors based on the object parameters of the virtual object. The weighting of the relationship vectors indicates how tightly the positional relationship between the virtual object and the real-world object or map segment is maintained as the map segment shifts and updates. For example, if the virtual object is a flower and has relationship vectors to the ground beneath it, the map segment it is in, and the map segments adjacent to the map segment it is in, the relationship vector between the flower and the ground beneath it can be highly weighted. A high weighting ensures that the flower remains on the ground as the map segment shifts or updates. Since the object parameters of the virtual flower indicate that the position of the flower is not associated with anything within the adjacent map segments, the weighting of the relationship vectors to the adjacent map segments can be low. In an example where the virtual flower is part of a line of other virtual flowers, such as running along a real-world sidewalk, each virtual flower can have a highly weighted relationship vector to other map segments that include the line of virtual flowers. Increasing the weighting to other map segments can keep the line of flowers together, either evenly spaced or in a straight line. The line of flowers can also have a highly weighted relationship vector to the real-world sidewalk to keep the flowers along the sidewalk.

[0054] When a player of a game associated with game server 110, for example, is at a specific location, or activates a game function or triggers the placement of virtual objects, object positioning module 320 determines the position on the screen of the client device associated with the player to display the virtual object. Object positioning module 320 determines the placement of the virtual object that satisfies the weighted relationship vector such that the highly weighted vectors remain substantially the same. Object positioning module 320 may determine the position for displaying the virtual object on the client device by solving a system of equations including the weighted relationship vector. Object positioning module 320 determines the position to place the virtual object at specific coordinates of the map segment, and then may determine the corresponding placement of the virtual object on the client device screen.

[0055] Data store 330 stores the data used by modules 310, 320. Data store 330 may include previous object placements so that they can be reused on virtual objects having similar object parameters. Data store 330 also stores a history of the positions of the map segments, and may return the position of the map segment to its previous position, or track the position of the map segment over time as needed when the position of the map segment is changed based on new information received by mapping module 116. Data store 330 further stores object parameters. In some embodiments, data store 330 further stores position data from client device 120 that provided image data to the map segment.

[0056] Figures 4A and 4B show two examples of positioning virtual objects using relationship vectors according to one or more embodiments. In both Figures 4A and 4B, map segments 405 and adjacent map segment 410 each have virtual objects 425, 435. The left - hand map segments 405, 410 represent a three - dimensional representation of the real world at a point in time before updated image data is received by game server 110. On the right, the map segments 405, 410 are shifted or drifted for new image data added to the three - dimensional representation of the real world. Before and after the shift of the map segments, the positions of virtual objects 425 and 435 can change. In the examples shown in Figures 4A and 4B, the new image data indicates that surfaces 415, 430 are straight walls, and thus the map segments are shifted to correct the angle of the walls such that surfaces 415, 430 are linear.

[0057] Figure 4A is an example of positioning a virtual object using a relationship vector 420 that defines only the relationship between virtual object 425 and segment 405 where the virtual object was first placed. Relationship vector 420 indicates the position of virtual object 425 relative to the center of segment 405. In the example of Figure 4A, since there is no relationship vector from virtual object 425 to map segment 410, when segment 405 is shifted, virtual object 425 is moved away from virtual object 435. The lack of relationship vectors between each virtual object and each map segment prevents the virtual objects from remaining aligned or evenly spaced when map segment 405 is shifted. Instead, virtual object 425 shifts with map segment 405 and does not remain aligned with object 435.

[0058] In FIG. 4B, virtual objects 425 have relationship vectors with respect to both the map segment 405 in which they are included and the adjacent map segment 410. Similarly, virtual objects 435 have relationship vectors to the map segment 410 in which they are present, as well as to the adjacent map segment 405. This is to maintain the alignment of virtual objects 425, 435 that have a strong positional relationship with each other, such as being different parts of a larger structure (e.g., pillars forming a fence). The relationship vectors can be weighted approximately evenly (e.g., within 10% of each other) so that virtual objects 425, 435 maintain their relative positions and alignment. As a result, when map segment 405 is shifted and surfaces 415, 420 are aligned, the virtual objects are kept equidistant on the same straight line.

[0059] Conversely, virtual objects having a physically strong positional relationship (e.g., a virtual hat on a physical image or a virtual poster on a physical wall) can set their relationship vector for the map segment to which it is closest to approximately 1 (e.g., exceeding 0.9, exactly or close to 1), and set the weights of the other relationship vectors to approximately 0 (e.g., less than 0.1, exactly or close to zero). Thus, the position of the virtual object is closely coupled to the map segment containing the physical object to which it is related, enabling the virtual object to be accurately positioned (e.g., the virtual hat remains on the head of the physical image rather than floating in the air on one side).

[0060] In a further example, a hybrid configuration where objects have a general positional relationship to physical objects but are somewhat related to each other can weight map segments based on proximity to the virtual object (e.g., the map segment closest to the virtual object can be weighted 0.4, the next closest 0.3, etc.). Thus, the position of the virtual object can be "stretched" by changes in nearby map segments but remains mainly fixed to the physical position within the closest map segment.

[0061] In some embodiments, to ensure additional accuracy of the alignment between virtual objects 425, 435, relationship vectors between each pair of virtual objects can be generated. Those relationship vectors can be weighted higher than the relationship vectors between the objects and the map segments such that the vectors between the objects do not change in length or direction when a shift of the map segment occurs. For example, the relative position of the map segment can be periodically recalculated taking into account sensor drift, newly available map data, and other inaccuracies, and the position of the virtual objects is updated using a weighted combination of the relationship vectors to maintain the desired spatial relationship of the virtual objects to each other and the physical environment.

[0062] FIG. 5 is a flowchart showing a method 500 for positioning virtual objects according to one or more embodiments. FIG. 5 is not an exhaustive representation of all steps that may occur in the process of map segment and object alignment. Some of the steps may be performed in a different order, in parallel, or not at all. Further, the method may include using other modules and steps before, during, or after the process flow described above. For example, method 500 may further include identifying virtual objects within the map segment and weighting each relationship vector based on the identity of the virtual object. In some embodiments, the steps of method 500 are performed by the mapping module 116 of game server 110.

[0063] The game server 110 determines 510 the real-world location of a client device (e.g., client device 120). The location of the client device can be determined by the game server 110 querying the positioning device 128 of the client device 120 via the network 130. In some embodiments, the location of the client device is represented as GPS coordinates. In another embodiment, the location of the client device can be represented as a vector related to the distance and direction between the client device and a known point such as a landmark.

[0064] Based on the location of the client device, the game server 110 obtains 520 a set of map segments. The set of map segments can include map segments within a threshold distance of the location of the client device. In some embodiments, the game server 110 can set the number of map segments to be obtained. For example, upon receiving location data from the client device, the game server 110 can obtain the six map segments closest to the location of the client device. Each map segment is a point cloud or mesh that is a three-dimensional representation of the real world around the location of the client device.

[0065] The game server 110 determines 530 virtual objects to be displayed on the client device. The virtual objects can be determined, for example, by a game event triggered by the user of the client device. In some embodiments, the location of the client device can be associated with the virtual objects such that the virtual objects are determined to be displayed when the location data of the client device indicates that the user is near a specific location.

[0066] The game server 110 obtains 540 a relationship vector between each map segment of a set of virtual objects and map segments. For example, when six map segments are obtained by the game server 110, the same server 110 obtains a vector from each map segment to the virtual object. The relationship vector indicates the distance and directional relationship between the obtained virtual object and each map segment.

[0067] The game server 110 weights 550 each relationship vector based on the object parameters of the virtual object. The object parameters of the virtual object indicate factors for placing the virtual object. For example, the object parameters may indicate that the virtual object is near another object (virtual or real-world) within the same map segment. Thus, a high weight is assigned to the relationship vector between the virtual object and the map segment in which it exists, ensuring that the virtual object remains within the current map segment.

[0068] The game server 110 determines 560 the position to display the virtual object on the display of the client device based on the weighted relationship vectors. To place the object, the game server 110 may keep the relationship vectors with high weights the same in length and direction and allow the other relationship vectors with lower weights to vary in length and direction. In some embodiments, the relationship vectors may be ranked based on their weights to determine the position to display the virtual object.

[0069] The game server 110 provides 570 the virtual object to be displayed on the client device at the determined position. Instructions indicating the position of the virtual object may be transmitted to the client device via the network. The instructions cause the client device to display the virtual object at the determined position on the client device display.

[0070] (Exemplary Computing Device Architecture) FIG. 6 is a block diagram illustrating a configuration of an exemplary machine capable of reading instructions from a machine-readable medium and executing them in a processor (or controller). Specifically, FIG. 6 shows a graphical representation of a machine in an exemplary form of a computer system 600. The computer system 600 may be associated with components (or modules) of the game server 110 and / or the client 120 and may be used to execute instructions 624 (e.g., program code or software), which are one or more of the methodologies (or processes) described herein, including the methodologies (or processes) described, to cause the machine to execute any one or more of them.

[0071] The machine may be any machine that executes instructions 624 (sequential or otherwise) that specify actions to be performed by the machine, such as a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a smartphone, a network router, a switch or bridge, a cell phone tower, or the like. Further, although only a single machine is shown, the term "machine" shall be construed to include any collection of machines that individually or jointly execute instructions 624 to perform any one or more of the methodologies described herein.

[0072] Exemplary computer system 600 includes one or more processing units (generally, one or more processors 602). Processor 602 can be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a controller, a state machine, one or more application specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof. Any reference to processor 602 herein may refer to a single processor or multiple processors. Computer system 600 also includes main memory 604. The computer system may include storage unit 616. Processor 602, memory 604, and storage unit 616 communicate via bus 608.

[0073] Furthermore, computer system 600 can include static memory 606, display driver 610 (e.g., to drive a plasma display panel (PDP), liquid crystal display (LCD), or projector). Computer system 600 also includes alphanumeric input device 612 (e.g., a keyboard), cursor control device 614 (e.g., a mouse, trackball, joystick, motion sensor, or other pointing instrument), signal generating device 618 (e.g., a speaker), and network interface device 620, which are also configured to communicate via bus 608.

[0074] Storage unit 616 includes machine-readable medium 622 on which instructions 624 (e.g., software) are stored that embody any one or more of the methods or functions described herein. Instructions 624 can also be fully or at least partially present in main memory 604 or in processor 602 (e.g., in the processor's cache memory) during execution by computer system 600, and main memory 604 and processor 602 also constitute a machine-readable medium. Instructions 624 can be transmitted or received over network 670 via network interface device 620.

[0075] Although the machine-readable medium 622 is shown as a single medium in the exemplary embodiment, the term "machine-readable medium" should be construed to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) that can store the instructions 624. The term "machine-readable medium" should also be construed to include any medium that is capable of storing the instructions 624 for machine execution and causing the machine to execute any of the one or more methodologies disclosed herein. The term "machine-readable medium" includes, but is not limited to, data repositories in the form of solid-state memory, optical media, and magnetic media.

[0076] (Additional Considerations) Some of the above sections describe embodiments from the perspective of algorithmic processes or operations. The description and representation of these algorithms are commonly used by those skilled in the data processing arts to effectively convey the content of their work to other skilled artisans. These operations are described functionally, computationally, or logically, but are understood to be implemented by a computer program that includes instructions for execution by a processor or equivalent electrical circuit, microcode, etc. Further, and without loss of generality, it may sometimes be convenient to refer to the arrangement of these functional operations as modules.

[0077] References to "one embodiment" or "an embodiment" mean that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The phrase "in one embodiment" appearing in various places in this specification does not necessarily refer to the same embodiment. Similarly, the use of "a" or "an" before an element or component is done merely for convenience. This description should be understood to mean that one or more of the elements or components are present unless it is obvious otherwise.

[0078] When a value is described as "about" or "substantially" (or derivatives thereof), such value should be construed as being accurate to + / - 10% unless a different meaning is apparent from the context. For example, "about 10" should be understood to mean within the range of 9 to 11.

[0079] As used herein, the terms "comprise," "comprising," "include," "including," "have," "having," or other variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, "or" refers to an inclusive or and not an exclusive or. For example, condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or absent), A is false (or absent) and B is true (or present), and both A and B are true (or present).

[0080] The subject matter of the present invention has been described in detail with respect to specific exemplary embodiments and methods thereof, but those skilled in the art will understand, upon obtaining the foregoing understanding, that they can readily generate modifications, variations, and equivalents to such embodiments. Accordingly, the scope of the present disclosure is by way of example and not limitation, and the disclosure of the subject matter does not exclude the inclusion of such modifications, variations, or additions to the subject matter that will be readily apparent to those skilled in the art.

Claims

1. A computer-implemented method for displaying virtual objects, Obtaining the relationship vector between the virtual object and each map segment of the set of map segments, Weighting each relation vector based on the object parameters of the virtual object, The position for displaying the virtual object is determined based on the weighted relation vector, A computer-implemented method comprising providing the virtual object for display at the determined position on a client device.

2. The computer-implemented method according to claim 1, further comprising obtaining a relational vector between the displayed virtual object and a second virtual object, wherein the second virtual object is located in a set of map segments that are within a threshold distance from the location of the client device.

3. The computer-implemented method according to claim 1, wherein, in response to the object parameter indicating a strong positional relationship between the virtual object and the physical object, the weight of the relation vector of the map segment containing the representation of the physical object is approximately 1, and the weight of the remaining relation vectors is approximately 0.

4. The computer-implemented method according to claim 1, wherein the relation vector is weighted substantially equally in response to the object parameter indicating a strong positional relationship between the virtual object and other virtual objects.

5. The computer-implemented method according to claim 1, wherein providing the virtual object for display includes sending a command to the client device, the command including a position on the display of the client device corresponding to the determined position of the virtual object.

6. The computer-implemented method according to claim 1, wherein each map segment includes a point cloud or a mesh.

7. Updating the relative position of the aforementioned map segment, A computer-implemented method according to claim 1, further comprising redetermining the position for displaying the virtual object using the updated relative position of the map segment and the weighted vector.

8. The computer-implemented method according to claim 1, wherein the virtual object is provided for display in response to a trigger event.

9. The computer-implemented method according to claim 8, wherein the trigger event is the client device being in a specific location or activating a game function.

10. The computer-implemented method according to claim 1, wherein determining the position comprises solving a system of equations including weighted relation vectors.

11. A non-temporary computer-readable storage medium that stores instructions for determining the display position of a virtual object, When executed by a computer system, the computer system will Obtaining the relationship vector between the virtual object and each map segment of the set of map segments, Weighting each relation vector based on the object parameters of the virtual object, The position for displaying the virtual object is determined based on the weighted relation vector, A non-temporary computer-readable storage medium that causes operations to be performed, including providing the virtual object for display at the determined position on a client device.

12. The non-temporary computer-readable storage medium according to claim 11, wherein the operation further comprises obtaining a relational vector between the displayed virtual object and a second virtual object, the second virtual object being in a set of map segments that are within a threshold distance from the location of the client device.

13. The non-temporary computer-readable storage medium according to claim 11, wherein, in response to the object parameter indicating a strong positional relationship between the virtual object and the physical object, the weight of the relation vector of the map segment containing the representation of the physical object is approximately 1, and the weight of the remaining relation vectors is approximately 0.

14. The non-temporary computer-readable storage medium according to claim 11, wherein the relation vector is weighted substantially equally in response to the object parameter indicating a strong positional relationship between the virtual object and other virtual objects.

15. The non-temporary computer-readable storage medium according to claim 11, wherein providing the virtual object for display includes transmitting a command to the client device, the command including a position on the display of the client device corresponding to the determined position of the virtual object.

16. The non-temporary computer-readable storage medium according to claim 11, wherein each map segment includes a point cloud or a mesh.

17. The non-temporary computer-readable storage medium according to claim 11, further comprising updating the positional relationships of the map segments and redetermining the position for displaying the virtual object using the updated positional relationships of the map segments and the weighted vectors.

18. The non-temporary computer-readable storage medium according to claim 11, wherein the virtual object is provided for display in response to a trigger event.

19. The non-temporary computer-readable storage medium according to claim 18, wherein the trigger event is the client device being in a specific location or activating a game function.

20. The non-temporary computer-readable storage medium according to claim 11, wherein determining the position involves solving a system of equations including weighted relation vectors.

Images