Maintaining object alignment in 3D map segments

Abstract

A method of determining a location of a virtual object is described. A location of the client device is determined, and a set of map segments is retrieved based on the determined location. A virtual object to be displayed on a client device is determined. A relationship vector between the virtual object and each map segment in the retrieved set of map segments is obtained. Each relationship vector is weighted based on an object parameter of the virtual object. A location of the displayed virtual object is determined based on the weighted relationship vector. The virtual object is provided as the determined location for display on the client device.

Description

Technical Field

[0001] The present disclosure relates generally to augmented reality (AR), and more particularly to maintaining spatial relationships between different three-dimensional (3D) map data sets in AR applications. Background Art

[0002] Parallel reality applications have a virtual geography that reflects at least a portion of the real-world geography. In order to create and maintain a virtual geography that is similar to the real-world geography, the AR system frequently acquires new images of the real world to update the virtual geography. In order to process multiple images into a three-dimensional representation of the world, some AR systems use visual anchors, such as a small piece of an image, and position other images in three-dimensional space based on the relative coordinates of the other images to the visual anchors. However, this approach can make it difficult to place virtual objects in the representation of the real world. Especially for larger AR environments. Summary of the invention

[0003] A method for positioning a virtual object to maintain its alignment in a map segment of a three-dimensional (3D) representation of a portion of the real world is described. The method creates a map of a large area of ​​the real world composed of map segments. Each map segment may be a point cloud of image data captured by a mobile device or other 3D representation of a physical environment. By recording the positioning of a virtual object relative to multiple map segments and using a combination (possibly weighted) of the positioning of the virtual object in at least some of the map segments to determine where to display the virtual object, drift and other errors of the map segments may be corrected.

[0004] In some embodiments, the method includes determining a location of a client device in the real world and retrieving a set of map segments based on the location of the client device. For each of the retrieved map segments, a relationship vector is obtained. The relationship vector is weighted based on an object parameter associated with the virtual object. A location at which the virtual object is to be displayed is determined based on the weighted relationship vector. The virtual object is provided for display in the determined location.

[0005] These and other features, aspects and advantages may be better understood with reference to the following description and the appended claims. The following description describes various embodiments in which the AR application is a parallel reality game, but it should be understood that the same or similar techniques are applicable to many AR applications. The accompanying drawings illustrate specific embodiments and, together with the description, are used to explain the various principles. However, the drawings should not be considered limiting. Instead, the scope of protection should be determined according to the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 Depicted is a computer-based system for implementing parallel reality gaming in accordance with one or more embodiments.

[0007] Figure 2 Depicted are representations of a virtual world having a geography parallel to the real world in accordance with one or more embodiments.

[0008] Figure 3 is a block diagram of a mapping module according to one or more embodiments.

[0009] Figure 4A and Figure 4B Two examples of methods of map segment alignment according to one or more embodiments are shown.

[0010] Figure 5 is a flow chart illustrating a method for aligning map segments in accordance with one or more embodiments.

[0011] Figure 6 is a diagram showing a method suitable for use in Figure 1 A block diagram of an exemplary computer system for use in a computer-based system. DETAILED DESCRIPTION

[0012] Reference will now be made in detail to various embodiments, one or more examples of which are shown in the accompanying drawings. Each example is provided by way of explanation of the described embodiments rather than by way of limitation of the claims. In fact, it will be apparent to those skilled in the art that various modifications and variations may be made without departing from the principles described. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover such modifications and variations that fall within the scope of the appended claims and their equivalents.

[0013] Overview

[0014] In general, the present disclosure relates to parallel reality games that take place in a virtual world that is mapped to real-world locations. The virtual world has experiences related to real-world actions, and such experiences contain virtual elements, such as virtual objects, virtual items, virtual energy, virtual characters, and other virtual elements, which can be used or collected by players of a parallel reality game that has a virtual world parallel to at least a portion of the real world. Specifically, the experience in the virtual world is determined based on data associated with one or more real-world actions. In this way, the virtual experience can correspond to the action in the real world, which makes the gameplay more immersive. In addition, positioning the virtual experience in the virtual world based on data associated with real-world actions and conditions can improve the link between the parallel virtual world and the real world, further enhancing the illusion that the virtual world is another dimension that players in the real world can perceive and interact with through the parallel reality game.

[0015] The game server may host a location-based parallel reality game having a player play area including a virtual environment having a geographic location parallel to at least a portion of real-world geography. The player may navigate a virtual space in the virtual world by navigating a corresponding geographic space in the real world. Specifically, the player may navigate a series of coordinates defining a virtual space in the virtual world by navigating a series of geographic coordinates in the real world.

[0016] In one aspect, a positioning system (e.g., a GPS system) associated with a player's mobile computing device (e.g., a cellular phone, smart phone, gaming device, AR headset, or other device) can be used, for example, to monitor or track the player's location. A positioning process that is more accurate than a positioning system can be used to determine the location and orientation (collectively referred to as the device's "pose") of the player's mobile computing device from sensor data (e.g., images captured by the device's camera). As the player moves around in the real world, the player's location information can be provided to the game server hosting the parallel reality game over the network. The game server can update the player's position in the parallel virtual world to correspond to the player's position in the real world.

[0017] The parallel reality game may include one or more virtual elements that a player may interact with during the course of the parallel reality game. In order to interact with the virtual elements, the player may have to travel to the corresponding location of the virtual element in the real world and perform any necessary interactions in the parallel reality game. According to aspects of the present disclosure, a virtual experience may be generated in the virtual world based on data associated with real-world actions. The data associated with the real-world actions may be analyzed to determine the experience in the virtual world. For example, an action in the real world may result in an experience in the virtual world generated by the real-world action.

[0018] Associating virtual experiences with real-world actions can provide a more engaging experience for players. In this manner, the disclosed subject matter can have the technical effect of providing an improved computer-based implementation of parallel reality games that provides for generating virtual experiences in parallel reality games in a manner that improves the link between the real world and the parallel virtual world.

[0019] In one embodiment, the game server associated with the parallel reality game can access the data associated with the position of the individual in the real world. The data associated with the position of the individual in the real world can be obtained or derived from any suitable source. The data associated with the position of the individual in the real world can include the position of the mobile device user in the real world. Specifically, the user of a mobile device (such as a smart phone) can optionally provide positioning information (with respect to the geographical location in the real world) to enhance certain location-based features or other functions. Any information optionally provided by the mobile device user can be provided under anonymous conditions to protect the privacy of the user who optionally provides positioning information.

[0020] The data associated with the location of individuals in the real world may also include data associated with the location of players of the parallel reality game. Specifically, the game server may receive positioning information from each of the multiple players during the play of the parallel reality game so that the game server may update the player's location in the parallel virtual world associated with the parallel reality game.

[0021] The game server can analyze data associated with the location of individuals in the real world and generate virtual experiences based on such data. For example, the game server can locate virtual elements in the virtual world for the user, which are collected when the user (or another different user) travels to a specific location in the real world. In some aspects, virtual elements can be used to enhance the experience in the real world. For example, virtual elements can be exchanged or presented for one or more goods or services in the real world. Generating virtual experiences in the virtual world based on real-world actions can give players a reason to travel to a specific location in the real world.

[0022] In a specific implementation, certain real-world actions can be directly and / or indirectly mapped to experiences in the virtual world. For example, weather data from the real world can be directly mapped to virtual weather in the virtual world. Similarly, weather data in the real world can be indirectly mapped to the virtual world, such as when weather conditions in the real world indicate rain, making it more challenging to locate certain virtual elements. As described herein, such mapping can include any real-world action and can be directly or indirectly mapped to one or more experiences in the virtual world, whether or not such experience is related to the real-world action. As another example, a solar eclipse in the real world can be indirectly mapped to the virtual world and result in a virtual experience in which the virtual energy of all players in the virtual world increases. Alternatively, or in combination with the above examples, a solar eclipse can be directly mapped to the virtual world and result in a virtual solar eclipse visible in the virtual world. In this way, a game server can generate a virtual experience in the virtual world based on real-world actions.

[0023] The game server may generate a virtual experience in a parallel virtual world based on other data associated with the real-world action. For example, the game server may create a virtual experience based on real-world actions associated with items of cultural, entertainment, or commercial value, map data, hazard data, weather data, event calendar data, and other suitable data. As an example, the game server may include a virtual experience in a virtual world based on actions associated with real-world items corresponding to locations of public, educational, commercial, or entertainment value, such as public artwork, tourist attractions, scenic spots, libraries, or hiking locations.

[0024] Exemplary Location-Based Parallel Reality Gaming System

[0025] An exemplary computer-implemented location-based gaming system according to an exemplary embodiment of the present disclosure will now be described. This topic will be discussed with reference to a parallel reality game. A parallel reality game is a location-based game having a virtual world geography that is parallel to at least a portion of the real-world geography, so that the movement and actions of the player in the real world affect the actions in the virtual world, and vice versa. Using the disclosure provided herein, one of ordinary skill in the art will understand that the subject matter of the present disclosure can be applied to other gaming systems. In addition, the inherent flexibility of computer-based systems allows for various possible configurations, combinations, and divisions of tasks and functions between and among the components of the system. For example, according to aspects of the present disclosure, systems and methods for modifying or verifying game data can be implemented using a single computing device or across multiple computing devices.

[0026] Figure 1 An exemplary computer-implemented location-based gaming system 100 configured in accordance with an embodiment is shown. The location-based gaming system 100 provides for the interaction of multiple players in a virtual world having a geography parallel to the real world. Specifically, geographic areas in the real world may be directly linked or mapped to corresponding areas in the virtual world. Players may move around in the virtual world by moving to various geographic locations in the real world. For example, the system 100 may track the location of a player in the real world and update the location of the player in the virtual world based on the current location of the player in the real world.

[0027] Figure 2A conceptual diagram of a virtual world 210 parallel to the real world 200 is depicted, which can serve as a game board for a player of a location-based game according to an exemplary embodiment of the present disclosure. As shown, the virtual world 210 can include a geography parallel to the geography of the real world 200. Specifically, a coordinate range that defines a geographic area or space in the real world 200 is mapped to a corresponding coordinate range that defines a virtual space in the virtual world 210. The coordinate range in the real world 200 can be associated with a town, community, city, campus, region, county, continent, the entire earth, or other geographic area. Each geographic coordinate in the geographic coordinate range in the real world 200 is mapped to a corresponding coordinate in the virtual space in the virtual world 210.

[0028] The position of the player in the virtual world 210 corresponds to the position of the player in the real world 200. For example, player A located at position 212 in the real world 200 has a corresponding position 222 in the virtual world 210. Similarly, player B located at position 214 in the real world 200 has a corresponding position 224 in the virtual world 210. When the player moves around within the range of geographic coordinates in the real world, the player also moves around within the range of coordinates defining the virtual space in the virtual world 210. Specifically, a positioning system (e.g., a GPS system) associated with a mobile device carried by the player can be used to track the player's position as the player navigates the range of geographic coordinates in the real world 200. Data associated with the player's position in the real world 200 is used to update the player's position within the corresponding range of coordinates defining the virtual space in the virtual world 210. In this way, the player can navigate a continuous trajectory within the range of coordinates defining the virtual space in the virtual world 210 simply by traveling within the corresponding range of geographic coordinates in the real world 200, without having to check in or regularly update position information at specific discrete locations in the real world 200.

[0029] Location-based games may include multiple game objectives that require players to travel to and / or interact with various virtual elements and / or virtual objects at various locations dispersed in a virtual world. Players may travel to these virtual locations by traveling to corresponding locations of virtual elements or objects in the real world. For example, a positioning system may continuously track the player's location so that as the player continuously navigates the real world, the player also continuously navigates a parallel virtual world. The player may then interact with various virtual elements and / or objects at specific locations to achieve or execute one or more game objectives.

[0030] For example, a game objective may include a player interacting with or otherwise claiming ownership of virtual elements 230 located at various virtual locations in the virtual world 210. These virtual elements 230 may be linked to landmarks, geographic locations, or objects 240 in the real world 200. The real world landmarks or objects 240 may be works of art, monuments, buildings, businesses, libraries, museums, or other suitable real world landmarks or objects. To capture these virtual elements 230, a player may travel to a landmark or geographic location 240 in the real world that is linked to the virtual elements 230 and interact with the virtual elements 230 in the virtual world 210. For example, Figure 2 Player A in the game can travel to a landmark 240 in the real world 200 to interact with a virtual element 230 linked to that particular landmark 240. The interaction with the virtual element 230 can be linked to an action in the real world, such as taking a photo and / or verifying, obtaining, or capturing other information about the landmark or object 240 associated with the virtual element 230. In another example, the player can send the virtual character to a landmark 240 at a particular remote virtual location so that the virtual character can interact with the virtual elements 230, objects 240, or other players 214 there.

[0031] The game objective may include the player using one or more virtual items collected by the player in the location-based game. For example, the player may travel in the virtual world to find virtual items (e.g., weapons, food, medical supplies, soldiers, creatures, or other items) that may help complete the game objective. These virtual items may be found or collected by traveling to different locations in the real world or by completing various actions in the virtual world or the real world. For example, the player may interact with virtual elements 230 in the virtual world to obtain virtual items. In Figure 2 In the embodiment shown in , a player uses a virtual item 232 to capture one or more virtual elements 230. Specifically, a player can deploy a virtual item 232 at a location near a virtual element 230 in the virtual world 210. Deploying one or more virtual items 232 at a location near a virtual element 230 can result in a particular player or a team and / or faction of players capturing the virtual element 230.

[0032] In one particular implementation, players can gather virtual energy as part of a location-based game. Figure 2 As depicted in FIG. 2 , virtual energy 250 may be dispersed at different locations in virtual world 210. Players may collect virtual energy 250 by traveling to corresponding locations of virtual energy 250 in real world 200. Virtual energy 250 may be used to power virtual items and / or perform various game objectives in the game. A player who loses all virtual energy 250 may be temporarily disconnected from the game.

[0033] According to aspects of the present disclosure, location-based games can be large-scale multi-player location-based games, in which each participant in the game shares the same virtual world. Players can be divided into separate teams or factions, and can work together to achieve one or more game goals, such as capturing or claiming ownership of virtual elements. For convenience, all such player groups are referred to as teams herein. In this way, location-based games can essentially be a social game that encourages cooperation between players in the game. In location-based games, players from opposing teams can confront each other. Players can use virtual items to attack or hinder the progress of players of opposing teams. In some cases, players from different teams can cooperate in some shared virtual experiences (e.g., boss battles) to achieve common goals.

[0034] Location-based games can have various features to enhance and encourage gameplay within location-based games. For example, players can accumulate virtual currency or other virtual rewards that can be used throughout the game. As players complete one or more game goals and gain experience in the game, players can advance to various levels. Players can communicate with each other through one or more communication interfaces provided in the game. Players can also obtain enhanced "abilities" or virtual items, which can be used to complete game goals in the game. It should be understood by those of ordinary skill in the art using the disclosure provided herein that various other game features can be included in location-based games without departing from the scope of the present disclosure.

[0035] Return to reference Figure 1 , the computer-implemented location-based gaming system 100 shown includes a client-server architecture in which a gaming server 110 communicates with one or more clients 120 via a network 130. Figure 1 1, two clients 120 are shown, but any number of clients 120 may be connected to the game server 110 via a network 130. The server 110 may host a general game module 112 that controls various aspects of the player's location-based game and receives and processes input from the player in the location-based game. On the client side, each client 120 may include a game module 125 that runs as a game application to provide an interface to the system 100 for the user. The game server 110 sends game data to the client 120 via the network 130 for use by the game module 125 at the client 120 to provide a local version of the game (e.g., a portion of the virtual world specific to the player's location) to the player at a location remote from the game server 110.

[0036] It will be appreciated that the term "module" refers to computer logic for providing desired functionality. Thus, a module may be implemented in hardware, firmware, and / or software that controls a general purpose processor. In one embodiment, a module is a program code file stored on a storage device, loaded into a memory and executed by a processor, or may be provided from a computer program product, such as computer executable instructions, stored in a tangible computer readable storage medium such as a RAM hard disk or optical or magnetic media.

[0037] The game server 110 may be any computing device and may include a processor and a memory. The memory may store instructions that cause the processor to perform operations. The game server 110 may include a game database 115 or may communicate with a game database 115. The game database 115 stores game data used in location-based games or provided to (multiple) clients 120 via the network 130.

[0038] The game data stored in the game database 115 may include: (1) data associated with the virtual world in the location-based game (e.g., image data for representing the virtual world on a display device, geographic coordinates of locations in the virtual world, etc.); (2) data associated with players of the location-based game (e.g., player information, player experience level, player currency, player inventory, current player location in the virtual world / real world, player energy level, player preferences, team information, etc.); (3) data associated with game objectives (e.g., data associated with the current game objective, the state of the game objective, past game objectives, future game objectives, desired game objectives, etc.); (4) data associated with virtual elements in the virtual world (e.g., the positioning of virtual elements, the location of virtual elements, etc.); The game database 115 may include data associated with the game elements, the type of virtual element, the game objective associated with the virtual element, the corresponding actual real-world location information of the virtual element, the behavior of the virtual element, the relevance of the virtual element, etc.); (5) data associated with real-world objects, landmarks, locations linked to virtual-world elements (e.g., the location of real-world objects / landmarks, descriptions of real-world objects / landmarks, relevance of virtual elements linked to real-world objects, etc.); (6) game status (e.g., the current number of players, the current state of game objectives, player leaderboards, etc.); (7) data associated with player actions / inputs (e.g., current player location, past player locations, player movements, player inputs, player queries, player communications, etc.); and (8) any other data used, involved, or obtained during the implementation of the location-based game. The game data stored in the game database 115 may be populated by a system administrator offline or in real time, and / or by data received from users / players of the system 100, such as from one or more clients 120 via a network 130.

[0039] As will be discussed in further detail below, the game server 110 may include or may also communicate with a real-world condition database 117. The real-world condition database 117 may be part of, integrated with, or independent of the game database 115. The real-world condition database 117 stores data associated with real-world conditions, such as individual and / or overall locations of players in the real world; actions associated with locations of cultural or commercial value; map data providing locations of roads, highways, and waterways; current and past locations of individual players; hazard data, weather data; event calendar data; and other suitable data. The data stored in the real-world condition database 117 may be collected or obtained from any suitable source. For example, in one aspect, the real-world condition database 117 may be coupled to, include, or be part of a map database storing map information, such as one or more map databases accessed by a map service. According to another exemplary aspect, the real-world condition database 117 may obtain or access data associated with past and current locations of players, for example, from the game database 115. According to yet another exemplary aspect, the real-world conditions database 117 may be coupled to one or more external data sources or services that periodically provide population data, hazard data, weather data, event calendar data, or other data to the real-world conditions database 117 .

[0040] The game server 110 may be configured to receive requests for game data from one or more clients 120 (e.g., via remote procedure calls (RPCs)) and respond to these requests via the network 130. For example, the game server 110 may encode the game data in one or more data files and provide the data files to the clients 120. In addition, the game server 110 may be configured to receive game data (e.g., player positions, player actions, player input, etc.) from one or more clients 120 via the network 130. For example, the client devices 120 may be configured to periodically send player input, player positions, and other updates to the game server 110, which uses these to update the game data in the game database 115 to reflect the changed conditions of the game.

[0041] As shown, the game server 110 may include a universal game module 112. The universal game module 112 hosts location-based games for all players and serves as an authoritative source of the current state of the location-based games for all players. The universal game module 112 receives game data (e.g., player input, player location, player action, player status, landmark information, etc.) from the client 120 and merges the received game data into the overall location-based game for all players of the location-based game. The universal game module 112 may also manage the delivery of game data to the client 120 via the network 130.

[0042] exist Figure 1 In the embodiment shown in , the game server 110 also includes a locator module 114. The locator module 114 can be part of the general game module 112 or independent of the general game module 112. The locator module 114 is configured to access data associated with real-world actions, analyze the data, and determine a virtual experience in the virtual world based on the data associated with the real-world actions. For example, the locator module 114 can modify the game data stored in the game database 115 to locate the virtual experience in the virtual world based on the data associated with the real-world actions.

[0043] The mapping module 116 creates and updates a map of the real world geography. For example, the mapping module 116 receives images of the real world from one or more client devices 120 and combines these images into a three-dimensional representation of the real world. The game server 110 can use the three-dimensional representation to update the geography of the virtual world to more closely reflect the real world. In addition to the images of the real world, the mapping module 116 additionally receives location data from the positioning device 128 of the client device 120 to assist the mapping module 116 in placing the image data in the correct location in the three-dimensional representation.

[0044] The mapping module 116 creates a three-dimensional representation of the real world by creating map segments, each of which describes a portion of the real world and is labeled with a real-world location (e.g., GPS coordinates). In one embodiment, each map segment is a point cloud or mesh having a three-dimensional geometric shape. The map segments are generated, for example, by a user of the client device 120 capturing a video of a local area around the client device. The image data (e.g., video) is transmitted to the mapping module 116 of the game server 110, which generates the map segments based on the image data. The mapping module 116 can determine the relative positioning (e.g., translation vectors between the map segment coordinate space) between each map segment and nearby (e.g., adjacent map segments) to create a three-dimensional representation of the real world composed of interrelated map segments.

[0045] Other modules can be used with game server 110. Any number of modules can be programmed or otherwise configured to perform the server-side functions described herein. In addition, various components on the server side can be rearranged. For example, game database 115 can be integrated into game server 110. In view of this disclosure, other configurations will be apparent, and this disclosure is not intended to be limited to any particular configuration.

[0046] The client 120 is any computing device that can be used by a player to interact with the gaming system 100. For example, the client 120 can be a wireless device, a personal digital assistant (PDA), a portable gaming device, a cellular phone, a smart phone, a tablet computer, a navigation system, a handheld GPS system, or other such device. In short, the client 120 can be any computer device or system that can execute the game module 125 to allow a player to interact with a virtual world.

[0047] The game module 125 executed by the client 120 provides an interface between the player and the location-based game. The game module 125 can present a user interface on a display device associated with the client 120, which displays a virtual world associated with the game and allows the user to interact in the virtual world to perform various game objectives. The game module 125 can also control various other outputs to allow the player to interact with the game without the player viewing the display screen. For example, the game module 125 can control various audio, vibration or other notifications that allow the player to play the game without looking at the display screen. The game module 125 can access the game data received from the game server 110 to provide an accurate representation of the game to the user. The game module 125 can receive and process player input and provide updates to the game server 110 via the network 130.

[0048] Since the game system 100 is used for location-based games, the client 120 is preferably a portable computing device that the player can easily carry or otherwise transport, such as a smart phone or other portable device. The player can interact with the virtual world simply by carrying or transporting the client 120 in the real world. The client 120 may include a positioning device 128, which monitors the player's location during the game. The positioning device 128 can be any device or circuit for monitoring the location of the client 120. For example, the positioning device 128 can determine the actual or relative location by using a satellite navigation positioning system (e.g., GPS system, Galileo positioning system, global navigation satellite system (GLONASS), Beidou satellite navigation positioning system), an inertial navigation system, a dead reckoning system, based on an IP address, by using triangulation and / or proximity to a cellular tower or WiFi hotspot, and / or other suitable techniques for determining positioning.

[0049] As the player moves around with the client 120 in the real world, the positioning device 128 tracks the player's location and provides the player's location information to the game module 125. The game module 125 updates the player's location in the virtual world based on the player's actual location in the real world. Specifically, the player's location in the virtual world can correspond to the player's location in the real world. The game module 125 can provide the player's location information to the game server 110 via the network 130 so that the general game module 112 keeps tracking the player's location throughout the game.

[0050] The network 130 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof. The network may also include a direct connection between the client 120 and the game server 110. In general, the communication between the game server 110 and the client 120 may be performed via a network interface using any type of wired and / or wireless connection, using various communication protocols (e.g., TCP / IP, HTTP, S1v1TP, FTP), encodings or formats (e.g., HTML, JSON, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0051] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. Those of ordinary skill in the art will recognize that the inherent flexibility of computer-based systems allows for various possible configurations, combinations, and divisions of tasks and functions between and among components. For example, the server processes discussed herein can be implemented using a single server or a combination of servers. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0052] In addition, where the systems and methods discussed herein access and analyze personal information about a user or utilize personal information (such as location information), the user may be provided with an opportunity to control whether a program or feature collects information and to control whether and / or how content is received from the system or other applications. Such information or data will not be collected or used until the user is provided with meaningful notice of what information is to be collected and how the information is to be used. Information will not be collected or used unless the user consents, and the user may revoke or modify consent at any time. Thus, the user may control how information about the user is collected and how the information is used by the application or system. In addition, certain information or data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the identity of the user may be processed so that personally identifiable information of the user cannot be determined.

[0053] Exemplary Mapping Module

[0054] Figure 3 is a block diagram of the mapping module 116 according to one or more embodiments. The mapping module 116 includes a map segmentation module 310, an object localization module 320, and a data store 330. These modules 310-330 enable the mapping module 116 to receive image data from the client device 120 and create or update a three-dimensional representation of the real world.

[0055] The map segmentation module 310 receives image data or map data of a three-dimensional representation of the real world and generates discrete map segments. A map segment may consist of an instance of the image data being received. For example, a user of a client device may capture image data (e.g., a picture or video) of an area surrounding the client device, and the video is transmitted to the map segmentation module 310, which converts the image data into map segments. A single map segment may include image data received from a single client device in a preset time period (e.g., a 30-second video received from a client device). In some embodiments, a map segment may combine image data from multiple client devices captured at the same location and time (e.g., within a preset time period and within a threshold distance). In another embodiment, the map segmentation module 310 sets a fixed size for each map segment and collects image data until the data meets the fixed size. For example, a map segment may represent a cubic volume of the real world, where each side of the cube is a fixed length (e.g., 3 meters). The map segmentation module 310 may determine whether the image data belongs to a specific cubic map segment based on the location data received from the client device, and add the received image data to the segment until all the data required to complete the map segment is received. The map segments are arranged to produce a three-dimensional representation of the local area. Each map segment can be rotated or translated independently of the other map segments to create a representation of the real world that aligns with the current sensor data being captured by the client device 120.

[0056] Once the map segmentation module 310 creates a map segment, the map segment is stored in the data store 330. The map segmentation module 310 can determine the placement of the map segment relative to other map segments, and the data store 330 also stores the relationship. As more image data is received and more map segments are created, previous map segments may be shifted and / or rotated to improve the accuracy of the three-dimensional representation of the real world. As the map segments are shifted, the data store 330 is updated to contain the updated relationship between the map segments.

[0057] The map segmentation module 310 also receives location data of a client device associated with the game server 110 to assist the mapping module 116 in locating virtual objects. The map segmentation module 310 uses the location data (e.g., GPS coordinates in the real world) to retrieve a set of map segments corresponding to the location of the client device from the data store 330. For example, if the location data of the client device indicates that the client device is close to a landmark, the map segmentation module 310 will retrieve all map segments associated with the landmark from the data store 330. In some embodiments, the map segmentation module 310 retrieves a set number of map segments or a set of map segments associated with a location within a threshold distance of the received location data. For example, the map segmentation module 310 can retrieve a map segment that includes the location of the client device and the 8 closest map segments. In another example, the map segmentation module 310 retrieves a set of map segments that include locations within a 100-foot radius of the location data. The set of map segments is used to locate virtual objects for display on the client device.

[0058] The object positioning module 320 places the virtual object within the map segment and updates the positioning of the virtual object when the map segmentation module 310 receives additional image data. The administrator of the game server 110 can create a virtual object using a set of object parameters. The object parameters specify the rules for placing the virtual object. For example, a virtual object as paint may include an object parameter indicating that the paint is placed on a flat vertical surface (such as a wall). A virtual object as a fence may include an object parameter indicating that the fence is straight and continuous.

[0059] For each virtual object, the object positioning module 320 generates a relationship vector between the virtual object and a set of map fragments. The set of map fragments can be the map fragments closest to the virtual object. In some embodiments, the set of map fragments is selected based on object parameters. The relationship vector indicates the position of the virtual object relative to each map fragment in the set of map fragments. The relationship vector can be from any point in the map fragment (such as the center or edge) to the virtual object. An example relationship vector is shown in Figure 4. In some embodiments, the object positioning module 320 can also detect real-world surfaces or objects in the set of map fragments and generate a relationship vector between the virtual object and the real-world surfaces or objects.

[0060] In some embodiments, the object localization module 320 may have preset thresholds for lower and upper limits on the number of relationship vectors to determine. Having a large number of relationship vectors may improve the accuracy of map segment alignment, but requires a large amount of computing power to solve. The number of relationship vectors used is adjusted for accuracy while keeping computing power low, and therefore achieving low lag.

[0061] The object positioning module 320 also determines the relative weight of the relationship vector based on the object parameters of the virtual object. The weight of the relationship vector indicates the strictness with which the positioning relationship between the virtual object and the real-world object or map segment is maintained as the map segment is shifted and updated. For example, if the virtual object is a flower and has a relationship vector with the ground below it, the map segment it is in, and the map segment adjacent to the map segment it is in, the relationship vector between the flower and the ground below can be highly weighted. The high weight ensures that the flower will remain on the ground as the map segment is shifted or updated. Since the object parameters of the virtual flower indicate that the positioning of the flower is not related to anything in the adjacent map segment, the weight of the relationship vector toward the adjacent map segment may be low. In an example where the virtual flower is part of a row of other virtual flowers (such as extending along a sidewalk in the real world), each virtual flower can have a highly weighted relationship vector with the other map segments containing the row of virtual flowers. This high weight with other map segments allows the row of flowers to be kept together, such as keeping the virtual flowers evenly distributed or in a straight line. The row of flowers can also have a highly weighted relationship vector with the real-world sidewalk to keep the flowers along the sidewalk.

[0062] When a player of a game associated with the game server 110 triggers placement of a virtual object (e.g., by being at a certain location or activating a game feature), the object positioning module 320 determines the location on the screen of the client device associated with the player where the virtual object is to be displayed. The object positioning module 320 determines the placement of the virtual object that satisfies the weighted relationship vectors so that the highly weighted vectors remain nearly the same. The object positioning module 320 can determine the location on the client device where the virtual object is to be displayed, such as by solving an equation that includes the weighted relationship vectors. The object positioning module 320 can determine the location at which the virtual object is to be placed at a specific coordinate of a map segment, and then determine the corresponding placement of the virtual object on the client device screen.

[0063] The data store 330 stores data used by the modules 310 and 320. The data store 330 may contain previous object placements so that those placements may be reused on virtual objects having similar object parameters. The data store 330 may also store a history of the location of a map segment so that if the location of a map segment changes based on new information being received by the mapping module 116, the location of the map segment may be restored to its previous location if needed, or the location of the map segment may be tracked over time. The data store 330 additionally stores object parameters. In some embodiments, the data store 330 additionally stores location data from the client device 120 that provided the image data for the map segment.

[0064] Figure 4A and Figure 4BTwo examples of using relationship vectors to locate virtual objects according to one or more embodiments are shown. Figure 4A and Figure 4B In both, map segment 405 and adjacent map segment 410 have virtual objects 425 and 435, respectively. The map segments 405 and 410 on the left represent a three-dimensional representation of the real world at a time before updated image data was received by the game server 110. The map segments 405 and 410 on the right have been displaced or drifted due to new image data being added to the three-dimensional representation of the real world. The positioning of virtual objects 425 and 435 may change before and after the displacement of the map segments. Figure 4A and Figure 4B In the example shown in , the new image data indicates that surface 415 and surface 430 are straight walls, and therefore the map fragments are shifted to correct the angle of the walls so that surfaces 415 and 430 are linear.

[0065] Figure 4A is an example of using a relationship vector 420 for virtual object positioning, which only defines the relationship between a virtual object 425 and the segment 405 where the virtual object was originally located. The relationship vector 420 indicates the positioning of the virtual object 425 relative to the center of the segment 405. Figure 4A In the example of FIG. 4 , since there is no relationship vector from virtual object 425 to map segment 410, virtual object 425 is moved away from virtual object 435 when segment 405 is displaced. The lack of a relationship vector between each virtual object and each map segment prevents the virtual objects from remaining aligned or evenly distributed when map segment 405 is displaced. Instead, virtual object 425 is displaced with map segment 405 without remaining aligned with object 435.

[0066] exist Figure 4B , virtual object 425 has a relationship vector with both the map segment 405 in which it is located and the adjacent map segment 410. Similarly, virtual object 435 has a relationship vector with the map segment 410 in which it is located and the adjacent map segment 405. In order to maintain the alignment of virtual objects 425 and 435 that have a strong positioning relationship with each other, such as they are different parts of a larger structure (e.g., posts that make up a fence). So that virtual objects 425 and 435 maintain their relative positioning and alignment, the relationship vectors can be weighted approximately evenly (e.g., within 10% of each other). Therefore, when map segment 405 is shifted to align surfaces 415 and 420, the virtual objects are able to remain colinear and evenly distributed.

[0067] In contrast, a virtual object that has a strong positioning relationship with a physical object (e.g., a virtual hat on a physical statue or a virtual poster on a physical wall) can have its relationship vector with the closest map fragment set to about one (e.g., exactly or close to one, such as greater than 0.9), and the weights of other relationship vectors set to about zero (e.g., exactly or close to zero, such as less than 0.1). Thus, the positioning of the virtual object is closely related to the map fragment that includes the physical object it is related to, so that the virtual object can be accurately positioned (e.g., the virtual hat stays on the head of the physical statue, rather than floating in the air to one side).

[0068] In yet another example, a hybrid configuration where objects have a general positioning relationship with physical objects but are also slightly related to each other can weight map fragments based on proximity to virtual objects (e.g., the map fragment closest to the virtual object can have a weight of 0.4, the next closest map fragment can have a weight of 0.3, etc.) Thus, virtual object positioning can be "stretched" by changes in nearby map fragments, but remain primarily rooted in the physical location in the nearest map fragment.

[0069] In some embodiments, to ensure additional accuracy of the alignment between virtual objects 425 and 435, a relationship vector between each virtual object can be generated. These relationship vectors can be weighted more highly than the relationship vectors between objects and map segments so that the vectors between objects do not change length or direction when a map segment shift occurs. For example, the relative positioning of map segments can be periodically recalculated to account for sensor drift, newly available map data, and other inaccuracies, and the positioning of virtual objects can be updated using a weighted combination of relationship vectors to maintain the desired spatial relationship between virtual objects and the physical environment.

[0070] Figure 5 is a flow chart illustrating a method 500 for positioning a virtual object in accordance with one or more embodiments. Figure 5 This is not a comprehensive representation of all steps that may occur in the process of aligning map segments and objects. Some steps may be performed in a different order, in parallel, or not performed at all. In addition, the method may include using other modules and steps before, during, or after the above process flow. For example, method 500 may additionally include identifying virtual objects within the map segment and weighting each relationship vector based on the identity of the virtual object. In some embodiments, the steps of method 500 are performed by mapping module 116 of game server 110.

[0071] The game server 110 determines 510 the real-world location of a client device (e.g., client device 120). The location of the client device may be determined by the game server 110 querying a positioning device 128 of the client device 120 via a network 130. In some embodiments, the location of the client device is represented as GPS coordinates. In another embodiment, the location of the client device may be represented as a vector related to the distance and direction between the client device and a known point (such as a landmark).

[0072] The game server 110 retrieves 520 a set of map segments based on the location of the client device. The set of map segments may include map segments within a threshold distance of the location of the client device. In some embodiments, the game server 110 may set the number of map segments to retrieve. For example, after receiving the location data from the client device, the game server 110 may retrieve the six map segments closest to the location of the client device. Each map segment is a point cloud or mesh that is a three-dimensional representation of the real world around the location of the client device.

[0073] The game server 110 determines 530 a virtual object to be displayed on the client device. The virtual object may be determined, for example, by a game event triggered by a user of the client device. In some embodiments, the location of the client device may be associated with the virtual object, such that when the location data of the client device indicates that the user is near a particular location, it is determined that the virtual object is to be displayed.

[0074] The game server 110 obtains 540 a relationship vector between the virtual object and each map fragment in the set of map fragments. For example, if six map fragments are retrieved by the game server 110, the same server 110 obtains a vector from each map fragment to the virtual object. The relationship vector indicates the distance and direction relationship between the virtual object and each of the retrieved map fragments.

[0075] The game server 110 weights 550 each relationship vector based on object parameters of the virtual object. The object parameters of the virtual object indicate factors used to locate the virtual object. For example, the object parameters may indicate that the virtual object is located near another object (virtual or real world) within the same map segment. Therefore, the relationship vector between the virtual object and the map segment it is in will have a high weight to ensure that the virtual object stays within the current map segment.

[0076] The game server 110 determines 560 the location at which the virtual object is to be displayed on the display of the client device based on the weighted relationship vectors. To position the object, the game server 110 can keep the relationship vectors with high weights the same length and direction and allow other lower weighted relationship vectors to change in length and direction. In some embodiments, the relationship vectors can be sorted based on their weights to determine the location at which the virtual object is to be displayed.

[0077] The game server 110 provides 570 the virtual object for display on the client device at the determined location. Instructions indicating the location of the virtual object can be sent to the client device via the network. The instructions cause the client device to display the virtual object at the determined location of the client device display.

[0078] Example Compute Device Architecture

[0079] Figure 6 is a block diagram showing components of an example machine capable of reading instructions from a machine-readable medium and executing those instructions in a processor (or controller). Specifically, Figure 6 A schematic representation of a machine is shown in the example form of a computer system 600. The computer system 600 may be used to execute instructions 624 (e.g., program code or software) to cause the machine to perform any one or more of the methods (or processes) described herein, including those methods associated with and described with respect to components (or modules) of the server 110 and / or the client 120.

[0080] The machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a smart phone, a network router, a switch or a bridge, a cellular phone tower, or any machine capable of executing instructions 624 (sequential or otherwise) that specify actions to be taken by the machine. In addition, while only a single machine is shown, the term "machine" should also be taken to include any collection of machines that individually or collectively execute instructions 624 to perform any one or more of the methodologies discussed herein.

[0081] The example computer system 600 includes one or more processing units (typically one or more processors 602). The processor 602 is, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a controller, a state machine, one or more application specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination of these. Any reference to the processor 602 herein may refer to a single processor or multiple processors. The computer system 600 also includes a main memory 604. The computer system may include a storage unit 616. The processor 602, the memory 604, and the storage unit 616 communicate via a bus 608.

[0082] In addition, the computer system 600 may include a static memory 606, a display driver 610 (e.g., for driving a plasma display panel (PDP), a liquid crystal display (LCD), or a projector). The computer system 600 may also include an alphanumeric input device 612 (e.g., a keyboard), a cursor control device 614 (e.g., a mouse, a trackball, a joystick, a motion sensor, or other pointing instrument), a signal generating device 618 (e.g., a speaker), and a network interface device 620, which are also configured to communicate via the bus 608.

[0083] The storage unit 616 includes a machine-readable medium 622 on which are stored instructions 624 (e.g., software) embodying any one or more of the methods or functions described herein. The instructions 624 may also reside completely or at least partially within the main memory 604 or within the processor 602 (e.g., within a cache memory of the processor) during execution thereof by the computer system 600, the main memory 604 and the processor 602 also constituting machine-readable media. The instructions 624 may be sent or received over the network 670 via the network interface device 620.

[0084] Although in the example embodiment, the machine-readable medium 622 is shown as a single medium, the term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) that can store the instructions 624. The term "machine-readable medium" should also be taken to include any medium that can store the instructions 624 for execution by a machine and cause the machine to perform one or more of the methods disclosed herein. The term "machine-readable medium" includes, but is not limited to, data storage repositories in the form of solid-state memories, optical media, and magnetic media.

[0085] Additional considerations

[0086] Some parts of the above description describe embodiments in terms of algorithmic processes or operations. These algorithmic descriptions and representations are often used by those skilled in the art of computers to effectively convey the essence of their work to other skilled in the art. Although described in a functional, computational or logical manner, these operations are understood to be implemented by a computer program, which includes instructions executed by a processor or equivalent circuit, microcode, etc. In addition, it has sometimes proven convenient to refer to these arrangements of functional operations as modules without loss of generality.

[0087] As used herein, any reference to "one embodiment" or "embodiment" means that a particular element, feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment. The phrase "in one embodiment" that appears in various places in the specification does not necessarily refer to the same embodiment. Similarly, the use of "one" or "an" before an element or component is only for convenience. Unless otherwise apparent, the description should be understood to mean that there are one or more elements or components.

[0088] When values ​​are described as "approximately" or "substantially" (or derivatives thereof), unless otherwise apparent from the context, these values ​​should be understood to be accurate to + / - 10%. For example, "approximately ten" should be understood to mean "within the range of from nine to eleven".

[0089] As used herein, the terms "comprises," "includes," "contains," "has," "having," or any other variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent in such process, method, article, or apparatus. In addition, unless expressly stated to the contrary, "or" refers to an inclusive or and not an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and both A and B are true (or exist).

[0090] Although the subject matter has been described in detail with respect to specific exemplary embodiments and methods thereof, it will be appreciated that those skilled in the art, after gaining an understanding of the foregoing, can readily generate changes, modifications, and equivalents to such embodiments. Therefore, the scope of the present disclosure is illustrative rather than limiting, and the subject disclosure does not exclude the inclusion of such modifications, modifications, and / or additions to the subject matter as would be apparent to one of ordinary skill in the art.

Claims

1. A method for displaying a virtual object, comprising: Determine the location of the client device; retrieving a set of map segments based on the location of the client device; determining a virtual object to be displayed on the client device; Obtain a relationship vector between the virtual object and each map segment in the map segment set; weighting each relationship vector based on an object parameter of the virtual object; determining a location at which to display the virtual object based on the weighted relationship vector; as well as The virtual object is provided for display on the client device at the determined location. 2 . The method of claim 1 , wherein the retrieved set of map segments is a set of map segments representing areas in the real world within a threshold distance of the determined location of the client device. 3 . The method according to claim 1 , further comprising obtaining a relationship vector between the virtual object and a second virtual object to be displayed, the second virtual object being within the retrieved set of map segments. The method of claim 1 , wherein determining a virtual object to display is based on the location of the client device. The method of claim 1 , wherein determining the virtual object to display is based on a gaming event.

6. The method according to claim 1, wherein: In response to the object parameters indicating a strong positional relationship between the virtual object and the physical object, a weight of the relationship vector for the map fragment including the representation of the physical object is approximately one and the remaining relationship vectors are approximately zero.

7. The method according to claim 1, wherein: In response to the object parameters indicating a strong positioning relationship between the virtual object and another virtual object, the relationship vectors are approximately uniformly weighted.

8. The method of claim 1, wherein providing the virtual object for display comprises sending instructions to the client device, the instructions comprising a position on a display of the client device that corresponds to the determined position of the virtual object.

9. The method of claim 1, wherein each map segment comprises a point cloud or a mesh.

10. The method according to claim 1, further comprising: Updating the relative positioning of the map fragment; as well as The location at which the virtual object is to be displayed is re-determined using the updated relative location of the map segment and the weighted relationship vector.

11. A non-transitory computer-readable storage medium storing instructions for determining a location at which a virtual object is to be displayed, the instructions, when executed by a computer system, causing the computer system to perform operations comprising: Determine the location of the client device; retrieving a set of map segments based on the location of the client device; determining a virtual object to be displayed on the client device; Obtain a relationship vector between the virtual object and each map segment in the map segment set; weighting each relationship vector based on an object parameter of the virtual object; determining a location at which to display the virtual object based on the weighted relationship vector; as well as The virtual object is provided for display on the client device at the determined location.

12. The non-transitory computer-readable storage medium of claim 11, wherein the retrieved set of map segments is a set of map segments representing areas in the real world within a threshold distance of the determined location of the client device.

13. The non-transitory computer-readable storage medium of claim 11, further comprising obtaining a relationship vector to be displayed between the virtual object and a second virtual object, the second virtual object being within the retrieved set of map segments.

14. The non-transitory computer-readable storage medium of claim 11, wherein determining a virtual object to display is based on the location of the client device.

15. The non-transitory computer-readable storage medium of claim 11, wherein determining the virtual object to display is based on a gaming event.

16. The non-transitory computer-readable storage medium of claim 11, wherein: In response to the object parameters indicating a strong positional relationship between the virtual object and the physical object, a weight of the relationship vector for the map fragment including the representation of the physical object is approximately one and the remaining relationship vectors are approximately zero.

17. The non-transitory computer-readable storage medium of claim 11, wherein: In response to the object parameters indicating a strong positioning relationship between the virtual object and another virtual object, the relationship vectors are approximately uniformly weighted.

18. The non-transitory computer-readable storage medium of claim 11, wherein providing the virtual object for display comprises sending instructions to the client device, the instructions comprising a position on a display of the client device corresponding to the determined position of the virtual object.

19. The non-transitory computer-readable storage medium of claim 11, wherein each map segment comprises a point cloud or a mesh.

20. The non-transitory computer readable storage medium of claim 11, wherein the operations further comprise: Updating the relative positioning of the map fragment; as well as The location at which the virtual object is to be displayed is re-determined using the updated relative location of the map segment and the weighted relationship vector.

Images