Data layer protocol for selecting data transmission paths

Abstract

A data flow layer protocol is implemented by one or more devices to optimize how one or more devices process datagrams for network communication. The data flow layer considers various network paths available for data flow. A device implementing the data flow layer selects one or more of the available network paths to provide low latency in data communication with other devices. This device may sample various available network paths to determine path metrics (e.g., latency) and select one or more network paths based on those metrics. Available network paths may include paths through one or more intermediary nodes, such as paths through game servers, paths through cell towers, and paths through networks.

Description

Technology Field

[0001] (Cross-reference to related applications)

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 951,926 filed on December 20, 2019, the entirety of which is incorporated by reference.

[0003] (Technology field)

[0004] The present invention relates to a computer network protocol, and more specifically, to a protocol for providing low-latency wireless communication between devices that are physically close to each other. Background Technology

[0005] A computer network is an interconnected set of computing devices that exchange data, such as the Internet. Communication protocols, such as the User Datagram Protocol (UDP), define a system of rules for exchanging data using computer networks. UDP adheres to a connectionless communication model in which the delivery, ordering, or non-redundancy of datagrams is not guaranteed. A datagram is the basic unit of communication and includes a header and a payload. The header is metadata that defines aspects of the datagram, such as the source port, destination port, length, and checksum. The payload is the data delivered by the datagram. Computing devices communicating using UDP transmit datagrams to each other over the computer network.

[0006] Connectionless communication protocols such as UDP generally have lower overhead and latency than connection-oriented communication protocols such as the Transmission Control Protocol (TCP), which establish a connection between computing devices before transmitting data. However, existing connectionless communication protocols are unsuitable for data transmissions that require lower latency than is acceptable to existing technologies. For example, an augmented reality (AR) environment streaming at 60 frames per second (FPS) may require much lower latency than current technologies provide. In such an AR environment, frames may be spaced at intervals of approximately 16 milliseconds, but current network protocols typically provide a latency of about 100 milliseconds (or more).

[0007] As such, using existing technology, users may encounter some degree of latency when interacting in an AR environment. This latency can lead to interactions with outdated AR location data. For example, in a parallel reality game, a player may view an AR object at an old location (e.g., where the object was 100 milliseconds ago), but the AR location data actually holds the object's new location (e.g., the AR object was moved by another player). In communication between the client and the server hosting or coordinating the parallel reality game, this latency can lead to a frustrating user experience. This issue can be particularly severe when two or more users participate in a parallel reality game, as this latency can cause a perceptible delay between the actions of one player appearing within the view of other players in the AR environment. Furthermore, as the number of players increases, the discrepancies caused by this latency can worsen. means of solving the problem

[0008] Augmented reality (AR) systems complement views of the real world with computer-generated content. Integrating AR into parallel reality games can enhance the convergence between the real and virtual worlds. Furthermore, AR can increase interaction among players by providing them with the opportunity to participate in shared, interactive game experiences. For example, in a tank battle game, players can navigate virtual tanks around real-world locations and attempt to destroy each other's tanks. The movement of the tanks can be restricted by real-world terrain (e.g., tanks move slower when crossing rivers, faster on roads, and cannot pass through walls).

[0009] Existing AR session technologies involve a server maintaining a master state and periodically synchronizing the local state of the environment on the client with the master state over a network (e.g., the Internet). However, synchronizing the device's local state can take a significant amount of time (e.g., ~100 milliseconds), which can be detrimental to the gaming experience. Players are effectively interacting with a past game state rather than the current game state. This issue can be particularly severe when two or more users participate in parallel reality games, as latency causes a perceptible delay between one player's actions appearing in another player's view. For example, if one player moves an AR object in the world, another player may not see that it has moved until 100 milliseconds (or more) have passed, which is a human-perceptible delay. Consequently, other players may attempt to interact with objects at the object's previous location and may become frustrated when the game compensates for the latency (e.g., by refusing to implement the action requested by the player or by canceling it after initially implementing it when the player's client synchronizes with the server).

[0010] This problem and other problems can be solved by implementing a dataflow hierarchy protocol that optimizes how one or more devices process datagrams. The dataflow hierarchy includes various network paths for dataflow. A device implementing the dataflow hierarchy may utilize one or more network paths to ensure low latency in data communication with other devices. In some embodiments, the device samples various network paths to determine the network path with the minimum latency for use in data communication through an intermediary node. The various network paths may include paths through one or more intermediary nodes, such as paths through a game server, paths through a cell tower, and paths through a network. Network paths may also include direct connections between devices (e.g., using Bluetooth). Using the dataflow hierarchy protocol can improve latency to the millisecond level (e.g., less than 5 milliseconds). Additionally, the dataflow hierarchy can optimize bandwidth usage based on bandwidth availability. Brief explanation of the drawing

[0011] FIG. 1 illustrates a network environment in which a data flow layer protocol can be implemented according to one embodiment. FIG. 2 is a block diagram illustrating a client according to one embodiment. FIG. 3 is a block diagram illustrating a game server according to one embodiment. FIG. 4 is a block diagram illustrating a cell tower according to one embodiment. FIG. 5 illustrates a process for using a data flow layer protocol according to one embodiment. FIG. 6 illustrates a process for transmitting data between clients within a geographical area around a cell tower according to one embodiment. FIG. 7 is a high-level block diagram illustrating an exemplary computer suitable for use within the computer network illustrated in FIG. 1, according to one embodiment. These drawings and the description below illustrate specific embodiments merely by way of example. Those skilled in the art will readily understand from the description below that alternative embodiments of this structure and method may be adopted without departing from the principles described. Several embodiments will be referenced, examples of which are illustrated in the accompanying drawings. Where feasible, similar or similar reference numbers are used in the drawings to indicate similar or similar functions. If elements share a common number followed by a different letter, those elements are similar or identical. Referring to a number alone indicates any one or any combination of these elements. Specific details for implementing the invention

[0012] Network environment

[0013] As disclosed herein, a data flow layer protocol can provide reduced computer network latency compared to previous approaches (e.g., in one embodiment, the latency is ~10 milliseconds). FIG. 1 illustrates a computer network (100) communicating using a data flow layer protocol according to one embodiment. This figure shows a simplified example using a block diagram for clarity. The computer network (100) includes two clients (110), a game database (115), a game server (120), a cell tower (130), and a local area network (LAN) (140). In other embodiments, the computer network may include fewer components or additional or other components such as additional clients (110), additional game servers (120), additional cell towers (130), or other network nodes. The clients (110) communicate with the game server (120) hosting the parallel reality game. The clients (110) communicate with each other during the parallel reality game when a player interacts with other players in the parallel reality game.

[0014] A client (110) is a computing device such as a personal computer, laptop, tablet computer, smartphone, etc. Clients (110) can communicate with each other using a data flow layer protocol, for example, while playing a parallel reality game. In one embodiment, each client (110) includes a local AR module and a game server (120) includes a master AR module. Each local AR module sends AR data to a local AR module on another client (110) and / or to a master AR module on the game server (120). The client (110) will be further described in FIG. 2.

[0015] A game server (120) is a computing device that hosts a parallel reality game played by various clients (110). The game server (120) establishes connections between clients (110) to receive and transmit information regarding the state of the parallel reality game. The game server (120) stores player profiles, game states, maps (real and virtual), and other information related to the parallel reality game in a game database (115). The game server (120) will be further described in FIG. 3.

[0016] A cell tower (130) is a network configured for data communication between clients (110). The cell tower (130) can facilitate data transmission between clients (110), for example, via UDP. In one or more embodiments, the cell tower (130) has a local AR module configured to help host a parallel reality game between clients (110) connected to the cell tower (130). The cell tower (130) can share game state with a game server (120), for example, periodically or in real time. The cell tower (130) is further described in FIG. 4.

[0017] A LAN (140) is a local network that clients can connect to. The LAN (140) includes one or more routers that clients (110) can connect to. The LAN (140) operates via Wi-Fi and Ethernet. Similar to a cell tower (130), the LAN (140) may include a local AR module configured to help host parallel reality games between clients (110) connected to the LAN. Likewise, the LAN (140) may share game state with a game server (120).

[0018] Clients (110) may communicate with each other using a data flow layer protocol or, in other embodiments, may communicate with each other using a different protocol. To communicate with each other, two or more clients transmit data through one or more paths. As illustrated in FIG. 1, there are three paths in the computer network (100) between client (110A) and client (110B). In the first path, clients (110) may communicate with each other through a game server (120) using, for example, TCP. In the second path, clients (110) may communicate with each other through a cell tower (130) using, for example, TCP or UDP. In the third path, clients (110) may communicate with each other through a LAN (140) using, for example, TCP or UDP. In other embodiments (not illustrated), other paths for communication between clients may exist, for example, via Ethernet, Bluetooth, Near Field Communication (NFC), etc. In each path, the client (110) communicates through an intermediary node, such as a game server (120), a cell tower (130), or a LAN (140). The data flow layer protocol evaluates available communication paths between the clients (110) and optimizes which of the one or more paths to use for data communication between the clients (110).

[0019] In one embodiment, the client (110) may be routed through a game server (120) or peer-to-peer (P2P) communication. Communication routed through the game server (120) may be sent from the first client (110A) to the game server (120) via a cell tower (130) and then sent again to the second client (110B) via the cell tower (130). In contrast, P2P communication may be sent from the first client (110A) to the cell tower (130) and then sent directly to the second client (110B). It should be understood that in some cases, the communication may pass through other intermediary devices, such as a signal booster. As used herein, if the communication is routed to the destination client (110B) without passing through the game server (120), the communication is considered P2P. For example, if the destination client (110B) is connected to the same cell tower (130) as the transmission client (110A), the message (e.g., datagram) can be transmitted via P2P, otherwise it can be routed through the game server (120). In another embodiment, the client (110) communicates entirely using P2P. Furthermore, in some embodiments, other techniques for P2P communication, including UDP hole punching or Network Address Translation (NAT), may be used to establish a connection between two or more clients (110).

[0020] In one embodiment, the client (110) synchronizes its IP address using a coordination service (e.g., hosted on a game server (120) and communicated via TCP). Then, the client (110) can communicate using a public facing IP address or a LAN (140) (e.g. via UDP). For example, the first client (110A) can send a request to the coordination service via TCP to participate in a local AR shared environment. The coordination service may provide the first client (110A) with the IP address of a second client (110B) connected to the AR environment (e.g. via the same cell tower (130)). Additionally, the coordination service may provide the first client's IP address to the second client (110B), or the first client (110A) may provide it directly using the second client's IP address (provided by the coordination service). In some embodiments, the coordination service may urge the second client (110B) to approve the first client (110A) before the IP address of the second client is provided (e.g., by requesting user confirmation or by checking the list of clients (110) approved to connect with the second client (110B).

[0021] In one embodiment, a client (110) communicates via UDP through a public pacing IP address, and this communication is protected by public-key cryptography. In a specific embodiment, elliptic curve cryptography (ECC) is implemented. ECC uses an elliptic curve over a finite field, which is a unidirectional and deterministic function, to encrypt a private key that is to be publicly provided to other computing systems and networks. One advantage of ECC over other public-key cryptography methodologies is that, because ECC is a simpler function, it enables a smaller key size, which consequently reduces bandwidth usage.

[0022] client

[0023] FIG. 2 is a block diagram of a client (110) according to one embodiment. The client (110) may be any portable computing device that can be used by a player to interface with a game server (120). For example, the client (110) may be a wireless device, a personal digital assistant (PDA), a portable game console, a mobile phone, a smartphone, a tablet, a navigation system, a portable GPS system, a wearable computing device, a display having one or more processors, or other such devices. In other examples, the client (110) includes a conventional computer system such as a desktop or laptop computer. Additionally, the client (110) may be a vehicle equipped with a computing device. In short, the client (110) may be any computer device or system that enables a player to interact with the game server (120). As a computing device, the client (110) may include one or more processors and one or more computer-readable storage media. The computer-readable storage media may store instructions that cause the processor to perform tasks. The client (110) is preferably a portable computing device that the player can easily carry or transport in other ways, such as a smartphone or tablet.

[0024] The client (110) communicates with the game server (120) to provide sensor data of the physical environment to the game server (120). The client (110) includes a camera assembly (210) that captures image data from a two-dimensional scene of the physical environment where the client (110) is located. Additionally, the client (110) includes a depth estimation model, for example, a machine learning model trained by the game server (120). In the embodiment illustrated in FIG. 1, each client (110) includes software components such as a game module (220) and a positioning module (230). The client (110) may include various other input / output devices for receiving information from the player and / or providing information to the player. Exemplary input / output devices include a display screen, a touch screen, a touchpad, a data input key, a speaker, and a microphone suitable for voice recognition. Additionally, the client (110) may include various other sensors for recording data from the client (110), including but not limited to motion sensors, accelerometers, gyroscopes, other inertial measurement units (IMUs), barometers, positioning systems, thermometers, light sensors, etc. The client (110) may further include a network interface for providing communication over a network. The network interface may include any suitable component for interfacing with one or more networks, such as a transmitter, a receiver, a port, a controller, an antenna, or other suitable components.

[0025] The camera assembly (210) captures image data of a scene in the environment where the client (110) is located. The camera assembly (210) may utilize various photo sensors having various color capture ranges with various capture rates. The camera assembly (210) may include a wide-angle lens or a telephoto lens. The camera assembly (210) may be configured to capture a single image or video as image data. Additionally, the orientation of the camera assembly (210) may be parallel to the ground, and the camera assembly (210) may be aimed at the horizon. The camera assembly (210) captures image data and shares the image data with a computing device on the client (110). The image data may be accompanied by metadata describing other details of the image data, including sensor data (e.g., temperature, brightness of the environment) or capture data (e.g., exposure, warmth, shutter speed, focal length, capture time, etc.). The camera assembly (210) may include one or more cameras capable of capturing image data. In one example, the camera assembly (210) includes one camera and is configured to capture monocular image data. In another example, the camera assembly (210) includes two cameras and is configured to capture stereoscopic image data. In various other embodiments, the camera assembly (210) includes a plurality of cameras, each configured to capture image data.

[0026] The game module (220) provides an interface to the player for participating in a parallel reality game. The game server (120) transmits game data to the client (110) for use by the game module (220) located in the client (110) to provide a local version of the game to the player located far from the game server (120). The game server (120) may include a network interface for providing various communication paths described in FIG. 1. The network interface may include any suitable component for interfacing with one or more networks, such as a transmitter, a receiver, a port, a controller, an antenna, or other suitable components.

[0027] A game module (220) executed by a client (110) provides an interface between the player and the parallel reality game. The game module (220) may provide a user interface on a display device associated with the client (110) that displays a virtual world associated with the game (e.g., renders an image of the virtual world) and enables the user to interact with the virtual world to perform various game objectives. In some other embodiments, the game module (220) provides image data from the real world that is augmented with virtual elements from the parallel reality game (e.g., captured by a camera assembly (210)). In these embodiments, the game module (220) may generate virtual content and / or adjust virtual content according to other information received from other components of the client. For example, the game module (220) may adjust virtual objects to be displayed on the user interface according to a depth map of the scene captured from the image data (e.g., determined by a depth estimation model).

[0028] Additionally, the game module (220) can control various other outputs to enable the player to interact with the game without requiring the player to look at the display screen. For example, the game module (220) can control various audio, vibration, or other notifications that allow the player to play the game without looking at the display screen. The game module (220) can access game data received from the game server (120) to provide the user with an accurate representation of the game. The game module (220) can receive and process player input and provide updates to the game server (120) over the network. Additionally, the game module (220) can generate and / or adjust game content to be displayed by the client (110). For example, the game module (220) can generate virtual elements based on depth information (e.g., determined by a depth estimation model).

[0029] The positioning module (230) may be any device or circuit for monitoring the location of the client (110). For example, the positioning module (230) may determine the actual or relative location using a satellite navigation positioning system (e.g., GPS system, Galileo positioning system, Global Navigation Satellite System (GLONASS), BeiDou Satellite Navigation and Positioning system), an inertial navigation system, a dead reckoning system, based on an IP address, using triangulation and / or proximity to a cellular tower or Wi-Fi hotspot, and / or other suitable techniques for determining location. The positioning module (230) may further include various other sensors that can help to accurately locate the client (110). In some embodiments, the client (110) locally stores a portion of the map used in a parallel reality game. The positioning module (230) can locate the client (110)'s location against the locally stored map. The positioning module (230) can also periodically synchronize a locally stored (or cached) map with map data maintained by the game server (120).

[0030] When a player moves with the client (110) in the real world, the positioning module (230) tracks the player's location and provides the player location information to the game module (220). The game module (220) updates the player's location within the virtual world associated with the game based on the player's actual location in the real world. Thus, the player can interact with the virtual world simply by carrying or transporting the client (110) in the real world. In particular, the player's location in the virtual world may correspond to the player's location in the real world. The game module (220) may provide the player location information to the game server (120) via a network. In response, the game server (120) may implement various techniques to verify the client's (110) location to prevent a fraudster from spoofing the client's (110) location. The player must understand that the location information associated with the player is used only when the player is notified of how that location information should be used in the context of the game (e.g., to update the player's location in the virtual world) and is granted permission. In addition, any location information related to the player is stored and maintained in a manner that protects the player's personal information.

[0031] The data flow module (240) communicates with other computing components. The data flow module (240) can establish communication using any type of wired and / or wireless connection using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encoding or format (e.g., HTML, XML, JSON), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0032] In one or more embodiments, the data flow module (240) communicates with the game server (120). The data flow module (240) provides the game server (120) with actions, updates to the game state, requests for various content items (e.g., for a parallel reality game), or other communications. The data flow module (240) establishes a connection with the game server (120), for example, via TCP. TCP provides a robust data transmission method. Using TCP, the data flow module (240) can detect packets lost during data transmission and request the retransmission of the lost packets. Furthermore, TCP enables the reconstruction of received sequential data packets out of order to generate the intended message.

[0033] Additionally, the data flow module (240) manages communication with one or more other clients (110). The data flow module (240) identifies one or more paths for communicating with other clients (110). At any given time, the paths available to the client (110) may include a path through the game server (120), a path through the cell tower (130), a path through the LAN (140), another path through another network, and any combination of direct connections (e.g., via Bluetooth). The data flow module (240) implements a data flow layer protocol to select one or more available paths to use for transmitting data between clients (110). In the event that there is only one available path, the data flow layer protocol directs the transmission of data through the uniquely identified path.

[0034] In a situation where multiple available paths exist, the data flow layer protocol selects one or more of the available paths to improve latency in data communication. The data flow module (240) calculates one or more metrics (i.e., performance metrics) of the available paths that represent the data transmission performance by the available paths. Metrics include latency (ping), jitter, data loss, connection strength (connection stability), bandwidth, other data transmission metrics, etc. In one embodiment, the data flow layer protocol selects the path (or multiple paths) with the lowest latency for use in data communication. In another embodiment, the data flow layer protocol considers one or more other metrics in addition to or as an alternative to latency. For example, the data flow layer protocol may combine one or more metrics for each path to determine an overall score (i.e., path score) for each path. This path score may be used to rank the available paths and select one path (e.g., the highest-ranked path) or multiple paths (e.g., multiple highest-ranked paths).

[0035] When multiple paths are selected based on the data flow layer protocol, each of the selected paths may be used to transmit separate copies of the same data to improve data transmission speed. However, in this case, there is a trade-off for using more paths in that transmitting multiple copies of the same data through different paths increases bandwidth usage. In some embodiments, the data flow layer protocol periodically samples available paths to re-evaluate which path(s) are optimal for use. For example, the data flow layer protocol evaluates available paths at regular intervals (e.g., every 30 seconds) to determine one or more optimal sets of paths for use in communication. The data flow layer protocol may sample available paths by transmitting data to an endpoint of an available path, such as from client (110A) to client (110B). Additionally, or alternatively, the data flow layer protocol may sample one or more available paths by transmitting data to an intermediary node connected to many endpoints, such as from a client (110A) to a game server (120) or a cell tower (130). In some embodiments, the data flow layer protocol selects all available paths to use for communication. The data flow layer protocol may adjust path selection based on bandwidth availability. For example, in situations where bandwidth availability is high, the data flow layer protocol may be instructed to select all available paths. In contrast, when bandwidth availability is limited, the data flow layer protocol may select only the optimal path (e.g., the lowest latency path).

[0036] In some embodiments, the dataflow module (240) encrypts data transmitted between clients during a shared AR experience, for example, in a parallel reality game. The dataflow module (240) may implement various encryption methods. In a specific embodiment, ECC is implemented to protect UDP transmissions between clients (110), for example, through a cell tower (130) or through a LAN (140).

[0037] Game server

[0038] FIG. 3 is a block diagram illustrating a game server according to one embodiment. The game server (120) may be any computing device and may include one or more processors and one or more computer-readable storage media. The computer-readable storage media may store instructions that cause the processor to perform tasks. The game server (120) may include a game database (115) or communicate with the game database (115). The game database (115) stores game data used in a parallel reality game to be served or provided to the client(s) (110).

[0039] The game data stored in the game database (115) includes (1) data related to the virtual world within the parallel reality game (e.g., image data used to render the virtual world on a display device, geographical coordinates of a location within the virtual world, etc.); (2) data related to the player of the parallel reality game (e.g., player profile including, but not limited to, player information, player experience level, player currency, current player location in the virtual world / real world, player energy level, player preference, team information, faction information, etc.); (3) data related to game goals (e.g., data related to the current game goal, state of the game goal, past game goal, future game goal, desired game goal, etc.); (4) data related to virtual elements within the virtual world (e.g., location of the virtual element, type of the virtual element, game goal related to the virtual element, corresponding real-world location information for the virtual element, operation of the virtual element, relevance of the virtual element, etc.); (5) data related to real-world objects, landmarks, and locations connected to the virtual world elements (e.g., location of the real-world object / landmark, description of the real-world object / landmark, relevance of the virtual element connected to the real-world object, etc.); (6) game state (e.g., current number of players, current state of game objectives, player leaderboard, etc.); (7) data related to player actions / inputs (e.g., current player position, past player position, player movement, player input, player questions, player communication, etc.); and (8) may include any other data used, related to, or acquired during the implementation of a parallel reality game. Game data stored in the game database (115) may be populated offline or in real time by data received from the system administrator and / or the client (110).

[0040] The game server (120) may be configured to receive requests for game data from the client (110) (e.g., via a remote procedure call (RPC)) and to respond to such requests. For example, the game server (120) may encode the game data into one or more data files and provide the data files to the client (110). Additionally, the game server (120) may be configured to receive game data (e.g., player position, player action, player input, etc.) from the client (110). For example, the client (110) may be configured to periodically send player input and other updates to the game server (120), and these updates are used by the game server (120) to update game data within the game database (115) to reflect any changed state of the game.

[0041] In the illustrated embodiment, the server (120) includes a general-purpose game module (310), a data collection module (320), and an event module (330). As previously mentioned, the game server (120) interacts with a game database (115) which may be part of the game server (120) or may be accessed remotely (for example, the game database (115) may be a distributed database accessed over a network). In other embodiments, the game server (120) includes different and / or additional elements. Additionally, these functions may be distributed among the elements in a manner different from that described. For example, the game database (115) may be integrated into the game server (120).

[0042] The universal game module (310) hosts a parallel reality game for all players and serves as an authoritative source for the current state of the parallel reality game for all players. As a host, the universal game module (310) generates game content to be provided to players, for example, through their respective clients (110). When hosting the parallel reality game, the universal game module (310) may access a game database (115) to retrieve and / or store game data. Additionally, the universal game module (310) receives game data (e.g., depth information, player input, player location, player action, landmark information, etc.) from the clients (110) and integrates the received game data into the entire parallel reality game for all players of the parallel reality game. Additionally, the universal game module (310) may manage the delivery of game data to the clients (110) over a network. Additionally, the general-purpose game module (310) can control security aspects of the client (110), including but not limited to protecting the connection between the client (110) and the game server (120), establishing connections between various clients (110), and verifying the location of various clients (110).

[0043] The game server (120) may further include a data collection module (320). In an embodiment in which one is included, the data collection module (320) may be separated from the general-purpose game module (310) or may be part of it. The data collection module (320) may manage the inclusion of various game features within a parallel reality game connected to data collection activities in the real world. For example, the data collection module (320) may modify game data stored in the game database (115) to include game features connected to data collection activities in the parallel reality game. Additionally, the data collection module (320) may analyze data collected by a player according to data collection activities and provide that data for access by various platforms.

[0044] The event module (330) manages player access to events in a parallel reality game. Although the term “event” has been used for convenience, it should be understood that this term does not need to refer to a specific event at a specific place or time. Rather, this term may refer to any provision of access-controlled game content that uses one or more access criteria to determine whether a player can access the content. Such content may be part of a larger parallel reality game containing game content with little or no access control, or it may be an independent access-controlled parallel reality game.

[0045] Cell Tower

[0046] FIG. 4 is a block diagram illustrating a cell tower according to one embodiment. In the illustrated embodiment, the cell tower (130) includes a routing module (410), a data collection module (420), an AR environment module (430), a map processing module (440), an authorization check module (450), and a local data storage (460). Additionally, the cell tower (130) includes hardware and firmware or software (not illustrated) for establishing connections to a game server (120) and a client (110) for data exchange. For example, the cell tower (130) may be connected to the game server (120) via fiber optics or other wired internet connection and connected to the client (110) using a wireless connection (e.g., 4G or 5G). In other embodiments, the cell tower (130) may include different or additional components. Additionally, these functions may be distributed among the elements in a manner different from that described. In one or more embodiments, the LAN (140) may include a module similar to that described for the cell tower (130), so the LAN may also help to host a parallel reality game.

[0047] The routing module (410) receives data packets and transmits these packets to one or more receiving devices. In one embodiment, the routing module (410) receives datagrams from a client (110) and sends the received datagrams to an appropriate destination. Additionally, the routing module (410) may receive data packets from a server that are addressed to a specific client (110) or any client connected to the cell tower (130). The routing module (410) forwards the data packets to the client (110) to which the data packets are addressed. In some embodiments, the routing module (410) receives information indicating a path selected by the transmitting client (110) to route the data packets (e.g., information within the data packets) to another client (110) either directly or through the game server (120). In this case, the routing module (410) routes the data packets using the selected path.

[0048] The data ingest module (420) receives data from one or more sources used by the cell tower (130) to provide a shared AR experience to the player through the connected client (110). In one embodiment, the data ingest module (420) receives real-time or substantially real-time information about real-world conditions (e.g., from a third-party service). For example, the data ingest module (420) may periodically (e.g., every hour) receive weather data from a weather service indicating the weather conditions of a geographical area around the cell tower. As another example, the data ingest module (420) may search for the operating hours of a park, museum, or other public place. As yet another example, the data ingest module (420) may receive traffic data indicating how many vehicles are traveling on the roads in a geographical area around the cell tower (130). This information about real-world conditions can be used to improve synergy between the virtual world and the real world.

[0049] The AR environment module (430) manages an AR environment in which players in a geographical area around the cell tower (130) can participate in a shared AR experience. In one embodiment, a client (110) is connected to the cell tower (130) while running a parallel reality game, and the AR environment module (430) connects the client to an AR environment for the game. All game players connected to the cell tower (130) can share a single AR environment, or the players can be divided into multiple AR environments. For example, a maximum number of players (e.g., 10, 20, 100, etc.) may exist in a specific AR environment. If multiple AR environments exist, a newly connected client (110) may be randomly placed within a session, or the client may be provided with a user interface (UI) that allows the player to choose which session to join. Thus, the player can choose to participate in one AR environment with friends. In some embodiments, the player may establish a private AR environment that is access-protected (e.g., requiring a password or code to participate).

[0050] In various embodiments, to make it appear as though an AR object (e.g., a creature, a vehicle, etc.) is interacting with features of the real world (e.g., to jump over obstacles rather than pass through them), the AR environment module (430) provides map data representing the real world near the client (110) (e.g., stored in a local data storage (460)) to the connected client (110). The AR environment module (430) may receive location data (e.g., GPS location) for the client (110) and provide map data for a geographical area around the client (e.g., within a threshold distance from the client's current location).

[0051] The received map data may include one or more representations of the real world. For example, the map data may include point cloud models, planar matching models, line matching models, geographic information system (GIS) models, building recognition models, landscape recognition models, etc. Additionally, the map data may include two or more representations of a given type at various levels of detail. For example, the map data may include two or more point cloud models, each containing a different number of points.

[0052] The client (110) can improve the client's location by comparing map data with data collected by one or more sensors. For example, by mapping an image captured by a camera on the client (110) to a point cloud model, the client's location and orientation can be accurately determined (e.g., within 1 centimeter and 0.1 degrees). The client (110) provides the determined location and orientation back to the AR environment module (430) along with any action taken by the player (e.g., taking a picture, selecting a virtual item to interact with, dropping a virtual item, etc.). Thus, the AR environment module (430) can update the state of the game for all players participating in the AR environment.

[0053] The map processing module (440) updates the map data based on the current state (e.g., data from the data ingest module (420)). Because the real world is not static, the map data in the local data store (460) may not represent the current real-world state. For example, the same park trail in Vermont may look very different depending on the season. In the summer, the trail may be clear and the surrounding trees may be covered in foliage. In contrast, in the winter, the trail may be blocked by snowdrifts and the trees bare. The map processing module (440) can transform the map data to approximate these changes.

[0054] In one embodiment, the map processing module (440) retrieves current state data to identify changes and applies those changes to the map data. Changes to various states may be defined by empirical rules, or may take the form of a trained machine learning model, or may use a combination of both approaches. For example, the map processing module (440) may receive current weather state data, select changes to the current weather state, and apply those changes to the map data. Alternatively, the map processing module (440) may pre-calculate the changed maps and store them (e.g., in a local data store (460)). In this case, when the client (110) connects to the cell tower, the map processing module determines the current state, selects an appropriate pre-calculated version of the map data, and provides that version to the client.

[0055] The authorization check module (450) maintains synchronization between the game states of different clients (110). In one embodiment, the authorization check module (450) checks whether a game action received from a client (110) matches the game state maintained by the AR environment module (430). For example, if two players both attempt to pick up the same in-game item, the authorization check module (450) determines which player receives the item (e.g., based on the timestamp associated with the request). As described, using a P2P protocol and local processing in the cell tower can significantly reduce the delay of a player's action being displayed on another player's client (110). Thus, the likelihood (and frequency) of such conflicts occurring and being resolved by the authorization check module (450) is reduced. Thus, the AR experience can be improved.

[0056] Additionally, the authorization check module (450) may maintain synchronization between a copy of its AR environment state (intermediate node state) and a master state maintained by the game server (120). In one embodiment, the authorization check module (450) receives global updates regarding the state of the AR environment from the game server (120) periodically (e.g., every 1 to 10 seconds). The authorization check module (450) compares these updates with the intermediate node state and resolves any discrepancies. For example, if a player's item pickup request is initially approved by the authorization check module (450), but a game update from the game server (120) indicates that another player has picked up the item (or otherwise made it unavailable) before the player attempts to pick up the item, the authorization check module (450) may send an update to the player's client (110) indicating that the item should be removed from the player's inventory.

[0057] This process can provide a value for a client (110) located near the boundary between coverages provided by two or more different cell towers (130). In this case, players with client devices (110) connected to different cell towers (130) can all interact with the same virtual element. Thus, each individual cell tower (130) can initially approve interaction with the conflicting element from different clients (110). In this case, the server (120) can detect these conflicts after synchronizing with these different cell towers (130) and send an update to resolve the conflict (e.g., instructing one of the cell towers to cancel their initial action approval and update its local state accordingly).

[0058] A local data storage (460) is one or more non-transient computer-readable media configured to store data used by a cell tower. In one embodiment, the stored data may include map data, current state data, a list of currently (or recently) connected clients (110), a local copy of the game state for a geographical area, etc. Although the local data storage (460) is depicted as a single entity, the data may be distributed across multiple storage media. Additionally, some data may be stored elsewhere within a communication network and accessed remotely. For example, the cell tower (130) may access current state data remotely (e.g., from a third-party server) as needed.

[0059] Exemplary method

[0060] FIG. 5 illustrates a process (500) for using a data flow layer protocol according to one embodiment. The steps of FIG. 5 are described from the perspective of a client (110A) performing the method (500). However, some or all steps may be performed by other entities or components. Additionally, some embodiments may perform the steps simultaneously, or perform the steps in a different order, or perform different steps.

[0061] The client (110A) identifies one or more available network paths for data transmission (510). The client (110A) identifies these network paths by determining which of a set of predetermined paths for the target client (110B) is currently available to the client (110A). If the client (110A) can establish a connection with the game server (120) and the target client (110B) is also connected to the game server (120), the client (110A) identifies a network path to another client (110) through the game server (120). If the client (110A) is connected to the cell tower (130), the client (110) can determine whether the other client (110B) is likewise connected to the cell tower (130). If so, the client (110A) identifies another network path through the cell tower (130). Additionally, if client (110A) and another client (110B) are connected to the same LAN (140), client (110A) can identify a network path through the LAN (140). Client (110A) can identify another network or path connecting both clients (110) as another available network path (e.g., a direct connection between clients (110), such as a Bluetooth connection). One or more network paths may implement UDP for data transmission. In the case of a UDP path, client (110) may use public key cryptography for encryption of data. In particular, client (110) may implement ECC.

[0062] A client (110A) detects one or more performance metrics for each available network path (520). The client (110A) detects performance metrics by transmitting test data packets through the available network paths. The data packets may request another test data packet to be transmitted back from another client (110B). Based on the successful transmission of these test packets, the client (110A) may calculate (or detect) the metrics of the network paths. The metrics include latency (ping), jitter, data loss, connection strength, bandwidth availability, etc.

[0063] The client (110A) selects one or more available network paths to use for data transmission based on the detected metric (530). The client (110A) may consider one or more specific metrics when selecting a network path. In one embodiment, the client (110A) considers only latency (ping). In another embodiment, the client (110A) considers latency and connection strength. As previously described, the client (110A) may select the "optimal" path or paths (indicated by one or more metrics), or may select all available network paths to use for data transmission.

[0064] The client (110A) transmits data through one or more selected network paths (540). In the environment of a parallel reality game, the client (110A) transmits game data to other clients (110) and a game server (120). Client-to-client communication is particularly efficient when having a shared AR experience in a parallel reality game, where multiple players on their clients (110) can interact with each other and with the game server (120). Since some of these interactions can be hosted locally by the client (110) or another network (e.g., a cell tower (130) or a LAN (140)), the game state does not need to be synchronized by the game server (120). Hosting the AR experience locally provides a low-latency and more streamlined AR experience.

[0065] The client (110A) can repeat the process (500) to periodically sample available paths and repeatedly optimize which path to use to maintain data transmission between devices.

[0066] FIG. 6 illustrates a method (600) for transmitting data between clients (110) within a geographical area around a cell tower according to one embodiment. The steps of FIG. 6 are described in terms of a cell tower (130) performing the method (600) of providing P2P communication between client devices (110) within the geographical area. However, some or all steps may be performed by other entities or components. Additionally, some embodiments may perform the steps simultaneously, or perform the steps in a different order, or perform different steps.

[0067] In the embodiment illustrated in FIG. 6, the cell tower (130) provides P2P communication via a network path for a client (110) within a geographical area surrounding the cell tower (610). For example, the cell tower (130) may provide P2P communication to a client (110) that provides a parallel reality game (e.g., hosted by a game server (120)). The cell tower (130) receives data from a first client (110A) via a network path (620). In particular, the network path is selected by the first client (110A) from the available network paths based on one or more metrics for each available network path. For example, the first client device may select a network path that provides P2P communication using a data flow layer protocol as previously described. The data received from the first client (110A) may be related to virtual content of an AR environment, such as AR data corresponding to a geographical location within the geographical area.

[0068] Using a network path, the cell tower (130) transmits data to a second client (110B) within a geographic area (630). For example, if the data describes an update to virtual content in an AR environment related to a parallel reality game, the cell tower (130) may provide this update to some or all of the clients (110) of the parallel reality game connected to the cell tower (130). As another example, data such as interactions or messages with virtual content related to a player using the second client may be specifically addressed to the second client (110B).

[0069] Exemplary computer

[0070] FIG. 7 is a high-level block diagram illustrating an exemplary computer (700) suitable for use within the computer network illustrated in FIG. 1, according to one embodiment. The exemplary computer (700) includes at least one processor (702) connected to a chipset (704). The chipset (704) includes a memory controller hub (720) and an input / output (I / O) controller hub (722). Memory (706) and a graphics adapter (712) are connected to the memory controller hub (720), and a display (718) is connected to the graphics adapter (712). A storage device (708), a keyboard (710), a pointing device (714), and a network adapter (716) are connected to the I / O controller hub (722). Other embodiments of the computer (700) have different architectures.

[0071] In the embodiment illustrated in FIG. 7, the storage device (708) is a non-transient computer-readable storage medium such as a hard drive, a compact disc read-only memory (CD-ROM), a DVD, or a solid-state memory device. The memory (706) holds instructions and data used by the processor (702). The pointing device (714) is a mouse, trackball, touchscreen, or other type of pointing device and is used in conjunction with the keyboard (710) (which may be an on-screen keyboard) to input data into the computer system (700). The graphics adapter (712) displays images and other information on the display (718). The network adapter (716) connects the computer system (700) to one or more computer networks.

[0072] The type of computer used by the entity of FIG. 1 may vary depending on the processing power and embodiment required by the entity. For example, the game server (120) may include a distributed database system comprising a plurality of blade servers working together to provide the described function. Additionally, the computer may not include some of the previously described components, such as a keyboard (710), a graphics adapter (712), and a display (718).

[0073] Those skilled in the art can make various uses, modifications, and variations of the devices and technologies disclosed herein without departing from the described concepts. For example, the components or features illustrated or described herein are not limited to the locations, settings, or environments illustrated or described. Examples of devices according to the invention may include all, or fewer or different, components described with reference to one or more of the preceding drawings. Accordingly, the invention is not limited to the specific embodiments described herein, but rather should be given the broadest possible scope consistent with the appended claims and their equivalents.

[0074] Additional considerations

[0075] The description of the foregoing embodiments is provided for illustrative purposes only and is not intended to limit the patent rights to the exact form disclosed or to describe everything thoroughly without omission. Those skilled in the art will understand that many modifications and variations are possible in light of the foregoing disclosure.

[0076] Parts of this description explain embodiments in terms of algorithms and symbolic representations of operations on information. Such algorithmic descriptions and representations are widely used by those skilled in the art of data processing technology to effectively communicate the content of their operations to others skilled in the art. These operations are described functionally, computationally, or logically, but are understood to be implemented by computer programs, equivalent electrical circuits, microcode, etc. Furthermore, it has been proven that referring to these modes of operation as modules is sometimes convenient without losing generality. The described operations and their associated modules may be implemented in software, firmware, hardware, or any combination thereof.

[0077] Any step, operation, or process described herein may be performed or implemented through one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, a software module is implemented as a computer program product comprising a computer-readable medium containing computer program code, which may be executed by a computer processor to perform some or all of the described steps, operations, or processes.

[0078] Additionally, embodiments may relate to devices for performing the operations of this specification. Such devices may include general-purpose computing devices that are specifically configured for the required purpose and / or are selectively activated or reconfigured by a computer program stored in a computer. Such computer programs may be stored on a non-transient tangible computer-readable storage medium that can be connected to a computer system bus, or on any type of medium suitable for storing electronic instructions. Furthermore, any computing system mentioned in this specification may include a single processor, or may be an architecture using a multi-processor design for enhanced computing power.

[0079] Additionally, the embodiments may relate to a product created by the computing process described herein. Such a product may include information generated from the computing process, wherein the information is stored on a non-transient type computer-readable storage medium and may include any embodiment of a computer program product or other combination of data described herein.

[0080] As used herein, any reference to “one embodiment” or “one embodiment” means that a specific element, feature, structure, or feature described in relation to that embodiment is included in at least one embodiment. The appearance of the phrase “in one embodiment” in various places in this specification does not necessarily refer to the same embodiment. Similarly, the use of “one” or “one” before an element or component is for convenience only. Unless it is evident that such description implies otherwise, it should be understood to mean that one or more of the element or component are present.

[0081] Where a value is described as "approximately" or "substantially" (or a derivative thereof), such value should be interpreted as + / - 10% of the exact value unless otherwise evident from the context. For example, "about 10" should be understood to mean "within the range of 9 to 11."

[0082] As used herein, the terms “include,” “include,” “equip,” “equipped,” “have,” “have,” or any other variation thereof are intended to mean non-exclusive inclusion. For example, a process, method, article, or device comprising a list of elements is not necessarily limited to such elements only and may include other elements not explicitly listed or not unique to such process, method, article, or device. Also, unless otherwise explicitly stated, “or” means “inclusive or” and not “exclusive or”. For example, condition A or B is satisfied by any of the following: A is true (or exists) and B is false (or does not exist); A is false (or does not exist) and B is true (or exists); and both A and B are true (or exist).

[0083] Those skilled in the art who read this disclosure will understand that there are other alternative structural and functional designs that can be used to employ the described techniques and approaches. Accordingly, while specific embodiments and applications have been illustrated and described, it should be understood that the described invention is not limited to the exact configurations and components disclosed. The scope of protection should be limited only by the claims below.

[0084] Finally, the language used in this specification has been chosen primarily for readability and illustrative purposes and is not intended to describe or limit the patent rights. Accordingly, the scope of the patent rights is intended to be limited by any claims made at the time of filing based thereon, rather than by this detailed description. Accordingly, the disclosure of the embodiments is for illustrative purposes only and is not intended to limit the scope of the patent rights as described in the claims below.

Claims

Claim 1 A non-transient computer-readable storage medium for storing instructions that cause a computing device to perform an operation when executed by a computing device, wherein the operation comprises: identifying an available network path for data transmission passing through a plurality of intermediary nodes including a cell tower and a server corresponding to an interactive application; detecting one or more performance metrics for each of the available network paths passing through the plurality of intermediary nodes; selecting a first available network path from the available network paths for use in data transmission according to one or more detected performance metrics; transmitting first data describing a first interaction with the interactive application through the first available network path; and receiving second data describing a second interaction with the interactive application through a second available network path. Claim 2 A non-transient computer-readable storage medium according to claim 1, wherein the step of selecting the available network path comprises: determining a score for each of the first available network paths using one or more detected performance metrics; ranking the plurality of available paths according to the determined score; and selecting the first available network paths using the ranking. Claim 3 A non-transient computer-readable storage medium according to claim 1, wherein one or more performance metrics include latency, and the step of selecting the first available network path further comprises: determining that the latency for the first network path is smaller than the latency of one or more other available network paths; and selecting the first available network path based on the determination. Claim 4 A non-transient computer-readable storage medium according to claim 1, wherein the one or more performance metrics include bandwidth, and the operation further comprises: selecting each of the one or more available network paths in response to a determination that the bandwidth of the set of the one or more available network paths is greater than a bandwidth threshold; and transmitting a first data through each of the available network paths. Claim 5 A non-transient computer-readable storage medium according to claim 1, wherein the operation further comprises: the step of re-detecting one or more performance metrics for each of the available network paths; the step of re-selecting a third available network path for use in data transmission from the available network paths according to one or more re-detected performance metrics; and the step of transmitting the third data through the re-selected available network path. Claim 6 A non-transient computer-readable storage medium according to claim 1, characterized in that the available network path includes a path through a local area network (LAN). Claim 7 A non-transient computer-readable storage medium according to claim 1, wherein the one or more network paths are for transmitting data between a first client and a second client, and the step of transmitting the first data includes the step of transmitting the first data to the second client by the first client through the first available network path. Claim 8 A non-transient computer-readable storage medium according to claim 7, wherein the operation further comprises: requesting, by the first client, from a network coordination service, one or more network addresses associated with a shared network environment, wherein the shared network environment includes network addresses for client devices within a geographic area; receiving, in response to the request, a network address of the second client from the network coordination service; and transmitting a first data through the first available network path using the network address of the second client. Claim 9 A non-transient computer-readable storage medium characterized in that, in claim 7, the first client and the second client are associated with public network addresses, and the step of transmitting the first data includes the step of protecting the first data with a password through public key encryption. Claim 10 In claim 8, a non-transient computer-readable storage medium characterized in that the interactive application corresponds to geographic data related to a geographic location within the geographic area. Claim 11 A non-transient computer-readable storage medium characterized in that, in claim 10, the cell tower maintains the local state of the geographic location data, the first interaction is oriented toward an action on the geographic location data, and the operation further comprises the step of receiving information describing the local state of the geographic location data from the cell tower after transmitting first data describing the action, wherein the information indicates whether the local state has been successfully updated by the cell tower to reflect the action. Claim 12 A non-transient computer-readable storage medium according to claim 1, wherein the step of transmitting the first data further comprises: selecting an additional available network path from the plurality of available network paths according to the one or more detected performance metrics; and transmitting an additional copy of the first data through the additional available network path. Claim 13 A computer implementation method comprising: a step of providing peer-to-peer communication to a client within a geographical area around the cell tower by means of a cell tower through one of a plurality of available network paths passing through a plurality of intermediary nodes including a cell tower and a server corresponding to an interactive application; a step of receiving data from a first client that describes an interaction with an interactive application by the first client for transmission through the selected network path, based on one or more performance metrics for each of the plurality of available network paths within the geographical area; and a step of transmitting data to a second client within the geographical area by means of the selected network path by means of the cell tower. Claim 14 A computer-implemented method according to claim 13, further comprising: a step of maintaining a local state of geographic location data for an interactive application in the cell tower, wherein the geographic location data is associated with a geographic location within the geographic area surrounding the cell tower, and the interaction with the interactive application is oriented toward an action on the geographic location data; a step of comparing the action with the local state of the geographic location data; a step of updating the local state to reflect the action in response to a determination that the action does not conflict with the local state based on the comparison; and a step of transmitting information describing the updated local state of the geographic location data to the second client according to a selected network path. Claim 15 A computer-implemented method according to claim 14, further comprising: receiving data from the first client describing an additional action related to the geographic location data; comparing the additional action with the local state of the geographic location data; rejecting the additional action in response to a determination that the additional action conflicts with the local state of the geographic location data based on the comparison; and transmitting information describing the conflicting local state of the geographic location data to the first client. Claim 16 A computer-implemented method according to claim 14, further comprising: receiving, by the cell tower, information describing the master state of geographic location data maintained by the server from a server associated with an interactive application; and synchronizing the local state of the geographic location data with the master state. Claim 17 A computer-implemented method according to claim 16, wherein the master state reflects an additional action by a third client device within an overlapping area between the geographic area and an additional geographic area around an additional cell tower, and the synchronizing step further comprises: determining by the cell tower whether the additional action conflicts with the action; updating the local state to reflect the additional action instead of the action in response to the determination that the additional action conflicts with the action; and transmitting information describing the updated local state of the geographic location data to the first client. Claim 18 A computer-implemented method according to claim 13, wherein the first client selects the network path by: determining a score for each of the available network paths using one or more detected performance metrics; ranking the plurality of available paths according to the determined score; and selecting the available network paths using the ranking. Claim 19 A computer-implemented method according to claim 13, wherein the one or more performance metrics include one or more of latency, jitter, data loss, connection stability, or bandwidth. Claim 20 A system comprising: a server; a cell tower; and a first client device connected to a second client device through a plurality of network paths, wherein the plurality of network paths include a first network path through the server, a second network path through the cell tower, and a third network path through a local area network, and wherein the first client device is configured to: detect one or more performance metrics for each of the plurality of network paths; select a first network path from the plurality of network paths for use in data transmission according to the one or more detected performance metrics; transmit first data describing a first interaction with an interactive application to the second client device through the first available network path, and receive second data describing a second interaction with the interactive application through the second available network path.

Images