Client authoritative status distribution system

Abstract

A process includes establishing a peer-to-peer connection between a first client computing device and a second client computing device. The first client computing device is associated with a first participant in a coordinate grid environment provided by an application and the second client computing device is associated with a second participant in the coordinate grid environment. The process includes: determining that a coordinate grid state of the first participant has undergone a state change; and in response, providing at least a portion of first participant state information associated with the state change from the first client computing device to the second client computing device via the peer-to-peer connection, such that at least a portion of the first participant state information updates previously stored first participant state information of the first participant maintained by a second participant associated with the second client computing device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 288,435, filed on December 10, 2021, entitled “MESH NETWORK SIGNAL CACHING SYSTEM,” U.S. Provisional Patent Application No. 63 / 391,652, filed on July 22, 2022, entitled “HYBRID PEER-TO-PEER MESH AND STREAMING SYSTEM,” and U.S. Provisional Patent Application No. 63 / 398,485, filed on August 16, 2022, entitled “AUTHORITATIVE STATE DISTRIBUTION.” The entire contents of each of the foregoing patent applications are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to peer-to-peer mesh networks, and more particularly, to a system for providing authoritative state of clients distributed over peer connections in a mesh network. Background Art

[0004] Many software applications can stream media such as video, audio or other data between participants of the application. Video and audio streaming can enhance the user experience of the application by allowing more personal interactions between participants of the application instance. For example, in some game applications, video and audio can stream between participants of the game instance. However, as the number of streams between a specific participant and other participants increases, the quality of audio and video streams decreases and may cause streaming sessions to fail.

[0005] As a result, systems have been developed to limit the number of streams per participant to maintain acceptable stream quality. One such system includes a peer-to-peer mesh topology in which a subset of participants are interconnected for streaming video and / or audio. However, the peer-to-peer mesh topology presents problems discussed below. Summary of the invention

[0006] The following is a non-exhaustive list of some aspects of the present technology. These and other aspects are described in the following disclosure.

[0007] Some aspects include a process comprising: establishing, with a first client computing device, a first peer-to-peer connection between the first client computing device and a second client computing device, wherein the first client computing device is associated with a first participant in a coordinate grid environment provided by an application, and the second client computing device is associated with a second participant in the coordinate grid environment; determining, with the first client computing device, that a coordinate grid state of the first participant has undergone a state change; and providing, with the first client computing device and in response to the state change to the coordinate grid state satisfying a state change condition, at least a portion of first participant state information associated with the state change of the first participant from the first client computing device to the second client computing device via the first peer-to-peer connection, wherein the at least a portion of the first participant state information associated with the state change of the first participant updates previously stored first participant state information of the first participant maintained by a second participant associated with the second client computing device.

[0008] Some aspects include a process comprising: establishing, with a first client computing device, a first peer connection between the first client computing device and a second client computing device, wherein the first client computing device is associated with a first participant in a coordinate grid environment provided by an application and the second client computing device is associated with a second participant in the coordinate grid environment; receiving, with the first client computing device, second participant state information from the second client computing device via the first peer connection, wherein the second participant state information is associated with a state change of the second participant; and updating, with the first client computing device, previously stored second participant state information using the second participant state information.

[0009] Some aspects include a tangible, non-transitory machine-readable medium storing instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform operations including the above-mentioned procedures.

[0010] Some aspects include a system comprising: one or more processors; and a memory storing instructions, wherein the instructions, when executed by the processors, cause the processors to implement the operations of the above-mentioned processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above-mentioned aspects and other aspects of the present technology will be better understood when the present application is read in light of the following drawings, in which like numbers represent similar or identical elements:

[0012] Figure 1 shows a computing environment in which the present technology may be implemented according to some embodiments of the present disclosure;

[0013] Figure 2is a diagram showing some embodiments of the present disclosure Figure 1 A block diagram of an example of a client computing device of a computing environment;

[0014] Figure 3 is a diagram showing some embodiments of the present disclosure Figure 1 A block diagram of an example of a server computing device of a computing environment;

[0015] Figure 4 According to some embodiments of the present disclosure, Figure 1 A flowchart of a process for participant state distribution performed by a computing environment;

[0016] Figure 5 According to some embodiments of the present disclosure, Figure 1 A flowchart of a process for peer signal caching performed by a computing environment;

[0017] Figure 6 Some embodiments of the present disclosure are shown Figure 4 and Figure 5 A block diagram of an example of a process;

[0018] Figure 7 According to some embodiments of the present disclosure, Figure 1 A flowchart of a process performed by a computing environment of providing participant state information via a peer-to-peer connection;

[0019] Figure 8 Some embodiments of the present disclosure are shown Figure 7 A block diagram of an example of a process;

[0020] Fig. 9 According to some embodiments of the present disclosure, Figure 1 A flowchart of a process for hybrid peer-to-peer and selective forwarding unit media streaming performed by a computing environment of;

[0021] Fig.10 Some embodiments of the present disclosure are shown Fig. 9 A block diagram of an example of a process;

[0022] Fig.11 According to some embodiments of the present disclosure, Figure 1 A flowchart of a process for throttling a peer connection bit rate by a computing environment;

[0023] Fig. 12A , Fig. 12B and Fig. 12C Some embodiments of the present disclosure are shown Fig.11 A block diagram of an example flow of a process;

[0024] Fig.13An example computing device is shown by which the present technology may be implemented in accordance with some embodiments of the present disclosure.

[0025] Although the present technology is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will be described in detail herein. The drawings may not be drawn to scale. However, it should be understood that the drawings and their detailed description are not intended to limit the present technology to the specific forms disclosed, but on the contrary, the present invention is intended to cover all modifications, equivalents and alternatives that fall within the spirit and scope of the present technology as defined by the appended claims. DETAILED DESCRIPTION

[0026] In order to alleviate the problems described herein, the inventors must invent solutions and, in some cases, equally important, recognize problems that have been overlooked (or not yet foreseen) by others in the fields of media streaming, virtual coordinate grid environments, and peer-to-peer mesh networks. In fact, the inventors wish to emphasize the difficulty of recognizing these emerging problems, and if industry trends continue as the inventors expect, these problems will become more apparent in the future. Further, because multiple problems are solved, it should be understood that some embodiments are for specific problems, and not all embodiments solve every problem of the conventional systems described herein or provide every benefit described herein. That is, improvements that solve various arrangements of these problems are described below.

[0027] As discussed above, it is increasingly common to stream media (e.g., video, audio, and / or other data) between participants in application environments such as gaming environments, communications environments, social media environments, virtual reality environments, applications incorporating coordinate grids, or any other application environment apparent to one skilled in the art having the present disclosure. As clusters of application instances grow, it becomes increasingly difficult to reliably maintain connections between subsets of participants. It is often rarely necessary to maintain streams between all participants, but in many cases, proximity connections are very valuable.

[0028] Some common streaming architectures include peer-to-peer (P2P) (also called mesh or P2P mesh) architecture, selective forwarding unit (SFU) architecture, multipoint conferencing unit (MCU) architecture, and experience delivery network (XDN) architecture.

[0029] The P2P mesh topology is probably the easiest to build and is usually the most cost-effective architecture that can be used in real-time communication applications, such as those including WebRTC (Web Real-Time Communication). However, it is also the least scalable. In a P2P mesh topology, two or more peers (clients) talk to each other directly, or when on opposite sides of a firewall, via a relay server that relays audio, video, and other data streams to them.

[0030] P2P applications can be resource intensive because the burden of encoding and decoding media streams is offloaded to each peer, which is why P2P performs best when there are only a small number of concurrent users (e.g., 10 or less). Although some degree of scalability can be achieved on P2P applications by configuring a P2P mesh network, P2P applications are resource intensive and inefficient. However, P2P applications provide the best end-to-end encryption because it does not rely on a centralized server to encode / decode the stream.

[0031] A peer-to-peer mesh topology can adequately connect a subset of a cluster of application instances, but quickly and accurately negotiating a connection between two individual peers remains a challenge.

[0032] For these reasons, real-time communication application developers usually rely on other architectures, such as the SFU architecture including a direct routing system designed to offload some of the stream processing from each client to the server. Each participant sends their encrypted media stream once to a central server, which then forwards these streams to other participants in the application without further processing. Other real-time communication applications use an MCU architecture in which each client is connected to a centralized MCU server, which decodes, rescales and mixes all incoming streams into a single new stream, and then encodes the signal stream and sends it to all clients. Other other real-time communication applications use an XDN architecture, which uses a cloud-based cluster architecture instead of a centralized server to solve the scalability problem of WebRTC.

[0033] Most video games or other coordinate grid-based applications that involve real-time communication utilize a central state server to resolve the accuracy of the resolution or authenticity of events that each participant can agree on, and report these updates to the client. These updates can be for participant movement and other events (e.g., movement of objects in the environment, attributes of the participant (e.g., health, strength, speed, or other attributes of the participant)). The server usually confirms the behavior before reporting it to the client so that the client can trust the updates given. This is important for creating an agreement between the participants that the results occurred. However, confirming the behavior before reporting it to the client requires additional processing on the server to resolve the results of the interaction between the participants. As a result, the MCU architecture can provide both real-time communication and meet the authenticity requirements. There are limitations on the scalability of systems like this, making the trade-off between the accuracy of the participant's position and the efficiency of the central state server important as the virtual world population grows. As the participant count increases, reaching a consensus on facts becomes a limiting factor in the size of the world. More and more processing power and network bandwidth are required to distribute these changes and updates to all other participants. So many confirmed updates controlled by a single server or process limit the number of participants in the world (such as a specific coordinate grid instance). However, some state- or coordinate-grid-based applications do not require real-time consensus between participants. Therefore, a centrally controlled state server is not needed. In this case, it is ideal to use a P2P mesh topology to remove the additional hardware of the SFU. Distributing this work to clients is critical to building services that can accommodate hundreds, thousands, or tens of thousands of participants simultaneously.

[0034] Peer signal caching and server authority status management

[0035] Embodiments of the present disclosure provide an application platform in which each participant can determine its own state or the coordinate grid state of the participant's position relative to other participants in the coordinate grid-based environment currently provided by the application. The only important fact is the position of each participant relative to other participants. If there is a slight difference between the relative positioning of two participants (e.g., actual physical location or location in a virtual application environment), it has no effect on the application platform. This will eventually be resolved when the participant stops moving.

[0036] In distributing state management to participants, the application platform implements a highly scalable server. The only work that the server may do is to receive user input events, such as click events or other user input from participants. The user input captures the current position of the participant in the virtual environment (e.g., coordinates in a coordinate grid (e.g., a three-dimensional grid or a two-dimensional grid)), the user input position (e.g., the coordinates in the coordinate grid where the user instructs the participant (e.g., avatar) to move in the virtual environment), and the time of the user input. The central server can obtain these three data points and overwrite any existing position description or state of the moving participant. This is the current state (e.g., position description) of the participant and will remain so until new movement instructions are received and distributed. This payload is distributed to all participants of the application platform and the server's work is completed. The network load to achieve this is smaller than in the case where the server determines all the states of the participants itself. Then, each client computing device associated with the participant in the coordinate grid of the application environment is responsible for updating the position of the (n) other participants at any frame rate that each client can run.

[0037] The distribution of state computation relieves a central server from having to perform O(N) updates per frame to all positions of all participants in the world and then transmit all deltas to all participants every frame. This greatly reduces processing overhead and network bandwidth on the entire system and allows thousands of participants to participate in a single world.

[0038] However, distributing state introduces other problems with peer-to-peer architectures. Connecting two or more participants in a single lobby or peer connection requires relaying an "offer" signal from an "initiator" to a "receiver" who will respond with a "response" signal. If these participants correctly exchange P2P connection initiation signals, a peer connection (such as a WebRTC connection) between the two can be opened.

[0039] However, there are challenges in transmitting these exchanged P2P connection initiation signals back and forth between two participants. For example, the first challenge is the need for a relay server. This is usually done using a WebSocket with a persistent connection from each participant to a server node. Each initiator participant can send their P2P connection initiation signal to the relay server, and the relay server relays the signal to the corresponding receiver participant. The second challenge is the ability of the "receiver" participant to respond with a "response" signal after receiving the "proposal" signal from the "initiator". In some scenarios, the "receiver" may be unavailable due to being unable or unwilling to connect. There are several potential reasons why the "receiver" is in this unavailable state. One reason may be that the "receiver" has run out of available connection slots and is not processing any incoming "proposals".

[0040] Another reason may be that the "receiver" is not yet aware of the "initiator's" connection intention (e.g., the state between the participants has not yet been agreed upon). As discussed above, distributed state management introduces new problems in peer-to-peer networks, where participants can determine that they will be connected to each other asynchronously. In applications without a centrally controlled state server, since each participant controls its own state, each participant can conclude that they are close enough to connect to another participant via webRTC or other peer connection protocols at different times. For example, participant A (i.e., the "initiator") can determine that they are within the connection radius of participant B (i.e., the "receiver"), and participant B may confirm this fact after a period of time. Therefore, there is a possibility of missing peer connection initiation signals (such as WebRTC "proposals" to the "receiver") and peer connection response signals (such as WebRTC "responses" to the "initiator"), which makes the P2P architecture less reliable.

[0041] Thus, additional embodiments of the present disclosure address these issues by including a local signal cache for each participant so that the client computing device associated with that participant stores any peer connection initiation signals. The "recipient" should do one of two things with the incoming "proposal" signal. If the "recipient" is ready to process the "proposal" because it knows of a valid incoming connection attempt from the "initiator", then it should process the "proposal" and respond with a "response" (e.g., the two participants agree on the status). If this "recipient" cannot process the incoming "proposals" when received from the "initiator", then the "recipient" should store these incoming "proposals" in the local signal cache. The local signal cache assigned to the "initiator" that sent the "proposal" saves the "proposal" for processing later.

[0042] If the "receiver" knows of valid conditions to connect to the "initiator", and the local signal buffer is filled with valid proposals from this "initiator", the "receiver" can process the "proposal" and send a "response" immediately without triggering the generation of a new "proposal". At this point, the "initiator" can still wait for the "response" of the "receiver" to process. If the "initiator" has moved on and determines that the "receiver" is no longer "close" to the "initiator", there is nothing to do because the "initiator" will no longer accept "responses".

[0043] Figure 1An example of a computing environment 10 in which the present technology may be implemented is shown. In some embodiments, the computing environment 10 is a distributed computing environment that implements a client / server architecture, although other architectures are also contemplated, including a monolithic architecture running on a single computing device. In some embodiments, the computing environment 10 includes a server 12, a client computing device 14, a relay server 16, a selective forwarding unit 18, and a network 20 (such as the Internet) through which these components can communicate.

[0044] In some embodiments, the client computing device 14 is a desktop computer, laptop computer, in-store kiosk, tablet computer, mobile phone, head mounted display, game console, set-top box, or any other computing device running an operating system and a web browser or native application in which the described user interface is presented as will be apparent to one skilled in the art having the present disclosure. Three client computing devices 14 are shown, but embodiments may support many more concurrent sessions, e.g., more than 100 or more than 1000 sessions geographically distributed in the United States or around the world.

[0045] In some embodiments, server 12 is a non-blocking web server or application interface server configured to service multiple concurrent sessions (e.g., implementing a model-view-controller architecture or other design) with different client computing devices 14. In some embodiments, server 12 may dynamically generate assets, markup language instructions, and scripting language instructions in response to requests from client computing devices 14 to send a user interface to those client computing devices 14 or to update a user interface on those client computing devices 14. The user interface may evolve over time (e.g., in a web application), and in some cases, new resources (e.g., images and other data) sent from server 12 are displayed in response to user input to the user interface.

[0046] The configuration engine and rendering engine included on the server can be used to generate image files and metadata used by the server 12 to generate a user interface, such as providing a virtual two-dimensional or virtual three-dimensional coordinate grid environment, an augmented reality environment, a virtual reality environment, a metaverse environment, or any other coordinate grid environment or application environment that is apparent to those skilled in the art who have mastered and benefited from the teachings of the present disclosure.

[0047] In some embodiments, relay server 16 is a server configured to create a websocket or other persistent connection with each client 14. Each client 14 can then send their initiation and response signals via relay server 16 to establish a real-time communication session (e.g., a WebRTC connection). Selective forwarding unit 18 can be a server configured to receive media from participants and decide which media streams should be forwarded to other participants and then perform the forwarding. Although server 12, relay server 16, and selective forwarding unit 18 are shown as separate servers, various embodiments of the present disclosure may include a single server that provides application, relay, or media forwarding functionality of server 12, relay server 16, or selective forwarding unit 18.

[0048] Figure 2 An embodiment of a client computing device 200 is shown, which may be the client computing device described above with reference to FIG. Figure 1 Any of the client computing devices 14 discussed above. In the illustrated embodiment, the client computing device 200 includes a chassis 202 that houses the components of the client computing device 200. Several of these components are Figure 2 For example, chassis 202 may house a processing system (not shown) and a non-transitory memory system (not shown) that includes instructions that, when executed by the processing system, cause the processing system to provide a peer-to-peer (P2P) application controller 204 that is configured to perform the functions of a peer-to-peer application controller or a client computing device discussed below. Figure 2 In the specific example shown in , the peer application controller 204 is configured to provide a local application or browser application (e.g., a peer application), such as an Internet web browser, to communicate with an application instance operating on the server 12 or the relay server 16. The peer application controller 204 can be configured to call a peer application programming interface (API) (e.g., a WebRTC API) or other peer program to facilitate peer communication. The peer application controller 204 can also include a signal caching engine 204a, which is configured to cache peer signals (e.g., initiator "proposals"), determine whether peer connection conditions exist based on coordinate grid status or other information, or other functions discussed in further detail below. The peer application controller 204 can also include a coordinate grid state engine 204b, which is configured to capture participant or object position change input or other participant state changes in the environment, send position changes to a state server (e.g., server 12), receive participant state payloads for all participants in the coordinate grid environment, and determine the coordinate grid state of all participants in the environment based on the participant state payload.

[0049] The chassis 202 may also house a communication system 210 coupled to the peer application controller 204 (e.g., via a coupling between the communication system 210 and a processing system). The communication system 210 may include software or instructions stored on a computer-readable medium and that allow the client computing device 200 to send and receive information over the communication network discussed above. For example, the communication system 210 may include a communication interface 212 to provide communications over the network 20 as detailed above (e.g., (multiple) first (e.g., long-range) transceivers). In an embodiment, the communication interface 212 may include a wireless antenna configured to provide communications utilizing the IEEE 802.11 protocol (Wi-Fi), cellular communications, satellite communications, other microwave radio communications, and / or communications. The communication system 210 may also include a communication interface 214 (e.g., (multiple) second (e.g., short-range) transceivers) configured to provide communications with the client computing device 200. Figure 1 Other client computing devices 14, sensors, storage devices or the above references Figure 1 For example, the communication interface 214 may include a wireless antenna configured to communicate with other devices discussed above in accordance with a wireless communication protocol such as Bluetooth, Bluetooth Low Energy ( Low Energy (BLE), near field communication (NFC), infrared data association (IrDA), The wireless communication protocol may operate in accordance with the IEEE 802.11 protocol (Wi-Fi) and / or other wireless communication protocols that allow direct communication between devices.

[0050] The chassis 202 may house a storage device (not shown) that provides a storage system 216 that is coupled to the peer application controller 204 through the processing system. The storage system 216 may be configured to store a local signal cache 216a (e.g., a cache of peer initiated signals), a coordinate grid state 216b that stores one or more coordinate grid states indicating the current location of a participant in a coordinate grid environment provided by the application. The coordinate grid state may include a calculated coordinate grid state determined based on participant state information of the participant collected by the server 12. The participant state information may include raw participant state information collected by the client computing device 14 / 200 or collected by the server 12. Although specific instructions and data are described as being stored in the storage system 216, those skilled in the art having knowledge of the present disclosure will recognize that other instructions may be used to perform the functions described herein, as described below.

[0051] In various embodiments, the chassis 202 also houses a user input / output (I / O) system 218 coupled to the peer application controller 204 (e.g., via a coupling between the processing system and the user I / O system 218). In embodiments, the user I / O system 218 may be provided by a keyboard input subsystem, a mouse input subsystem, a track pad input subsystem, a touch input display subsystem, a microphone, an audio system, a haptic feedback system, and / or any other input subsystem. The chassis 202 also houses a display system 220, which is coupled to the peer application controller 204 (e.g., via a coupling between the processing system and the display system 220) and may be included in the user I / O system 218. In embodiments, the display system 220 may be provided by a display device that is integrated into the client computing device 200 and includes a display screen (e.g., a display screen on a laptop / notebook computing device, a tablet computing device, a mobile phone, or a wearable device), or provided by a display device that is directly coupled to the client computing device 200 (e.g., a display device that is coupled to a desktop computing device via a wired or wireless connection). Although in Figure 2 A particular client computing device 200 is shown in FIG. 1 , but those skilled in the art having possession of the present disclosure will recognize that the client computing device 200 may include more or fewer components and still fall within the scope of the present disclosure.

[0052] Figure 3 An embodiment of a server computing device 300 is shown, which may be the server computing device described above with reference to FIG. Figure 1 However, in other embodiments, the server computing device 300 may be a selective forwarding unit 18 or a relay server 16. In the illustrated embodiment, the server computing device 300 includes a chassis 302 that houses the components of the server computing device 300. Figure 3Only some of these components are shown in the figure. For example, the chassis 302 can house a processing system (not shown) and a non-transitory memory system (not shown), the non-transitory memory system including instructions that, when executed by the processing system, cause the processing system to provide a peer application controller 304 that is configured to perform the functions of a peer application controller and / or server device discussed below. The peer application controller 304 can be configured to provide a peer application 304a that provides one or more coordinate grid environments to the client 14 / 200 so that the client 14 / 200 can participate in the peer application 304a hosted by the peer application controller 304. In various embodiments, the peer application controller 304 can include a coordinate grid state engine 304b that receives participant state information for each of the participants in each coordinate grid environment instance hosted by the peer application controller 304, compiles the participant states into a participant state payload 308a, and provides the participant state payload to various participants (e.g., the client 14 / 200), as discussed in further detail below.

[0053] The chassis 302 may also house a communication system 306 that is coupled to the peer application controller 304 (eg, via a coupling between the communication system 306 and a processing system) and is configured to communicate with the peer application controller 304 via a communication system 306. Figure 1 The communication system 306 may allow the server computing device 300 to communicate with the network 20 as described in detail below. Figure 1 The chassis 302 may also house a storage device (not shown) that provides a storage system 308 coupled to the peer application controller 304 via the processing system. The storage system 308 may be configured to store a participant state payload 308a or other data or instructions that will be apparent to one of ordinary skill in the art having the benefit of the present disclosure. In various embodiments, the storage system 308 may be provided on the server computing device 300 or on a database accessible via the communication system 306. Additionally, in some embodiments, the server computing device 300 may host a relay engine (not shown) to perform the functionality of the relay server 16. In other embodiments, the server computing device 300 may be a distributed server computing device in which functionality is performed by more than one server. Although in Figure 3 A particular server computing device 300 is shown in FIG. 1 , but those skilled in the art having possession of the present disclosure will recognize that the server computing device 300 may include more or fewer components and still fall within the scope of the present disclosure.

[0054] 1. Server authoritative coordinate grid status system

[0055] In some embodiments, server 12, relay server 16, or client 14 may operate, for example, by executing program code stored on a tangible, non-transitory machine-readable medium. Figure 4 The process 400 of . The operations shown may be executed in a different order, omitted, duplicated, executed concurrently, executed serially, insert additional operations, be fully automated, involve human intervention, or otherwise modified relative to the depicted arrangement, which is not meant to be limiting of any other description herein.

[0056] In some embodiments, process 400 Figure 4 4. Process 400 may begin at decision block 402 where it is determined whether a user input condition is satisfied. In an embodiment, at decision block 402, client computing device 14 / 200 and in particular grid state engine 204b may monitor client computing device 14 / 200 for any user input of a participant (e.g., avatar, object) provided by an application (e.g., peer application 304a hosted by server computing device 12 / 300 or peer application 204c provided on client computing device 14 / 200) that instructs the application to change state (e.g., move to a different location in the grid, change appearance or shape, or any other state change apparent to one skilled in the art having the present disclosure) that provides the grid environment. The user input may be a click of a mouse on a location in the coordinate grid environment of the application on a graphical user interface, a touch input on a touch screen of the client computing device 14 / 200, an input of a coordinate instruction, a voice command, a gesture, or any other user input that may cause a participant in the peer application 304a to change its state (such as its position in the coordinate grid environment provided by the peer application 304a). If no user input that affects a change in the state of a participant is detected, the process 400 continues to monitor user input at decision block 402.

[0057] If user input is detected, process 400 proceeds to block 404 where participant state information is captured. In an embodiment, at block 404, coordinate grid state engine 204b may capture participant state information associated with a participant in the coordinate grid environment of peer application 204c / 304a. For example, coordinate grid state engine 204b may capture participant location information associated with a participant. The participant location information may include the current location of that participant in the coordinate grid environment (e.g., two-dimensional coordinates, three-dimensional coordinates, or any other information describing the current location of the participant), the user input location of the participant (e.g., the coordinates in the coordinate grid environment entered by the user from decision block 402 indicating that the participant is moved to), and the user input time of the participant. Although specific participant state information is discussed, other state information that is apparent to one of ordinary skill in the art having the present disclosure may be used.

[0058] The process 400 may then proceed to block 406, where the client computing device provides the participant status information to the server computing device. In an embodiment, at block 406, the peer application controller 204 may provide the participant status information to the server computing device 12 / 300. For example, the peer application controller 204 may send the participant status information to the server 12 via the network 20 using the communication systems 210 and 306. The participant status information may include a client computing device identifier, an identifier of an application instance on which the participant is provided (e.g., in some embodiments, the client computing device 200 may have multiple application instances of, for example, peer application 204c open, so that each application instance may have different participants in the coordinate grid environment provided by the peer application 204c), or any other identifier that associates a participant in the coordinate grid environment with the participant status information of the participant. The participant status information may also include participant location information, such as the participant's current location in the coordinate grid, the location in the coordinate grid identified by the user input, and the time when the user input was received. However, the participant status information provided to the server 12 / 300 may include other participant status information.

[0059] The process may proceed to block 408, where the coordinate grid state is updated based on the participant state information. In an embodiment, at block 408, the coordinate grid state engine 204b may update the coordinate grid state 216b based on the participant state information, which may include participant location information of participants associated with the application instance operating on the peer application 204c. For example, the coordinate grid state 216b may include the distance between the participant in the coordinate grid environment and those other participants based on the participant state information of other participants. For example, the participant location information may include the current location of the participant in the coordinate grid environment, the user input location of the participant, and the time of the user input of this participant. The client computing device 14 / 200 may include the participant state information 216c for all participants in the coordinate grid environment. As discussed in more detail below with reference to block 416, the client computing device 14 / 200 may have previously received a participant state payload from the server computing device 12 / 300, which may include participant state information for at least a portion of the participants in the coordinate grid environment. The participant state information for the participant of the client computing device 14 / 200 and the participant state information 216c for the participant in the coordinate grid environment can be used to determine the distance between the participant in the coordinate grid environment and other participants (e.g., using a distance formula), which is stored as the coordinate grid state 216b. The coordinate grid state 216b can be used by the client computing device 14 / 200 for other determinations, such as Figure 5 As discussed in process 500.

[0060] Process 400 may include a decision block 410 at which the server computing device determines whether participant status information has been received from a client computing device. In an embodiment, at decision block 410, grid state engine 304b may determine whether any updated participant status information has been received. Server computing device 12 / 300 may communicate with client computing devices 14 / 200, which may provide participant status information for their respective participants. If not, grid state engine 304b may continue to monitor for participant status information at decision block 410.

[0061] If the server computing device has received participant state information, such as the participant state information from block 406, the process 400 may proceed to block 412, where the server computing device updates the participant state payload. In an embodiment, at block 412, the coordinate grid state engine 304b may update the participant state payload 308a with any received participant state information. As discussed above, the participant state information may include participant location information, such as the participant's current location in the coordinate grid environment, the participant's user input location, the time of the participant's user input, and an identifier for the participant or client computing device 14 / 200. The server computing device 12 / 300 may compile all participant state information for participants in the coordinate grid environment provided by the peer application 304a running on the peer application controller 304 into the participant state payload 308a. Thus, the server computing device 12 / 300 collects participant states for each participant, which may include three data points of the participant's current location in the coordinate grid environment, the participant's user input location, and the time of the participant's user input. The server computing device 12 / 300 simply updates any old participant state information with the new participant state information in the participant state payload 308a. The server computing device 12 / 300 may not compute the coordinate grid state for a participant. As discussed above, this allows the coordinate grid environment to scale in terms of participants by reducing the processing power and network bandwidth necessary to distribute state changes and updates to all other participants. Having so many confirmed updates controlled by a single server or process limits the number of participants in a world (such as a particular coordinate grid environment instance).

[0062] The process 400 then proceeds to block 414, where the participant state payload is distributed to at least a portion of the participants in the coordinate grid environment. In an embodiment, at block 414, the server computing device 12 / 300 may distribute the participant state payload 308a to at least a portion of the client computing devices 14 / 200 in the coordinate grid environment. The distribution of the participant state payload 308a may occur in response to an update to the participant state payload 308a. As discussed above, the participant state payload may include participant state information for at least a portion of the participants in the coordinate grid environment. For example, the participant state information may include participant location information including the current location of the participant in the coordinate grid environment, the user input location of the participant, and the time of the user input of the participant. However, as discussed herein, the participant state information may include many parameters that are apparent to those skilled in the art who have mastered the present disclosure.

[0063] The process 400 then proceeds to block 416 where the client computing device receives a participant state payload from the server computing device. In an embodiment, the client computing device 14 / 200 may receive the participant state payload 308a from the server computing device 12 / 300 at block 416. The participant state payload 308a may be received by at least a portion of the client computing devices 14 / 200 in the computing environment 10 to calculate their own coordinate grid state 216b.

[0064] Process 400 then proceeds to block 418, where the client computing device updates its coordinate grid state based on the participant state information included in the participant state payload. In an embodiment, at block 418, when comparing the participant state information received in the participant state payload with the previous version of the participant state information received in the previous participant state payload, the coordinate grid state engine 204b may determine whether there has been a change in the participant state information. The coordinate grid state engine 204b may update the participant state information 216c and then recalculate the coordinate grid state 216b. Thus, the processing of the coordinate grid state 216b is distributed, so that each client computing device 14 / 200 performs the calculation instead of the server computing device 12 / 300. This reduces the use of processing resources and other resources of the server computing device 12 / 300 and the communication requirements on the network 20, which allows the coordinate grid environment to scale while still maintaining satisfactory consensus facts in the coordinate grid state between participants.

[0065] As discussed above, problems arise in peer-to-peer communication establishment by offloading coordinate grid state calculations to individual client computing devices when those peer communication initiators rely on proximity between participants. This is because the coordinate grid state can be calculated by client computing devices at different times. Thus, a client computing device that has calculated the coordinate grid state and has determined that another participant is within range of its participant can initiate a peer connection without the other participant determining that it can initiate a peer connection, which can cause the other participant to ignore the peer connection initiation signal sent from the client computing device. Thus, the systems and methods of the present disclosure provide a signal cache that allows a client computing device to store a peer connection initiation signal until the coordinate grid state is updated by the client computing device and it can determine that it meets the conditions for negotiating a peer connection.

[0066] 2. Peer-to-peer mesh network signal cache

[0067] In some embodiments, server 12, relay server 16, or client 14 may operate, for example, by executing program code stored on a tangible, non-transitory machine-readable medium. Figure 5The process 500 of . The operations shown may be performed in a different order, omitted, duplicated, performed concurrently, performed serially, insert additional operations, fully automated, involve human intervention, or otherwise modified relative to the depicted arrangement, which is not meant to be limiting of any other description herein.

[0068] In some embodiments, process 500 Figure 5 . The process 500 may begin at a decision block 502 where it is determined whether a peer connection condition is met to initiate a peer connection between client computing devices associated with a participant. In an embodiment, at decision block 502, a peer application 204c of a first client computing device 14a / 200a may determine whether a condition is met to establish a peer connection with a second client computing device 14b / 200b. For example, the peer application 204c may determine whether a participant associated with the second client computing device 14b / 200b is adjacent to a participant associated with the first client computing device 14a / 200a. The peer application 204c may determine the distance between the participant of the client computing device 14a / 200a and other participants in a coordinate grid environment provided by the peer application 204c. To this end, the peer application 204c may reference a coordinate grid state 216b determined by the client computing device 14a / 200a using the process 400 described above. While peer connection conditions may be based on proximity or a predetermined distance, other peer connection conditions are also contemplated in which asynchrony between participants occurs, such that one participant is ready for peer connection while another participant is not ready for peer connection. In another embodiment, a peer connection condition for initiating a peer connection may include a user input for connecting a participant with another participant. While some conditions for initiating a peer connection are discussed, those skilled in the art having access to the present disclosure will recognize that other conditions may result in a client computing device initiating a peer connection with another participant.

[0069] Process 500 may proceed to block 504 where the first client computing device sends a peer connection initiation signal to the second client computing device and the second client computing device receives the connection initiation signal at block 506. In various embodiments, the first client computing device 14a / 200a may determine that it will be the initiator of the peer connection with the second client computing device 14b / 200b. For example, the first client computing device 14a / 200a may determine that it is the initiator based on its participant identifier and the participant identifier associated with the second client computing device 14b / 200b. Specifically, the first client computing device 14a / 200a may determine that its participant identifier is greater than the participant identifier associated with the second client computing device 14b / 200b, which satisfies the participant identifier condition. However, in other embodiments, the participant identifier assigned the lowest participant identifier may be the initiator of the peer connection, or other information other than the participant identifier is used to determine the initiator status.

[0070] Thus, the first client computer device 14a / 200a generates a peer connection initiation signal and sends it to the second client computer device 14b / 200b. For example, the peer application 204c can communicate with a real-time communication (RTC) application programming interface (API) (e.g., a WebRTC API). The WebRTC protocol may include an interactive connectivity establishment (ICE) protocol for establishing a peer connection. Thus, the peer initiation signal may include an SDP proposal according to a session description protocol (SDP) protocol. In various embodiments, the peer initiation signal is provided via the network 20 and the relay server 16. For example, the peer initiation signal is provided by a persistent connection (e.g., a web socket) established between the first client computing device 14a / 200a and the relay server 16 and a persistent connection established between the second client computing device 14b / 200b and the relay server 16. The second client computing device 14b / 200b can then receive the peer initiation signal.

[0071] The process 500 may then proceed to decision block 508 where a determination is made by the second client computing device whether a condition for processing a peer initiation signal is met. In an embodiment, at decision block 508, the second client computing device 14b / 200b may determine whether a condition for processing a peer initiation signal exists. For example, the second client computing device 14b / 200b may determine whether it is available to process a peer initiation signal. In some embodiments, the second client computing device 14b / 200b may be available when the second client computing device 14b / 200b determines that a peer connection condition is met, such as a proximity condition to the first client computing device 14a / 200a is met. As discussed above, the second client computing device 14b / 200b may determine that this condition is satisfied at a different time than the first client computing device 14a / 200a due to network connectivity in the case where the second client computing device 14b / 200b has established a maximum number of peer connections, differences in processing speed between the second client computing device 14b / 200b and the first client computing device 14a / 200a when determining the coordinate grid state 216b, or any other reason why the first client computing device 14a / 200a determines that a peer connection with the second client computing device 14b / 200b should be established before the second client computing device 14b / 200b determines that a peer connection is to be established. For example, the peer connection may depend on whether the second client computing device 14b / 200b has any available peer connections. Specifically, each client computing device 14 / 200 may be assigned a maximum number of peer connections (e.g., 10 peer connections, 8 peer connections, and / or any other number of peer connections apparent to one of ordinary skill in the art having the present disclosure). The number of peer connections may be based on the quality requirements of the media streams (audio streams, video streams, audiovisual streams or other data streams) to be provided via the peer connections.

[0072] In conventional systems, if the second client computing device 14b / 200b has determined that the peer condition is not met, the second client 14b / 200b may ignore the peer connection signal it receives. Thus, the first client computing device 14a / 200a may be waiting for a response (e.g., an SDP reply to an SDP offer) without a response reaching it each time.

[0073] If it is determined at decision block 508 that the second client computing device is not available, process 500 may proceed to block 510 where the peer connection initiation signal is stored in the local signal cache 216a. In an embodiment, at block 510, the relay server 16 or the second client computing device 14b / 200b may store the peer connection initiation signal in the local signal cache 216a via the signal cache engine 204a. For example, the local signal cache 216a may include a data structure including an identifier of an initiator (e.g., the first client computing device 14a / 200a or a participant associated with the first client computing device 14a / 200a), the peer connection initiation signal, and the time at which the peer connection initiation signal was received. Thus, the local signal cache 216a may include an entry for each initiator of the second client computing device 14b / 200b. If the peer connection initiation signal is already associated with the initiator (e.g., the first client computing device 14a / 200a) in the local signal cache 216a, the second client computing device 14b / 200b may update the peer connection initiation signal with the latest peer connection initiation signal and remove any previously received peer connection initiation signal. The second client computing device 14b / 200b may continue to monitor whether the peer connection condition exists by returning to decision block 508.

[0074] In various embodiments, the relay server 16 may provide a local signal cache 216a for each of the client computing devices 14. For example, the relay server 16 may cache peer-to-peer connection initiation signals and pass those peer-to-peer connection initiation signals between the first client computing device 14a / 200a and the second client computing device 14b / 200b when requested. If the signal is not present on the server-side signal cache, the participant / client may enter a wait state in which either side will process the signal in real time while bypassing the server-side signal cache.

[0075] In various embodiments of the present disclosure, a receiving client (e.g., a second client computing device 14b / 200b) may determine that it is available to receive a peer initiation signal before an initiating client (e.g., a first client computing device 14a / 200a) sends a proposal signal. In this scenario, the second client computing device 14b / 200b may first check the local signal cache 216a for the peer initiation signal, and then wait for the peer initiation signal from the first client computing device 14a / 200a. When the peer initiation signal arrives, the second client computing device 14b / 200b may immediately process the peer initiation signal in accordance with block 512 of process 500 without storing the proposal signal in the local signal cache.

[0076] If, at decision block 508, it is determined that the peer condition has been met, then process 500 may proceed to block 512 where a peer connection initiation signal is processed. In an embodiment, at block 512, the second client computing device 14b / 200b may process the peer connection initiation signal and, at block 514, respond to the first client computing device 14a / 200a via the relay server 16 with a peer connection response signal. In various embodiments, once the second client computing device 14b / 200b is available, the second client computing device 14b / 200a may check the local signal cache 216a to determine if the initiator identifier associated with the first client computing device 14a / 200a includes a peer connection initiation signal and, if so, proceed to block 514. For example, when the peer condition has been met, the second client computing device 14b / 200b will know to listen for a peer initiation signal from the first client computing device 14a / 200a, which may be identified by the identifier. Thus, the second client computing device 14b / 200b checks whether any peer connection initiation signals have been cached in the local signal cache 216n and processes those signals if the peer condition is met (eg, within proximity range of the client computing device associated with the cached signal).

[0077] In various embodiments, at block 514, the peer connection response signal may include an SDP answer or any other response signal apparent to one of ordinary skill in the art having the present disclosure. At block 516, the first client computing device 14a / 200a may receive the peer connection response signal and process the response signal to establish a peer connection (e.g., a real-time connection using WebRTC). At block 518, one or more media streams (e.g., a video stream, an audio stream, an audiovisual stream, or other media content streams) may then be provided over the peer connection between the first client computing device 14a / 200a and the second client computing device 14b / 200b.

[0078] In some embodiments, the first client computing device 14a / 200a may determine that the peer connection condition no longer exists while it is waiting for the peer connection response signal. Thus, the first client computing device 14a / 200a may ignore the peer connection response signal received from the second client computing device 14a / 200a, or disconnect the peer connection after the peer connection is established. In some embodiments, once the peer connection is established, the peer connection condition may be disconnected when the peer connection condition is not satisfied (e.g., once the participants move away from each other such that the proximity value does not satisfy the proximity value condition).

[0079] Figure 6An example of a peer-to-peer signal caching system 600 that may be provided by computer system 10 is shown, and examples of processes 400 and 500 discussed above are shown. Each participant 602, 604, and up to 606 may communicate location information to a coordinate grid state server 608 (e.g., server 12 / 300). The location information may include a current location, a user-entered next location, and a user-entered time. The coordinate grid state server 608 may store the location information of each participant 602 to 606 in a cache database 610 and provide the current participant state (location) payload from the cache database 610 to the participants 602 to 606. The cache database 610 may require only a single data point to distribute the information to each other participant through the coordinate grid state server 608. Given a constant velocity, each participant 602 to 606 may calculate the current location of each other participant based on a line drawn through the two coordinates of the participants (e.g., participant 602 and participant 604) and the user-entered time.

[0080] Because each participant 602, 604, and 606 independently determines the coordinate grid state using a participant state payload provided by a coordinate grid state server 608 (e.g., server 12) that stores updateable participant state payloads in a cache database 610, participant 602 can determine that they are within a connectable distance from participant 604 faster than participant 604 determines that participant 602 is within a connectable distance. This means that participant 602 can send a peer connection initiation signal via a peer signal relay service (e.g., provided by relay server 16) before participant 604 is ready to consume the signal. Participant 604 can store the peer connection initiation signal in its local signal cache 604a. Similarly, if participant 602 or 606 receives a peer connection initiation signal before they are ready to consume the peer connection initiation signal from the participant that sent the peer connection initiation signal, participant 602 can store the peer connection initiation signal in local signal cache 602a, and participant 606 can store the peer connection initiation signal in local signal cache 606a.

[0081] Thus, when the coordinate grid state for participants in a coordinate grid environment is determined by each participant's associated computing device rather than a state server and this coordinate grid state is used to determine whether a peer connection should be established between participants, the systems and methods of the present disclosure provide peer caching. As discussed above, participants may determine the coordinate grid state at different times, which may cause a client computing device to miss a peer connection initiation signal (e.g., if the client computing device does not determine that it is proximate to another client computing device that has determined that the participants of the two client computing devices are proximate to each other). This allows mesh networks of coordinate grid environments with peer connections, such that the scale of the coordinate grid environment can accommodate hundreds, thousands, or tens of thousands of participants and allow them to communicate with each other when they are proximate to each other, while limiting the number of participants that can communicate with them.

[0082] Client-side authoritative grid state distributed across peer connections

[0083] As discussed above, participant state information may be provided to a coordinate grid state server. The coordinate grid state server may compile at least a portion of the participant state information and distribute the participant state information in a participant state payload. The participant state information may include participant location information, and each participant determines the coordinate grid state of the participant within the coordinate grid according to the participant location information. The participant location information may include the location of the participant, the user input location, and the time of the user input according to which the distance or coordinate grid state between the participants can be determined. The distance may be used to determine whether a peer connection should be established by satisfying a proximity condition. A local signal cache is used to cache peer connection initiation signals, and because the generation of their own coordinate grid states by the participants may occur at different times, the participants may facilitate peer connections. In addition to providing the participant location information to the coordinate grid state server, other participant state information may also be provided to the coordinate grid state server. However, continuously providing participant state information via the coordinate grid server may be inefficient in terms of network, processing, and memory resources.

[0084] In various embodiments of the present disclosure, once participants in a coordinate grid environment provided by an application are connected to each other via a peer-to-peer channel or connection, each participant can pass participant status information or participant status changes to the connected participant when something about their own status changes or when the participant connects to another new participant. The participant receiving the status change can relay this status change to other peer-to-peer connected participants. If another connected participant has already received a status change from the initial participant and has received a relay message. The participant will ignore the relay message. In some embodiments, the participant can ignore the relay message with the most hops, the last received relay message, or based on any other conditions obvious to those skilled in the art.

[0085] In various embodiments, multiple independent participant mesh networks in a coordinate grid environment may be formed, where there is no peer-to-peer connection between two independent participant meshes. When a new participant enters the grid, the participant provides the participant state it is tracking to the participants already in the grid. The participants in the grid may provide the participant state it is tracking to the new participant of the grid. Thus, the new participant and the existing grid participants will merge their exchanged state databases together. The existing grid participants will relay the participant state of the new participant to the participants in the grid it is connected to, and so on, until all participants in the grid have the participant state information of the new participant.

[0086] In some embodiments, not all state changes will be communicated to all participants. In some embodiments, the state change of a participant may only affect one of the participants, without affecting other participants. For example, if the participant is in a paintball game, the first participant can shoot the second participant. The shooting behavior performed by the first participant can provide a state change to the world. For example, the shooting behavior can generate a visualization in the world of paintball shooting, which will need to be communicated to all other participants in the grid so that these participants can view the shooting of the paintball in the visualization. However, in other embodiments, the shooting of the paintball may not be associated with visualization. Thus, only the second participant who is shot and will receive the deduction in terms of health or life value as a result of the shooting will need to receive the state change generated by the first participant. In response to being shot, the second participant can experience a state change of reduction in terms of health or life value, and generate a state change communication to send to the connected participant. Thus, the peer application can be configured to communicate a state change (e.g., a global visualization associated with a state change or only local knowledge) based on satisfying a state change condition. Otherwise, in order to save network and processing resources, if there is no impact on other participants, the state change may not be communicated, or the state change may be sent only to the first group of participants but not to the second group of participants, where the state change of the first group of participants affects the first group of participants but not the second group of participants.

[0087] In various embodiments, a participant maintains all participant states that the participant is aware of, even if the participant is disconnected from the grid or another participant is disconnected from the grid. A participant sends the participant states of other participants until the participant receives a new participant state of the other participant. When a participant state is received by a participant, the participant will check the timestamp to determine if the received participant state is newer than the stored participant state. If the received participant state is newer, the participant will update the participant state of the other participant with the received participant state. Otherwise, if the received participant state is older, the participant will maintain the participant state it has stored.

[0088] In some embodiments, server 12, relay server 16, or client 14 may operate, for example, by executing program code stored on a tangible, non-transitory machine-readable medium. Figure 7 The process 700 of . The operations shown may be performed in a different order, omitted, duplicated, performed concurrently, performed serially, insert additional operations, fully automated, involve human intervention, or otherwise modified relative to the depicted arrangement, which is not meant to be limiting of any other description herein.

[0089] In some embodiments, process 700 Figure 7The process 700 may begin at block 702 where a peer-to-peer connection between a first client computing device and a second client device is established. In an embodiment, at block 702, a peer-to-peer connection between the first client computing device 14a / 200a and the second client computing device 14b / 200b may be established. The establishment of the peer-to-peer connection may be based on the above process. Figure 4 or Figure 5 4 and 500. However, in other embodiments, the peer connections may be established according to other processes apparent to one of skill in the art having knowledge of the present disclosure. In various embodiments, the first client computing device 14a / 200a or the second client computing device 14b / 200b may each establish multiple peer connections each establishing connections with a corresponding client computing device. Thus, for example, the first client computing device may establish a second peer connection with a third client computing device, a third peer connection with a fourth client computing device, a fourth peer connection with a fifth client computing device, and so on, until a maximum peer connection limit is reached, which may be based on performance, system definition, or client computing device capabilities.

[0090] With reference to the embodiments described in process 400 and process 500, the establishment of a peer-to-peer connection may occur when a first participant in a coordinate grid environment provided by a peer application 204c or 304a and associated with a first client computing device 14a / 200a meets a proximity condition with a second participant in the coordinate grid environment and associated with a second client computing device 14b / 200b. Thus, the state server computing device 12 / 300 may provide participant state information (e.g., participant location information) to each of the participants in the coordinate grid environment, and each participant may be able to calculate the distance and location or other coordinate grid state of each participant based on this participant state payload. However, in other embodiments of the present disclosure and in embodiments discussed in detail below, each of the participants may belong to the same mesh network of the participants of the peer connection. For example, the first client computing device 14a / 200a and the second client computing device 14b / 200b may be connected to each other via one or more intermediate client computing devices. Based on the shared participant state information provided by the peer-to-peer connection of the mesh network, the first client computing device 14a / 200a and the second client computing device 14b / 200b may obtain each other's participant location information and determine that they are in proximity to each other so that a peer-to-peer connection may be established.

[0091] In some cases, at least the participant location information of the participant status information needs to be communicated to the state server computing device 12 / 300. For example, if a first participant of a first client computing device 14a / 200a joins a coordinate grid environment, the other participants do not know the location of the first participant in the coordinate grid environment, and the first participant does not know the location of other participants in the coordinate grid environment. In another case, multiple mesh networks can operate in a coordinate grid environment without intermediate participants. Thus, when the participants of each grid move around in the coordinate grid environment, those participants of each mesh network may not know the location of the participants in other mesh networks, making it difficult to determine the current location of each participant between mesh networks so that the proximity condition is met to establish a peer-to-peer connection.

[0092] Process 700 may proceed to decision block 704 where it is determined whether a state transition condition exists. In an embodiment, at decision block 704, the first client computing device 14a / 200a may determine whether the state transition condition is satisfied such that the first client computing device 14a / 200a initiates a transfer of the stored participant state information 216c. The participant state information may include participant state information for a participant in a coordinate grid environment. As discussed above, the participant state information may include participant location information, but may also include other state information used to determine authenticity in a coordinate grid environment. For example, the participant state information may include participant health, abilities, virtual items associated with the participant, speed, appearance, blockchain wallet identifier, token identifier, or other attributes / states that may be updateable. In various embodiments, the state transition condition may be satisfied when the first participant changes one or more states such that the participant state information for the first participant is updated. In another embodiment, the state transition condition may be satisfied when the first client computing device 14a / 200a detects a state change or causes a state change of another participant in the coordinate grid environment. Detection of a state change may include receiving participant state information from another client computing device over a peer-to-peer connection with the client computing device that updated at least one data entry in participant state information 216c. If the state transition condition is not met, then at decision block 704, the first client computing device will continue to monitor for the state transition condition to be met.

[0093] If the state transfer condition is met at decision block 704, process 700 may proceed to block 706, where at least a portion of the participant state information stored on the first client computing device is provided over at least a portion of the peer connection that has been established by the first client computing device 14a / 200a. In an embodiment, at block 706, the first client computing device 14a / 200a may provide at least a portion of its participant state information 216c to other client computing devices connected to the first client computing device 14a / 200a via a peer connection. For example, at least a portion of the participant state information 216c may be provided via a peer connection with a second client computing device 14b / 200b. However, in some embodiments, the participant state information may not be transferred over a portion of the peer connection hosted by the first client computing device 14a / 200a. For example, if the reason for transmitting the participant state information is based on participant state information received via a peer connection, the participant state information may not be provided via the peer connection whose communication caused the participant state information to be transmitted. This may prevent state information from being circulated via a peer connection. However, in other embodiments, participant state information may be provided over all peer connections, and there may be other mechanisms to prevent circulation of participant state information over peer connections.

[0094] In some embodiments, all participant state information 216c may be transferred via a peer-to-peer connection. In other embodiments, only incremental updates from aspects of participant state information may be transferred via a peer-to-peer connection. In other embodiments, specific participant state information based on information type may be transferred without transferring another type of participant state information. For example, participant health information may be transferred without transferring participant location information. In other embodiments, participant state information associated with a certain group of participants may be transferred without transferring participant state information associated with another group of participants.

[0095] The process 700 proceeds to decision block 708 where the second client computing device is monitoring whether it has received any participant state information via its peer connection. In an embodiment, at decision block 708, the second client computing device 14b / 200b (although the first client computing device 14a / 200a may also perform this step) may monitor for any participant state information received on its established peer connection. If no participant state information is received, the process 700 may return to decision block 708.

[0096] However, if participant state information has been received at decision block 708, process 700 may proceed to block 710 where the stored participant state information is updated with any more recent participant state information. In various embodiments, at block 710, the second client computing device 14b / 200b may determine whether to update any of its participant state information 216c. The second client computing device 14b / 200b may compare each participant state information entry for each participant it has in the participant state information 216c with the participant state information it received via the peer-to-peer connection. For example, the second client computing device 14b / 200b may compare the participant entry for a first participant in the participant state information 216c with the participant state information it received for the first participant from the first client computing device 14a / 200a or other client computing devices. The second client computing device 14b / 200b may compare the timestamps to determine which participant state entry is the most recent. If the received participant state information is more recent than the stored participant state information, the more recent participant state information may be used to update the portion of the participant state information 216c maintained by the second client computing device 14b / 200b.

[0097] Process 700 may proceed to block 712, where the updated participant state information may result in a change in the I / O device output associated with the second client computing device. In an embodiment, at block 712, the updated participant state information may result in an output being generated via the user I / O system 218 or the display system 220. For example, the participant state information may have included updated participant position information for the first participant. Thus, the peer application 204c or 304a may cause a graphical user interface including a visual representation of the first participant to move via the display system 220 of the client computing device 14b / 200b. In other embodiments, the updated participant position information may be for a third participant whose third client computing device is upstream of the second client computing device 14b / 200b via the first client computing device 14a / 200a. Thus, if the position information now indicates that the third participant is entering a viewpoint displaying a coordinate grid environment on the second client computing device 14b / 200b, this coordinate grid environment may cause the display system 220 to display the third participant moving into the coordinate grid environment displayed by the second client computing device. Although specific examples of providing variations in I / O device outputs are shown, other I / O device outputs are contemplated and remain within the scope of the present disclosure.

[0098] Process 700 may proceed to decision block 714 where it is determined whether a state transition condition is satisfied. In an embodiment, at decision block 714, the second client computing device 14b / 200b may determine whether the state transition condition is satisfied. Decision block 714 may be operated by the first client computing device similar to decision block 704 and therefore will not be discussed in detail.

[0099] When the state transfer condition is met in decision block 714, process 700 may then proceed to block 716. At block 716, the second client computing device may transfer at least a portion of the updated participant state information over the peer-to-peer connection. The operations of block 716 may operate similarly to the operations of block 706, and therefore will not be discussed in detail. Thus, the second client computing device 14b / 200b may provide its participant state information to the first client computing device 14a / 200a via the peer-to-peer connection, and the first client computing device 14a / 200a may complete the operations in blocks 708 to 712 before returning to decision block 704. Similarly, the second client computing device 14b / 200b may provide its participant state information to another client computing device 14a / 200a via another peer-to-peer connection.

[0100] Figure 8 An example coordinate grid environment 800 according to various embodiments of the present disclosure is shown. The coordinate grid environment 800 or virtual world may include two multi-participant closed mesh networks 802 and 804. Mesh network 802 may include participants 1, 2, and 3, each connected to each other via peer connections 802a, 802b, and 802c. Mesh network 804 may include participants 4, 5, and 6, each connected to each other via peer connections 804a, 804b, and 804c. Initially, participant 7 is independent and can be considered a third "mesh network." In both mesh networks 802 and 804, all participants interact with each other and pass participant state information to indicate changes in participant states to each other when something about their own state changes or some other state transition condition is met. For example, if participant 2 updates some information about themselves, then participant 2 will send a time-stamped update to each other participant (e.g., participant 1 and participant 3) to which they are connected. Participants 1 and 3 will relay this information to all other participants they are connected to, but Participant 3 will ignore Participant 1's relay of Participant 2's update because Participant 3 is connected to Participant 2 and has or will receive this information from Participant 2. Participant 1 will do the same for Participant 3's relay. Participants 1 and 3 can distinguish the relayed information from other relays based on the timestamp and location of the information's origin.

[0101] In the example shown, participant 7 can move independently between mesh networks 802 and 804. At some point, participant 7 and participant 3 can establish a peer connection 806a as described above. Participant 3 is still part of mesh network 802, but has a peer connection 806a to participant 7. Participant 7 can join mesh network 802. Upon connection, the following may occur: (1) Participant 7 can provide participant 3 with at least a portion of the participant state information that participant 7 is tracking. Participant 3 can provide participant 7 with at least a portion of the participant state information that participant 3 is tracking. Participants 7 and 3 will each merge their exchanged state databases together. Participant 7 will now know everything that participant 3 knows, and participant 3 will also know everything that participant 7 knows.

[0102] Subsequently, if participant 7, which is connected to participant 3, decides to connect to participant 4, and participant 4 responds via peer connection 806b. Participants 4 and 7 may exchange each other's participant state databases (e.g., participant state information) in the manner that participant 7 and participant 3 have done above. As a result, participant 7 may now know the information that participant 4 knows, and vice versa. In addition, participant 7 may relay any new information received from participant 4 to participant 3. Participant 3 will do the same with the new participant state information received from participant 7, so that participants 1 and 2 will receive the new information. Similarly, participant 4 may relay some or all of the new participant state information received from participant 7 to participants 6 and 5.

[0103] At the end of the merge and relay, all participants (e.g., participants 1 to 7) across mesh networks 802 and 804 through participant 7 now know and agree on the participant location information or coordinate grid state of all other participants. This information exchange can be performed without the need for a central server.

[0104] Thus, the disclosed peer-to-peer mesh technology allows peer connections between peers to smoothly come and go out of existence. This peer state exchange mechanism allows game-like states to be shared between all participants in a large world. The use of peer connections allows coordinate grid states like this to be propagated throughout the large-scale world / coordinate grid without the overhead of a state server computing device for at least a portion of the participant state information.

[0105] Hybrid peer-to-peer grid and SFU flow distribution

[0106] As discussed above, in certain application environments, there are increasingly common applications for streaming video and audio between participants. As the cluster of application instances grows, it becomes increasingly difficult to reliably maintain connections between subsets of participants. It is rarely necessary to maintain streams between all participants, but in many cases, proximity connections are very valuable.

[0107] A peer-to-peer mesh topology can adequately and cheaply connect a subset of a cluster of instances, but there is still the challenge of distributing a single stream to a larger number of people. World events may want to make announcements to the entire world. Peer-to-peer has its limitations. Due to encoding processing and upload bandwidth limitations, a single client can only maintain more than 10 streams simultaneously. A streaming server (e.g., SFU) hybrid with peer-to-peer may help bridge this gap.

[0108] It is efficient and inexpensive to run world events and utilize peer-to-peer grids for most smaller group interactions. However, there are situations where it is necessary or valuable to distribute one or more streams to all users in a world or region of a world. Peer-to-peer grid streaming may not be sufficient to stream instances to a larger number of participants (e.g., more than 10 users, at least 15 users, at least 20 users, at least 50 users, at least 100 users, at least 1000 users, at least 10,000 users, or any other number of users that meet a threshold quality degradation level that is obvious to those skilled in the art who master the present disclosure). For example, most client computing devices are not designed to stream to only a few peers simultaneously through a peer-to-peer connection. Viewers who reach hundreds or thousands of viewers for a single stream benefit from a selective forwarding unit (SFU).

[0109] Therefore, compared to establishing peer connections and streaming through these peer connections, the systems and methods of the present disclosure provide knowing when to utilize SFU to publish streams and when to obtain streams from SFU. Knowing when to utilize SFU and peer networks can improve network efficiency to distribute streams in a streaming environment.

[0110] For example, each participant in the coordinate grid environment may be in: (1) a basic state, (2) a broadcast state, (3) a screen sharing state, or any other state that would be apparent to one skilled in the art having the present disclosure. A participant in the basic state can only connect to the participants around them. This participant can connect to every other participant in its immediate vicinity via a peer-to-peer connection, which can be based on the above in Figure 4 The course 400 and Figure 5In addition, participants in the basic state can upload their audio stream or video stream through this peer connection. In various embodiments, participants in the basic state can read the stream of each participant around them based on the state of each other participant. If a nearby participant is also in the basic state, the participant will read the stream from the peer connection with the nearby participant. If a nearby participant is in the broadcast state or the screen sharing state, this participant can read the stream from the SFU.

[0111] As discussed above, in some cases, a participant may be in a broadcast state. A participant in a broadcast state may provide a stream that will be seen / heard by more participants than those around them. In some embodiments, if a participant is broadcasting, the participant may upload their stream to the SFU. In various embodiments, the broadcasting participant may also establish a peer-to-peer connection with each nearby participant, which may be performed in accordance with the signal caching methods described above in processes 400 and 500. Participant state information, which may include the communication mode (e.g., basic state, broadcast state, or screen sharing state) that the participant is in, may be transmitted via a state server computing device or in accordance with the above-discussed signal caching methods. Figure 7 The process 700 is transferred to each participant. In various embodiments, since the broadcasting participant is uploading their stream to the SFU, it may not be necessary for the broadcasting participant to upload their stream to the peer connection, and each nearby participant will know to get the broadcasting participant's stream from the SFU. If the nearby participant is not in the broadcast state or the screen sharing state, as discussed in further detail below, the nearby participant will become a broadcasting participant through the established peer stream.

[0112] In some cases, a participant may be in a screen sharing state. In some embodiments, if the stream of a participant in a screen sharing state is at a high enough quality to make the stream readable, the stream can be distributed. If any participant shares their screen, this stream will be uploaded to and distributed from the SFU, similar to broadcast mode.

[0113] In various embodiments, through the exchange of participant status information discussed herein, each participant may be aware of the current participant status of every other participant in the world.

[0114] In some embodiments, the server 12, the relay server 16, the client 14, or the selective forwarding unit server 18 may operate, for example, by executing program code stored on a tangible, non-transitory machine-readable medium. Fig. 9The process 900 of . The operations shown may be performed in a different order, omitted, duplicated, performed concurrently, performed serially, insert additional operations, fully automated, involve human intervention, or otherwise modified relative to the depicted arrangement, which is not meant to be limiting of any other description herein.

[0115] In some embodiments, process 900 Fig. 9 The process 900 may begin at block 902 where a peer-to-peer connection between a first client computing device and a second client device is established. In an embodiment, at block 902, a peer-to-peer connection between a first client computing device 14a / 200a and a second client computing device 14b / 200b may be established. The establishment of the peer-to-peer connection may be based on the process described above. Figure 4 or Figure 5 4 and 500. However, in other embodiments, the peer connections may be established according to other processes apparent to one of skill in the art having knowledge of the present disclosure. In various embodiments, the first client computing device 14a / 200a or the second client computing device 14b / 200b may each establish multiple peer connections, each of which is a peer connection with a corresponding client computing device. Thus, for example, the first client computing device may establish a second peer connection with a third client computing device, a third peer connection with a fourth client computing device, a fourth peer connection with a fifth client computing device, and so on, until a maximum peer connection limit is reached, which may be based on performance, predetermined or dynamic system limits, or client computing device capabilities.

[0116] With reference to the embodiments described in process 400 and process 500, establishment of a peer-to-peer connection may occur when a first participant in a coordinate grid environment provided by peer application 204c or 304a and associated with a first client computing device 14a / 200a satisfies a proximity condition with a second participant in the coordinate grid environment and associated with a second client computing device 14b / 200b. Thus, state server computing device 12 / 300 may provide participant state information (e.g., participant location information) to each of the participants in the coordinate grid environment, and each participant may be able to calculate the distance and location or other coordinate grid state of each participant based on this participant state payload. However, in the present disclosure and with reference to above, Figure 7In other embodiments discussed in detail in process 700, each of the participants may belong to the same mesh network of participants of the peer-to-peer connection. For example, the first client computing device 14a / 200a and the second client computing device 14b / 200b may be connected to each other via one or more intermediate client computing devices. Based on the shared participant state information provided by the peer-to-peer connection of the mesh network, the first client computing device 14a / 200a and the second client computing device 14b / 200b may obtain each other's participant location information and determine that they are in proximity to each other so that a peer-to-peer connection may be established.

[0117] Process 900 may proceed to decision block 904 where it is determined whether a selective forwarding unit condition is satisfied. In an embodiment, at decision block 904, the first client computing device 14a / 200a may determine whether the selective forwarding unit condition is satisfied, such that the first client computing device 14a / 200a provides the media stream via the selective forwarding unit 18 when the selective forwarding unit condition is satisfied, or the first client computing device 14a / 200a provides the media stream via a peer connection when the selective forwarding unit condition is not satisfied. For example, as discussed above, a media stream that requires high quality or some other condition to reach a larger group of participants in a coordinate grid environment may be better transmitted via the selective forwarding unit 18 than a peer connection. Thus, at decision block 904, the first client computing device 14a / 200a may monitor the media stream and transmission information associated with the media stream to determine whether the number of participants to which the media stream is destined exceeds a participant threshold, whether the media stream exceeds a quality threshold that cannot be satisfied by a peer connection, or some other selective forwarding unit condition that is apparent to one skilled in the art having the present disclosure.

[0118] If it is determined at decision block 904 that the media stream does satisfy the selective forwarding unit condition, process 900 may proceed to block 906 where the first client computing device enters a selective forwarding unit state. In an embodiment, at block 906, the first client computing device 14a / 200a may update the communication state to a selective forwarding unit state. In some embodiments, the selective forwarding state may include, for example, a broadcast state when the media stream exceeds a participant threshold. In other embodiments, the selective forwarding state may include, for example, a screen sharing state when the media stream satisfies a quality threshold. In various embodiments, a change in the communication state identified in the participant state information may result in a change in the selective forwarding unit state. Figure 7 The state transition condition is satisfied in decision block 704 of process 700. Thus, the communication state of the first client computing device 14a / 200a changes according to Figure 7The process 700 is communicated to other participants and their associated client computing devices via a peer-to-peer connection. However, in other embodiments, the participant state information may be distributed to participants of the coordinate grid environment via a state server computing device in a participant state payload 308a, as described above in Figure 4 As discussed in the process.

[0119] Process 900 may proceed to block 908, where the first client computing device provides a media stream via a selective forwarding unit connection with a selective forwarding unit. In an embodiment, at block 908, the first client computing device 14a / 200a may provide the media stream to the selective forwarding unit 18 via the communication system 210. The selective forwarding unit 18 may provide the media stream to the participants of the coordinate grid environment identified in the broadcast. In some embodiments, the participant may be a portion of a plurality of participants in the coordinate grid environment. For example, when the first client computing device 14a / 200a is in a broadcast state, the selective forwarding unit 18 may provide the media stream to all participants in the coordinate grid environment or a portion of the participants in the coordinate grid environment that exceeds the peer connection maximum. In another example, when the first client computing device 14a / 200a is in a screen sharing state, the selective forwarding unit 18 may provide the media stream to all participants in the coordinate grid environment or a portion of the participants in the coordinate grid environment that may or may not exceed the peer connection maximum (because the screen sharing state is quality-based).

[0120] Referring back to decision block 904, if the selective forwarding unit condition is not met, process 900 may proceed to block 910, where the first client computing device may enter a peer-to-peer connected communication state. In an embodiment, at block 910, when the selective forwarding unit condition is not met, the first client computing device 14a / 200a may update the communication state to a peer-to-peer connected communication state. The peer-to-peer communication state may be referred to as a base state, and may be a default state for the first client computing device. Thus, the communication state may be updated only if the first client computing device 14a / 200a is not in a base state to start with. In various embodiments, Figure 7 In decision block 704 of process 700, a change in the communication state identified by the participant state information may result in the state transition condition being satisfied. Thus, the communication state change of the first client computing device 14a / 200a may be based on Figure 7 The process 700 is communicated to other participants and their associated client computing devices via a peer-to-peer connection. However, in other embodiments, the participant state information may be distributed to participants of the coordinate grid environment via a state server computing device in a participant state payload 308a, as described above in Figure 4 As discussed in process 400.

[0121] Process 900 may proceed to block 912 where the first client computing device provides the media stream via one or more peer connections. In an embodiment, at block 912, the first client computing device 14a / 200a may provide the media stream to the second client computing device 14b / 200b via the peer connection through the communication system 210, or to other participants in the coordinate grid environment via the peer connection established between the first client computing device 14a / 200a and other client computing devices.

[0122] Blocks 914 to 924 of process 900 are described from the perspective of the second client computing device 14b / 200b. However, the functionality of the second client computing device 14b / 200b may be performed by the first client computing device 14a / 200a, and vice versa. At decision block 914, the second client computing device may monitor participant state information. In various embodiments, at decision block 914, the second client computing device 14b / 200b may monitor for participant state information from a state server computing device or another client computing device such as the first client computing device 14a / 200a. If participant state information is not received, the second client computing device 14b / 200b may continue to monitor for participant state information. Decision block 914 may operate similarly to decision block 708 of process 700 discussed above. If participant state information is not received, process 900 may return to decision block 914.

[0123] However, if participant state information has been received at decision block 914, process 900 may proceed to block 916 (which may be similar to block 710 of process 700 discussed above) where the stored participant state information is updated with any more recent participant state information. In various embodiments, at block 916, the second client computing device 14b / 200b may determine whether to update any of its participant state information 216c. The second client computing device 14b / 200b may compare each participant state information entry for each participant it has in the participant state information 216c with the participant state information it received via a peer connection or a state server computing device. For example, the second client computing device 14b / 200b may compare the participant entry for the first participant in the participant state information 216c with the participant state information it received for the first participant from the first client computing device 14a / 200a when the first client computing device 14a / 200a updated its communication state. The second client computing device 14b / 200b may compare the timestamps to determine which participant state entry is the most recent. If the received participant state information is more recent than the stored participant state information, the more recent participant state information may be used to update the portion of the participant state information 216c maintained by the second client computing device 14b / 200b. Thus, if the second client computing device 14b / 200b has received a more recent communication state of the first client computing device 14a / 200a than the second client computing device 14b / 200b has in the participant state information 216c, the second client computing device 14b / 200b may update the participant state information 216c with the new state (e.g., a basic state, a broadcast state, a screen sharing state, or any other communication state).

[0124] The process 900 may proceed to block 918, where the second client computing device may monitor one or more communication channels for media streaming based on the communication state of each participant in the coordinate grid environment. In an embodiment, at block 918, the second client computing device 14b / 200b may monitor the communication channel as indicated by the communication state included in the participant state information 216c. For example, when the communication state of the first client computing device 14a / 200a is in a basic state, the second client computing device 14b / 200b may monitor the peer-to-peer connection between the first client computing device 14a / 200a and the second client computing device 14b / 200b for media streaming. In another embodiment, when the communication state of the first client computing device 14a / 200a is in a screen sharing state or a broadcast state, the second client computing device 14b / 200b may monitor the communication channel with the selective forwarding unit 18 for media streaming from the first client computing device 14b / 200b. In some embodiments, when the communication state of the first client computing device 14a / 200a is in a broadcast state or a screen sharing state, the second client computing device 14b / 200b may block the media stream from the first client computing device 14a / 200a.

[0125] Process 900 may proceed to decision block 920 where it is determined whether a media stream is detected by the second client computing device. If a media stream is not detected, the process may return to block 918.

[0126] However, if at decision block 920, a media stream is detected at a communication channel being monitored by the second client computing device, process 900 may proceed to block 922, where the media stream is received via the communication channel indicated by the communication state of this participant. In an embodiment, at block 922, the second client computing device 14b / 200b may receive the media stream according to the communication state of each participant it has. For example, if the communication state of the participants in the coordinate grid environment associated with the first client computing device 14a / 200a indicates a broadcast state or a screen sharing state, the second client computing device 14b / 200b may receive the media stream via a communication channel in communication with the selective forwarding unit 18. If the communication state of the participants in the coordinate grid environment associated with the first client computing device 14a / 200a indicates a basic state, the second client computing device 14b / 200b may receive the media stream via a peer-to-peer connection between the first client computing device 14a / 200a and the second client computing device 14b / 200b.

[0127] Process 900 may proceed to block 924 where the media stream is rendered or output by the second client computing device. In an embodiment, at block 924, the second client computing device 14b / 200b may output the media stream via the display system 220 or the user I / O system 218. For example, an audio stream may be provided via a speaker included on the user I / O system 218. In another example, a video stream may be provided via a display included in the display system 220. However, other media streams may be output via other output devices or combinations of output devices included in the display system 220 or the user I / O system 218 that are apparent to those skilled in the art having the present disclosure.

[0128] Fig.10 1002a, 1002b, 1002c, 1002d, 1002e, and 1002f. In addition, in this example, participant 2 is in screen sharing state, participant 4 is in broadcast state, and participant 1 and 3 are in basic state. This means that participant 2 and participant 4 are streaming their audio, video or other media content to SFU 1004, as indicated by arrow 1006 and arrow 1008, respectively. Participant 2 and participant 4 do not stream their media streams on the peer connections 1002a to 1002f that exist between them and the other three participants.

[0129] However, Participant 2 and Participant 4 each download media streams from Participants 1 and 3 from peer connections 1002a, 1002c, 1002d, and 1002f to these participants. Peer connection 1006e between Participant 2 and Participant 4 has no bytes flowing in both directions. Participants 1 and 3 each upload and download media streams between each other on peer connection 1002b. Participants 1 and 3 also upload their media streams to 1002a, 1002c, 1002d, and 1002f.

[0130] In various embodiments, participants 1, 3, and 4, which consume streams from participant 2, download these streams from SFU 1004 as indicated by arrows 1010, 1012, and 1014, respectively. Participants 1, 2, and 3 consume streams from participant 4 as indicated by arrows 1016, 1018, and 1020, respectively.

[0131] When Participant 2 or Participant 4 stops broadcasting or screen sharing, they will stop uploading streams to SFU 1004 and instead upload streams to the peer connection 1002a, 1002c, 1002d, 1002e, or 1002f they have established. If Participant 1 or Participant 3 were to start broadcasting or screen sharing, they would stop uploading their streams to peer connections 1002a, 1002b, 1002c, 1002d, and 1002f and instead upload to SFU 1004. All other participants in range would then retrieve their new streams from SFU 1004.

[0132] Thus, the systems and methods of the present disclosure provide an environment that operates in a peer-to-peer manner for normal proximity interactions, but still utilizes the SFU when higher quality is required (screen sharing) or a larger number of stream participants must access one participant's stream (broadcast) or a stream is required by non-local participants.

[0133] Peer-to-peer mesh network bitrate throttling

[0134] As discussed herein, in a large cluster application grid environment, a separate mesh network is used to connect participants that are close enough to merit direct peer connections. As groups of peer-connected participants of the grid environment connect, a mesh network forms. If these participants stream streams like video between each other, then processing and bandwidth consumption increases as each new participant is added to the grid. Since each participant in a grid or peer pair has different computing and bandwidth capabilities, it is important to be able to throttle the maximum bandwidth between each participant as the network size grows and shrinks so that the quality of the stream is balanced with the quality of the overall experience.

[0135] Therefore, as the size of the mesh network grows, the processing and bandwidth requirements to maintain the mesh also grow. As more participants join the mesh, the strain on each individual participant's client computing device and the network increases. Each participant must assess their capabilities and throttle their output to maintain all necessary connections.

[0136] In various embodiments of the present disclosure, bit rate throttling can be performed when participants enter and exit any given peer-to-peer mesh network. Each participant knows how many other participants they are currently connected to. For example, if participant A and participant B are connected to each other, they enter a mesh network of two participants. Since they are the only two participants, each of them may stream high quality. Specifically, participants A and B can stream video at a high quality of 1MB / s. If participant C enters the range of both A and B, the mesh will develop into a network of three participants. Participant C can detect the peer connection with two other participants. Participant C can keep its maximum bandwidth consumption below 1MB / s, and therefore limit the bit rate to both participant A and participant B to 500kb / s. Participant A and participant B will detect that another participant has entered the mesh and can limit the bit rate of participant C to 500kb / s. They will also renegotiate the connection between them and limit the bit rate to 500kb / s. After this adjustment is complete, there is now a peer-to-peer mesh network with three participants, each streaming to each other at a maximum of 500kb / s, and no participant individually consuming more than 1MB / s. If participant B were to leave the mesh, participant A and participant C could renegotiate their connection and raise the cap to 1MB / s so that quality can improve again.

[0137] In some embodiments, the server 12, the relay server 16, the client 14, or the selective forwarding unit server 18 may operate, for example, by executing program code stored on a tangible, non-transitory machine-readable medium. Fig.11 The process 1100 of . The operations shown may be performed in a different order, omitted, duplicated, performed concurrently, performed serially, insert additional operations, fully automated, involve human intervention, or otherwise modified relative to the depicted arrangement, which is not meant to be limiting of any other description herein.

[0138] In some embodiments, process 1100 Fig.11. Process 1100 may begin at decision block 1102, where a determination is made as to whether a peer connection condition is met for initiating a peer connection between client computing devices associated with a participant. In an embodiment, at decision block 1102, a peer application 204c of a first client computing device 14a / 200a may determine whether a condition is met for establishing a peer connection with a second client computing device 14b / 200b. For example, the peer application 204c may determine whether a participant associated with the second client computing device 14b / 200b is adjacent to a participant associated with the first client computing device 14a / 200a (e.g., within a predetermined distance, or a distance determined based on the congestion of the coordinate grid environment). The peer application 204c may determine the distance between the participant of the client computing device 14a / 200a and other participants in the coordinate grid environment provided by the peer application 204c. To do this, the peer application 204c may reference the method described above by the client computing device 14a / 200a. Figure 4 The coordinate grid state 216b is determined by the process 400. Although the peer connection condition may be based on proximity or a predetermined distance, other peer connection conditions are also contemplated.

[0139] Process 1100 may proceed to box 1104, where a maximum bit rate for a peer connection with a participant is determined. In an embodiment, at box 1104, a peer application controller 204 may determine a maximum bit rate for a peer connection. The maximum bit rate for a peer connection may be based on the number of peer connections to which the first client computing device 14a / 200a is connected and the maximum bit rate or bandwidth of the first client computing device 14a / 200a. An administrator may set a maximum bit rate for a coordinate grid environment so that a client computing device may have an actual maximum bit rate greater than the set maximum bit rate. Specifically, the maximum bit rate for a peer connection may be calculated as a function of the maximum bit rate of all peer connections established by the client computing device and the number of connections. In one example, the bit rate for a peer connection may be the maximum bit rate for a client computing device divided by the number of peer connections. However, in other embodiments, as discussed below, it may be determined based on the maximum bit rate of the connected client computing device. For example, if participant A has two peer connections and a maximum bitrate of 1 mbps, and participant B has four peer connections and a maximum bitrate of 1 mbps, then the connection between participant B and participant A will be limited to participant B's per-connection bitrate of 250 kbps. Thus, participant A's first peer connection with, for example, participant C may have a bitrate of 750 kbps, while the peer connection between participant A and participant B will be at 250 kbps, even though participant A may have a bitrate of 500 kbps with participant B. In yet other embodiments, the maximum bitrate for the peer connections may be calculated based on other factors (e.g., the priority of a particular participant, attributes in the participant status information, or any other factor).

[0140] Process 1100 may proceed to block 1106, where a peer initiation signal including a maximum bit rate for the connection is generated. In an embodiment, at block 1106, the first client computing device 14a / 200a may generate a peer connection initiation signal that announces the maximum bit rate that the first client computing device 14a / 200a must propose to the second client computing device 14b / 200b with which the peer connection will be established. The peer connection initiation signal may include a bandwidth or maximum bit rate entry or announcement. Continuing with the WebRTC example herein, the SDP offer packet provided by the first client computing device 14a / 200a to the second client computing device 14b / 200b to initiate the peer connection may include a bandwidth entry that sets the maximum bit rate for the peer connection. Typically, the SDP offer maintains a bandwidth that allows WebRTC to consume as much bandwidth as possible. The peer application 204c of the present disclosure may change the content of the SDP offer so that the maximum bit rate for this peer connection is announced in the bandwidth entry.

[0141] Process 1100 may proceed to block 1108 where the first client computing device sends a peer connection initiation signal to the second client computing device, and the second client computing device receives the peer connection initiation signal block 1110. In various embodiments, the first client computing device 14a / 200a may determine that it will be the initiator of the peer connection with the second client computing device 14b / 200b. For example, the first client computing device 14a / 200a may determine that it is the initiator based on its participant identifier and the participant identifier associated with the second client computing device 14b / 200b. Specifically, the first client computing device 14a / 200a may determine that its participant identifier is greater than the participant identifier associated with the second client computing device 14b / 200b, which satisfies the participant identifier condition. However, in other embodiments, the participant identifier assigned the lowest participant identifier may be the initiator of the peer connection.

[0142] Thus, the first client computer device 14a / 200a sends a peer connection initiation signal and a maximum bit rate announcement for the peer connection to the second client computer device 14b / 200b. For example, the peer application 204c can communicate with a real-time communication (RTC) application programming interface (API) (e.g., a WebRTC API). The WebRTC protocol may include an interactive connectivity establishment (ICE) protocol for establishing a peer connection. Thus, the peer initiation signal may include an SDP proposal including a maximum bit rate or bandwidth announcement according to a session description protocol (SDP). More specifically, the SDP protocol may include a video bandwidth announcement. In various embodiments, the peer initiation signal is provided via the network 20 and the relay server 16. For example, the peer initiation signal is provided by a persistent connection (e.g., a web socket) established between the first client computing device 14a / 200a and the relay server 16 and a persistent connection established between the second client computing device 14b / 200b and the relay server 16. The second client computing device 14b / 200b may then receive the peer initiation signal.

[0143] Process 1100 may proceed to block 1112 where the second client computing device determines a maximum bit rate for the peer connection. In an embodiment, at block 1112, the second client computing device 14b / 200b may determine its maximum bit rate for the peer connection. The second client computing device 14b / 200b may determine its maximum bit rate based on the operations performed by the first client computing device 14a / 200a described at block 1104, and therefore will not be repeated in detail. In various embodiments, if the maximum bit rate determined by the second client computing device 14b / 200b is greater than the maximum bit rate advertised in the peer initiation signal, the second client computing device 14b / 200b may determine its maximum bit rate for the peer connection to be the maximum bit rate advertised in the peer initiation signal.

[0144] The process 1100 may then proceed to block 1114, where the second client computing device generates a peer connection response signal using the maximum bit rate announcement determined by it. In an embodiment, at block 1114, the second client computing device 14b / 200b may generate a peer connection response signal announcing the maximum bit rate that the second client computing device 14a / 200a must propose to the first client computing device 14a / 200a with which the peer connection will be established. The peer connection response signal may include a bandwidth or maximum bit rate entry or announcement. Continuing with the WebRTC example here, the SDP answer packet provided by the second client computing device 14b / 200b in response to the SDP offer to the first client computing device 14a / 200a may include a bandwidth entry that sets the maximum bit rate of the peer connection. Typically, the SDP answer maintains a bandwidth that allows WebRTC to consume as much bandwidth as possible. The peer application 204c of the present disclosure may change the content of the SDP answer so that the maximum bit rate for this peer connection is announced in the bandwidth entry.

[0145] Process 1100 may proceed to block 1116, where the second client computing device sends a peer connection response signal. In an embodiment, at block 1116, the second client computing device 14b / 200b may respond to the peer connection initiation signal via the relay server 16 with a peer connection response signal to the first client computing device 14a / 200a. At block 1118, the first client computing device 14a / 200a may receive the peer connection response signal and process the response signal to establish a peer connection (e.g., a real-time connection using WebRTC). At block 1120, one or more media streams (e.g., video streams, audio streams, audiovisual streams, or other media content streams) may then be provided over the peer connection between the first client computing device 14a / 200a and the second client computing device 14b / 200b, such that the media streams do not exceed the negotiated maximum bit rate.

[0146] The process 1100 may then proceed to a decision block 1122 where the first client computing device determines whether a bit rate renegotiation condition is satisfied on the peer connection. In an embodiment, at decision block 1122, the first client computing device 14a / 200a may determine whether a bit rate renegotiation condition exists. For example, the bit rate renegotiation condition may be satisfied when the first client computing device 14a / 200a determines that it will establish a subsequent peer connection with another client computing device such that bandwidth needs to be released. In another example, the bit rate renegotiation condition may be satisfied when another peer connection on the first client computing device 14a / 200a is removed such that additional bandwidth may be used on the peer connection with the second client computing device 14b / 200b. In another embodiment, the bit rate renegotiation condition may be satisfied when a renegotiation signal is received from the second client computing device. In other embodiments, other bit rate renegotiation conditions that are apparent to one of ordinary skill in the art having the present disclosure may be monitored.

[0147] If the bitrate renegotiation condition does not exist, then at box 1120, process 1100 can continue to provide the media stream via the peer connection at a bitrate that does not exceed the maximum bitrate. If the bitrate renegotiation condition is met at determination box 1122, the first client computing device 14a / 200a can renegotiate the maximum bitrate with the second client computing device 14b / 200b. For example, at box 1104, the first client computing device 14a / 200a can determine the maximum bitrate for the peer connection and provide the updated maximum bitrate to the second client computing device 14b / 200b to renegotiate and update the maximum bitrate on both client computing devices. For example, when a connection needs to be negotiated over a signaling channel, a "negotiationneeded" event can be sent to a peer connection (e.g., RTCPeerConnection). This may occur during the initial establishment of a peer connection and at any time when a change in the communication environment requires reconfiguration of the peer connection. Although decision block 1122 is described from the perspective of the first client computing device 14a / 200a, the second client computing device 14b / 200b may also monitor for a bit rate renegotiation signal.

[0148] Fig. 12A 1 is a coordinate grid environment 1200 with a participant 1202 and a participant 1204. The participant 1202 and the participant 1204 may establish a peer connection 1206 when they are in proximity to each other, such that a peer connection initiation condition is satisfied, which may include: Figure 5Participant 1202 and participant 1204 may negotiate a maximum bit rate of 1 Mbps, which may be a limit that any client computing device may have in a coordinate grid environment, even if the client computing device is configured for a higher bit rate. Fig. 12B As shown, participant 1208 can enter coordinate grid environment 1200, enabling the participant to establish peer connection 1210 with participant 1202 and peer connection 1212 with participant 1204. The maximum bit rate for peer connection 1206 can be renegotiated to 500 kbps. In addition, peer connections 1210 and 1212 can be throttled to 500 kbps. Although the example shows that the bit rate is halved when throttling, other allocations of bit rates can be implemented. Fig. 12C shows that participant 1204 has left Fig. 12B 1204 and 1208 in the mesh network, so that peer connections 1206 and 1212 no longer exist. Participants 1202 and 1208 can renegotiate the maximum bit rate on peer connection 1210 to 1 mbps. As will be understood by those skilled in the art having the present disclosure and these examples, as various participants enter and exit the mesh network and form various mesh topologies, peer connections can be renegotiated throughout the mesh network based on the number of participants and the topology of the mesh network.

[0149] Therefore, the system and method of the present disclosure provides peer connection bit rate throttling. When establishing a peer connection, the client computing device announces the determined maximum bit rate and negotiates the maximum bit rate for this peer connection. As participants in a coordinate grid environment enter and exit a grid network including a peer connection, the maximum bit rate for this connection can be re-determined and renegotiated by the client computing device. Thus, as the network size grows and shrinks, the bit rate throttling of the present disclosure allows the quality of the media stream to be balanced with the quality of the overall experience.

[0150] This patent submission is one of a group of three patent applications filed on the same day by the same applicant, the group members having the following titles: Peer-to-Peer Signal Caching System; Hybrid Peer-to-Peer Meshing and Forwarding System; Client Authoritative State Distribution System. The entire contents of each of the other patent submissions are incorporated herein by reference.

[0151] Fig.131300 is a diagram illustrating an exemplary computing system 1300 according to an embodiment of the present technology. Portions of the systems and methods described herein may include or execute on one or more computer systems similar to computing system 1300. For example, client computing device 14 / 200, server computing device 12 / 300, relay server 16, or selective forwarding unit 18 may be provided by computing system 1300. Further, the processes and modules described herein may be executed by one or more processing systems similar to one or more processing systems of computing system 1300.

[0152] The computing system 1300 may include one or more processors (e.g., processors 1310a to 1310n) coupled to a system memory 1320, an input / output I / O device interface 1330, and a network interface 1340 via an input / output (I / O) interface 1350. The processor may include a single processor or multiple processors (e.g., a distributed processor). The processor may be any suitable processor capable of running or otherwise executing instructions. The processor may include a central processing unit (CPU) that implements program instructions to perform arithmetic, logic, and input / output operations of the computing system 1300. The processor may execute code (e.g., processor firmware, protocol stack, database management system, operating system, or a combination thereof) that creates an execution environment for program instructions. The processor may include a programmable processor. The processor may include a general-purpose or special-purpose microprocessor. The processor may receive instructions and data from a memory (e.g., system memory 1320). The computing system 1300 may be a single processor system including one processor (e.g., processor 1310a), or a multiprocessor system including any number of suitable processors (e.g., 1310a to 1310n). Multiple processors may be used to provide parallel or sequential execution of one or more portions of the techniques described herein. The processes described herein, such as logic flows, may be performed by one or more programmable processors running one or more computer programs to perform functions by operating on input data and generating corresponding outputs. The processes described herein may be performed by dedicated logic circuitry, and the apparatus may also be implemented as a dedicated logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The computing system 1300 may include multiple computing devices (e.g., a distributed computer system) to implement various processing functions.

[0153] I / O device interface 1330 may provide an interface for connecting one or more I / O devices 1360 to computer system 1300. I / O devices may include devices that receive input (e.g., from a user) or output information (e.g., to a user). I / O devices 1360 may include, for example, a graphical user interface presented on a display (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor), a pointing device (e.g., a computer mouse or trackball), a keyboard, a keypad, a touchpad, a scanning device, a voice recognition device, a gesture recognition device, a printer, an audio speaker, a microphone, a camera, etc. I / O devices 1360 may be connected to computer system 1300 via a wired or wireless connection. I / O devices 1360 may be connected to computer system 1300 from a remote location. For example, I / O devices 1360 located on a remote computer system may be connected to computer system 1300 via a network and network interface 1340.

[0154] The network interface 1340 may include a network adapter that provides a connection of the computer system 1300 to a network. The network interface 1340 may facilitate data exchange between the computer system 1300 and other devices connected to the network. The network interface 1340 may support wired or wireless communications. The network may include an electronic communication network such as the Internet, a local area network (LAN), a wide area network (WAN), a cellular communication network, etc.

[0155] System memory 1320 may be configured to store program instructions 1301 or data 1302. Program instructions 1301 may be executed by a processor (e.g., one or more processors 1310a to 1310n) to implement one or more embodiments of the present technology. Instructions 1301 may include computer program instruction modules for implementing one or more technologies described herein for various processing modules. Program instructions may include computer programs (which are referred to as programs, software, software applications, scripts, or codes in some forms). Computer programs may be written in programming languages, including compiled or interpreted languages, or declarative or procedural languages. Computer programs may include units suitable for use in a computing environment, including as independent programs, modules, components, or subroutines. Computer programs may or may not correspond to files in a file system. Programs may be stored in a portion of a file that stores other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files storing one or more modules, subroutines, or code portions). A computer program can be deployed to be executed on one or more computer processors that are locally located at one site or distributed across multiple remote sites and interconnected by a communication network.

[0156] The system memory 1320 may include a tangible program carrier on which program instructions are stored. The tangible program carrier may include a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include a machine-readable storage device, a machine-readable storage substrate, a memory device, or any combination thereof. The non-transitory computer-readable storage medium may include a non-volatile memory (e.g., flash memory, ROM, PROM, EPROM, EEPROM memory), a volatile memory (e.g., a random access memory (RAM), a static random access memory (SRAM), a synchronous dynamic RAM (SDRAM)), a mass storage memory (e.g., a CD-ROM and / or DVD-ROM, a hard disk drive), etc. The system memory 1320 may include a non-transitory computer-readable storage medium on which program instructions that can be executed by a computer processor (e.g., one or more of the processors 1310a to 1310n) to cause the subject and functional operations described herein may be stored. The memory (e.g., the system memory 1320) may include a single memory device and / or multiple memory devices (e.g., a distributed memory device). Instructions or other program codes that provide the functions described herein may be stored on a tangible, non-transitory computer-readable medium. In some cases, the entire instruction set may be stored on the medium at the same time, or in some cases, different portions of the instructions may be stored on the same medium at different times.

[0157] The I / O interface 1350 may be configured to coordinate I / O traffic between the processors 1310a to 1310n, the system memory 1320, the network interface 1340, the I / O devices 1360, and / or other peripheral devices. The I / O interface 1350 may perform protocol, timing, or other data transformations to convert data signals from one component (e.g., the system memory 1320) into a format suitable for use by another component (e.g., the processors 1310a to 1310n). The I / O interface 1350 may include support for devices attached via various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard.

[0158] Embodiments of the techniques described herein may be implemented using a single instance of computer system 1300 or multiple computer systems 1300 configured to host different portions or instances of the embodiments. Multiple computer systems 1300 may provide for parallel or sequential processing / execution of one or more portions of the techniques described herein.

[0159] Those skilled in the art will appreciate that computer system 1300 is merely illustrative and is not intended to limit the scope of the technology described herein. Computer system 1300 may include any combination of devices or software that can perform or otherwise provide the performance of the technology described herein. For example, computer system 1300 may include a cloud computing system, a data center, a server rack, a server, a virtual server, a desktop computer, a laptop computer, a tablet computer, a server device, a client device, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a car computer or a global positioning system (GPS), etc., or a combination thereof. Computer system 1300 may also be connected to other devices not shown, or may be operated as an independent system. In addition, in some embodiments, the functions provided by the components shown may be combined in fewer components or distributed in additional components. Similarly, in some embodiments, some of the functions of the components shown may not be provided, or other additional functions may be available.

[0160] Those skilled in the art will also understand that, although various items are shown as being stored in memory or on a storage device when in use, for the purpose of memory management and data integrity, these items or a portion of them may be transmitted between memory and other storage devices. Alternatively, in other embodiments, some or all of the software components may be executed in a memory on another device and communicated with the computer system shown via inter-computer communication. Some or all of the system components or data structures may also be stored (e.g., as instructions or structured data) on a computer-accessible medium or portable article to be read by an appropriate drive, various examples of which are described above. In some embodiments, instructions stored on a computer-accessible medium separated from the computer system 1300 may be transmitted to the computer system 1300 via a transmission medium or signal (such as an electrical, electromagnetic or digital signal), transmitted via a communication medium such as a network or wireless link. Various embodiments may also include receiving, sending or storing instructions or data implemented according to the foregoing description on a computer-accessible medium. Therefore, the present technology may be practiced using other computer system configurations.

[0161] In the block diagram, the components shown are depicted as discrete functional blocks, but the embodiments are not limited to the system in which the functions described herein are organized as shown. The functions provided by each of the components may be provided by software or hardware modules that are organized differently from the current description, for example, such software or hardware may be mixed, combined, copied, decomposed, distributed (e.g., distributed in a data center or geographically distributed), or otherwise organized differently. The functions described herein may be provided by one or more processors of one or more computers executing code stored on a tangible, non-transitory machine-readable medium. In some cases, although the singular term "medium" is used, the instructions may be distributed on different storage devices associated with different computing devices, such as where each computing device has a different subset of instructions, which is an implementation consistent with the use of the singular term "medium" herein. In some cases, a third-party content delivery network may host some or all of the information transmitted over the network, in which case, to the extent that the information (e.g., content) is supplied or otherwise provided, the information may be provided by sending instructions to retrieve the information from the content delivery network.

[0162] The reader should understand that this application describes several independently useful technologies. Instead of separating these technologies into multiple independent patent applications, the applicant groups these technologies into a single file because their related themes are useful for the economy in the application process. However, the different advantages and aspects of these technologies should not be lumped together. In some cases, the embodiments solve all the defects mentioned herein, but it should be understood that these technologies are independently useful, and some embodiments only solve a subset of these problems or provide other unmentioned benefits that are obvious to those skilled in the art who read this disclosure. Due to cost constraints, some of the technologies disclosed herein may not be claimed for protection at present, but may be claimed in later applications, such as continued applications or by amending the present claims. Similarly, due to space limitations, the abstract and content of the invention of the invention section of this document should not be regarded as a comprehensive list containing all these technologies or all aspects of these technologies.

[0163] It should be understood that the description and the drawings are not intended to limit the technology of the present invention to the specific form disclosed, but on the contrary, the present invention is intended to cover all modifications, equivalents and alternatives that fall within the spirit and scope of the technology of the present invention as defined by the appended claims. In view of this description, further modifications and alternative embodiments of various aspects of the technology of the present invention will be apparent to those skilled in the art. Therefore, this description and the drawings are interpreted as illustrative only, and are for the purpose of teaching those skilled in the art to implement the general manner of the technology of the present invention. It should be understood that the forms of the technology of the present invention shown and described herein will be regarded as examples of embodiments. Elements and materials may replace those shown and described herein, parts and processes may be reversed or omitted, and certain features of the technology may be used independently, all of which are apparent to those skilled in the art who benefit from the description of the technology of the present invention. The elements described herein may be changed without departing from the spirit and scope of the technology described in the appended claims. The titles used herein are for organizational purposes only and are not meant to limit the scope of the description.

[0164] As used throughout this application, the word "may" is used in a permissive sense (i.e., meaning likely), rather than in a mandatory sense (i.e., meaning must). The words "include, including, includes" and the like mean including, but not limited to. As used throughout this application, the singular forms "a", "an", and "the" include plural referents unless the context clearly indicates otherwise. Thus, for example, reference to "an element" includes a combination of two or more elements, even though other terms and phrases, such as "one or more", are used for one or more elements. Unless otherwise indicated, the term "or" is non-exclusive, i.e., includes "and" and "or". Terms describing conditional relationships, such as "in response to X, Y", "at X, Y", "if X, Y", "when X, Y", etc., encompass such causal relationships, where the antecedent is a necessary causal condition, the antecedent is a sufficient causal condition, or the antecedent is a contributing causal condition of the result, for example, "state X occurs when condition Y is obtained" is a generic term of "X only occurs when Y" and "X occurs when Y and Z". This conditional relationship is not limited to the results obtained directly following the antecedent condition, as some results may be delayed, and in the conditional statement, the antecedent conditions are connected to their results, for example, the antecedent conditions are related to the probability of the result occurring. A statement that multiple attributes or functions are mapped to multiple objects (e.g., one or more processors perform steps A, B, C, and D) encompasses all of these attributes or functions being mapped to all of these objects, as well as subsets of attributes or functions being mapped to subsets of attributes or functions (e.g., all processors each perform steps A to D, and where processor 1 performs step A, processor 2 performs step B and a portion of step C, and processor 3 performs a portion of step C and step D), unless otherwise specified. Similarly, references to a "computer system" that performs step A and a "computer system" that performs step B may include the same computing device within a computer system that performs both steps or different computing devices within a computer system that performs steps A and B. Further, unless otherwise specified, a statement that a value or action is "based on" another condition or value encompasses both the case where the condition or value is the only factor and the case where the condition or value is one of multiple factors. Unless otherwise indicated, a statement that "each" instance of some set has some property should not be read as excluding the case where some identical or similar members of the larger set do not have that property, that is, each does not necessarily mean every. Limitations regarding the sequence of recited steps should not be read as a claim unless expressly specified, such as by using explicit language such as "after performing X, performing Y", as opposed to statements that might be inappropriately construed to imply a sequence limitation (such as "perform X on the item, perform Y on the item after X") for the purpose of making the claim more readable, rather than prescribing a sequence.Statements referring to "at least Z of A, B, and C," etc. (e.g., "at least Z of A, B, or C") refer to at least Z of the listed categories (A, B, and C), and do not require at least Z elements in each category. Unless otherwise stated, it will be apparent from the discussion that it should be understood that throughout this specification, terms such as "processing," "computing or calculating," "determining," etc., are used to refer to the action or process of a specific device, such as a special purpose computer or similar special purpose electronic processing / computing equipment. Features described with reference to geometric configurations (such as "parallel," "perpendicular / orthogonal," "square," "cylindrical," etc.) should be interpreted to encompass terms that substantially embody the properties of the geometric configuration, for example, a reference to "parallel" surfaces encompasses substantially parallel surfaces. Reference is made to the ranges in the specification to determine the permissible range of deviations from the Platonic ideal of these geometric constructions, and in the absence of reference to industry norms in the field of use to describe these ranges, and in the absence of reference to industry norms in the field of manufacture of the specified features, these ranges are not defined, and in the absence of these ranges, the features that substantially embody the geometric construction should be interpreted as including those features within 15% of the defined attributes of the geometric construction. The terms "first", "second", "third", "given", etc. (if used in the claims) are used to distinguish or otherwise identify, rather than to show sequential or numerical limitations. As is the case in ordinary use in the art, data structures and formats described with reference to uses that are significant to humans do not need to be presented in a human-understandable format to constitute the described data structures or formats, for example, text does not need to be presented or even encoded in Unicode or ASCII to constitute text; images, maps, and data visualizations do not need to be displayed or decoded to constitute images, maps, and data visualizations, respectively; speech, music, and other audio do not need to be emitted through speakers or decoded to constitute speech, music, or other audio, respectively. Computer-implemented instructions, commands, etc. are not limited to executable code, and can be implemented in the form of data that causes functions to be called, for example, in the form of arguments to functions or API calls. To the extent that custom noun phrases (and other coined terms) are used in the claims and lack a self-explanatory construction, the definition of such phrases may be recited in the claims themselves, in which case the use of such custom noun phrases should not be construed to impart additional limitations by inspection of the specification or extrinsic evidence.

[0165] In this patent, to the extent that any U.S. patent, U.S. patent application, or other material (e.g., article) is incorporated herein by reference, the text of such material is incorporated herein by reference only to the extent that there is no conflict between such material and the statements and drawings set forth herein. If there is such a conflict, the text of this document controls, and terms in this document should not be given a narrower reading due to the way those terms are used in other materials incorporated by reference.

[0166] The present technology will be better understood with reference to the following examples:

[0167] 1. A non-transitory machine-readable medium storing instructions that, when executed by one or more processors, implement operations including: establishing, using a first client computing device, a first peer-to-peer connection between the first client computing device and a second client computing device, wherein the first client computing device is associated with a first participant in a coordinate grid environment provided by an application, and the second client computing device is associated with a second participant in the coordinate grid environment; determining, using the first client computing device, that a coordinate grid state of the first participant has undergone a state change; and providing, using the first client computing device and in response to the state change to the coordinate grid state satisfying a state change condition, at least a portion of first participant state information associated with the state change of the first participant from the first client computing device to the second client computing device via the first peer-to-peer connection, wherein at least a portion of the first participant state information associated with the state change of the first participant updates previously stored first participant state information of the first participant maintained by a second participant associated with the second client computing device.

[0168] 2. A medium as described in Example 1, wherein the operations further include: receiving, using the first client computing device, second participant state information from the second client computing device via the first peer connection, wherein the second participant state information is associated with a second state change of the second participant; and updating, using the first client computing device, the previously stored second participant state information using any of the more recent second participant state information.

[0169] 3. The medium of embodiment 2, wherein the operations further comprise: causing, using the first client computing device, an update of a coordinate grid environment displayed via a graphical user interface provided on a first display on the first client computing device based on the second participant state information.

[0170] 4. A medium as described in any of embodiments 1 to 3, wherein the operations further include: receiving, by the first client computing device, a peer connection initiation signal to establish a first peer connection with a second client computing device; determining, by the first client computing device, that a peer connection condition exists for the first client computing device, such that the first client computing device can be used to establish a first peer connection with the second client computing device; processing, by the first client computing device, the peer connection initiation signal; and sending, by the first client computing device, a peer connection response signal to the peer connection initiation signal to the second client computing device, wherein the peer connection response signal enables the first client computing device and the second client computing device to establish the first peer connection.

[0171] 5. The medium of embodiment 4, wherein the operations further include: obtaining, by the first client computing device, a peer connection initiation signal from a signal cache associated with the first client computing device, wherein when the first client computing device determines that the peer connection condition is not met, the peer connection initiation signal is stored in the signal cache.

[0172] 6. A medium as described in any of Examples 1 to 5, wherein the first participant state information includes first participant location information indicating the location of the first participant in the coordinate grid environment, wherein the operations also include: providing the participant location information to the coordinate grid state server computing device by the first client computing device; and receiving a participant location payload from the coordinate grid state server computing device by the first client computing device, the participant location payload including participant location information including the first participant location information for at least a portion of a plurality of participants in the coordinate grid environment.

[0173] 7. A medium as described in any of Examples 1 to 6, wherein the operations further include: receiving third participant state information from a third client computing device via a second peer connection using the first client computing device, wherein the third participant state information is associated with a third state change of a third participant in a coordinate grid environment; and updating previously stored third participant state information using the third participant state information using the first client computing device.

[0174] 8. A medium as described in Example 7, wherein the operations further include: receiving, using the first client computing device, fourth participant state information from a third client computing device via a second peer connection, wherein the fourth participant state information is associated with a fourth state change for a fourth participant in a coordinate grid environment and associated with the fourth client computing device; and updating, using the first client computing device, previously stored fourth participant state information using the fourth participant state information.

[0175] 9. A medium as described in any of embodiments 1 to 8, wherein the operations also include: receiving third participant state information from a second client computing device via a first peer connection using a first client computing device, wherein the third participant state information is associated with a third participant in a coordinate grid environment; determining, by the first client computing device and based on a timestamp included in the third participant state information and a timestamp in previously stored third participant state information maintained by the first client computing device, that the third participant state information received from the second client computing device is not the most recent third participant state information; and ignoring, by the first client computing device, the third participant state information received via the first peer connection.

[0176] 10. A medium as described in any of embodiments 1 to 9, wherein the operations also include: receiving user input by a first client computing device to move a first participant associated with the first client computing device and at a first location in a coordinate grid environment provided by an application to a second location; sending participant location information for the first participant to a state server computing device by the first client computing device, wherein the participant location information includes the first location, the second location, and a time of the user input; and providing the participant location information in first participant state information provided to a second client computing device via a first peer connection by the first client computing device.

[0177] 11. A medium as described in any of embodiments 1 to 10, wherein the operations further include: using the first client computing device to provide, via a first peer-to-peer connection, to a second client computing device participant state information for participants in a coordinate grid environment known to the first client computing device maintained by the first client computing device, wherein at least a portion of the participant state information updates previously stored participant state information for participants in the coordinate grid environment maintained by a second participant associated with the second client computing device.

[0178] 12. A non-transitory machine-readable medium storing instructions that, when executed by one or more processors, implement operations including: establishing, using a first client computing device, a first peer-to-peer connection between the first client computing device and a second client computing device, wherein the first client computing device is associated with a first participant in a coordinate grid environment provided by an application and the second client computing device is associated with a second participant in the coordinate grid environment; receiving, using the first client computing device, second participant status information from the second client computing device via the first peer-to-peer connection, wherein the second participant status information is associated with a status change of the second participant; and updating, using the first client computing device, with the second participant status information, previously stored second participant status information.

[0179] 13. The medium of embodiment 12, wherein the operations further comprise: causing, using the first client computing device, an update of a coordinate grid environment displayed via a graphical user interface provided on a first display on the first client computing device based on the second participant state information.

[0180] 14. A medium as described in any of Examples 12 to 13, wherein the operations further include: determining, using a first client computing device, that a coordinate grid state of a first participant has undergone a state change; and providing, using the first client computing device and in response to the state change of the coordinate grid state satisfying a state change condition, at least a portion of first participant state information associated with the state change of the first participant from the first client computing device to a second client computing device via a first peer connection, wherein at least a portion of the first participant state information associated with the state change of the first participant updates previously stored first participant state information of the first participant maintained by a second participant associated with the second client computing device.

[0181] 15. A medium as described in Example 14, wherein the operations further include: using the first client computing device to provide, via the first peer-to-peer connection, to a second client computing device participant state information for participants in a coordinate grid environment known to the first client computing device maintained by the first client computing device, wherein at least a portion of the participant state information updates previously stored participant state information for participants in the coordinate grid environment maintained by a second participant associated with the second client computing device.

[0182] 16. A medium as described in any of embodiments 12 to 15, wherein the operations further include: receiving, by the first client computing device, a peer connection initiation signal to establish a first peer connection with a second client computing device; determining, by the first client computing device, that a peer connection condition exists for the first client computing device, such that the first client computing device can be used to establish a first peer connection with the second client computing device; processing, by the first client computing device, the peer connection initiation signal; and sending, by the first client computing device, a peer connection response signal to the peer connection initiation signal to the second client computing device, wherein the peer connection response signal enables the first client computing device and the second client computing device to establish a first peer connection.

[0183] 17. The medium of embodiment 16, wherein the operations further include: obtaining, by the first client computing device, a peer connection initiation signal from a signal cache associated with the first client computing device, wherein when the first client computing device determines that the peer connection condition is not met, the peer connection initiation signal is stored in the signal cache.

[0184] 18. A medium as described in any of Examples 12 to 17, wherein the second participant state information includes participant location information indicating the location of the second participant in the coordinate grid environment, wherein the operations also include: receiving, by the first client computing device, a participant location payload from a coordinate grid state server computing device, the participant location payload including participant location information for at least a portion of a plurality of participants in the coordinate grid environment, including at least the second participant state information.

[0185] 19. A medium as described in any of Embodiments 12 to 18, wherein the operations further include: receiving third participant state information from a third client computing device via a second peer connection using the first client computing device, wherein the third participant state information is associated with a third state change of a third participant in a coordinate grid environment; and updating previously stored third participant state information using the third participant state information using the first client computing device.

[0186] 20. A medium as described in Example 19, wherein the operations further include: receiving, using the first client computing device, fourth participant state information from a third client computing device via a second peer connection, wherein the fourth participant state information is associated with a fourth state change for a fourth participant in a coordinate grid environment and associated with the fourth client computing device; and updating previously stored fourth participant state information using the fourth participant state information using the first client computing device.

[0187] 21. A system comprising: one or more processors; and one or more computer-readable media storing instructions, which, when executed by the one or more processors, implement operations including the following: operations according to any one of embodiments 1 to 11.

[0188] 22. A method comprising: performing an operation according to any one of embodiments 1 to 11.

[0189] 23. A system comprising: one or more processors; and one or more computer-readable media storing instructions, which, when executed by the one or more processors, implement operations including the following: operations according to any one of embodiments 12 to 20.

[0190] 24. A method comprising: performing an operation according to any one of embodiments 12 to 20.

Claims

1. A non-transitory machine-readable medium storing instructions, wherein the instructions, when executed by one or more processors, perform the following operations: Establishing, using a first client computing device, a first peer-to-peer connection between the first client computing device and a second client computing device, wherein: The first client computing device is associated with a first participant in a coordinate grid environment provided by an application and displayed via a graphical user interface, and the second client computing device is associated with a second participant in the coordinate grid environment; determining, using the first client computing device, that a coordinate grid state of the first participant has undergone a state change; providing, using the first client computing device and in response to the state change to the coordinate grid state satisfying a state change condition, at least a portion of first participant state information associated with the state change of the first participant from the first client computing device to the second client computing device via the first peer-to-peer connection, wherein the at least a portion of the first participant state information associated with the state change of the first participant updates any previously stored first participant state information of the first participant maintained by the second participant associated with the second client computing device; and providing, using the first client computing device to the second client computing device via the first peer-to-peer connection, at least a portion of other participant state information maintained by the first client computing device for other participants in the coordinate grid environment known to the first client computing device, wherein the at least a portion of the participant state information updates any previously stored other participant state information of any other participant of the coordinate grid environment maintained by the second participant associated with the second client computing device, and Wherein, at least one of the first participant status information or the other participant status information includes participant position information indicating a position of a corresponding participant in the coordinate grid environment.

2. The medium according to claim 1, wherein The operations also include: receiving, with the first client computing device, second participant state information from the second client computing device via the first peer-to-peer connection, wherein the second participant state information is associated with a second state change of the second participant; and The previously stored second participant state information is updated with the first client computing device with any of the more recent second participant state information.

3. The medium according to claim 2, wherein: The operations also include: Causing, with the first client computing device, an update of the coordinate grid environment displayed via the graphical user interface provided on a first display on the first client computing device based on the second participant state information.

4. The medium according to claim 1, wherein The operations also include: receiving, by the first client computing device, a peer connection initiation signal to establish the first peer connection with the second client computing device; determining, by the first client computing device, that a peer connection condition exists for the first client computing device such that the first client computing device is operable to establish the first peer connection with the second client computing device; processing, by the first client computing device, the peer connection initiation signal; and The first client computing device sends a peer connection response signal to the second client computing device in response to the peer connection initiation signal, wherein the peer connection response signal enables the first client computing device and the second client computing device to establish the first peer connection.

5. The medium according to claim 4, wherein The operations also include: The peer connection initiation signal is obtained by the first client computing device from a signal cache associated with the first client computing device, wherein the peer connection initiation signal is stored in the signal cache when the first client computing device determines that the peer connection condition is not satisfied.

6. The medium according to claim 1, wherein The first participant status information includes first participant location information indicating a location of the first participant in the coordinate grid environment, wherein the operations further include: providing, by the first client computing device, the participant location information to a coordinate grid state server computing device; and A participant location payload is received by the first client computing device from the coordinate grid state server computing device, the participant location payload comprising participant location information including the first participant location information for at least a portion of a plurality of participants in the coordinate grid environment.

7. The medium according to claim 1, wherein The operations also include: receiving, with the first client computing device, third participant state information from a third client computing device via a second peer-to-peer connection, wherein the third participant state information is associated with a third state change for a third participant in the coordinate grid environment; and The previously stored third participant state information is updated using the first client computing device using the third participant state information.

8. The medium according to claim 7, wherein The operations also include: receiving, with the first client computing device, fourth participant state information from the third client computing device via the second peer-to-peer connection, wherein the fourth participant state information is associated with a fourth state change for a fourth participant in the coordinate grid environment and associated with a fourth client computing device; and The fourth participant state information is used to update previously stored fourth participant state information using the first client computing device.

9. The medium according to claim 1, wherein The operations also include: receiving, using the first client computing device and via the first peer-to-peer connection, third participant state information from the second client computing device, wherein the third participant state information is associated with a third participant in the coordinate grid environment; determining, by the first client computing device and based on a timestamp included in the third participant state information and a timestamp in previously stored third participant state information maintained by the first client computing device, that the third participant state information received from the second client computing device is not the most recent third participant state information; and The third participant state information received via the first peer-to-peer connection is ignored by the first client computing device.

10. The medium according to claim 1, wherein The operations also include: receiving, by the first client computing device, a user input to move the first participant associated with the first client computing device and at a first location in the coordinate grid environment provided by the application to a second location; sending, by the first client computing device, participant location information for the first participant to a state server computing device, wherein the participant location information includes the first location, the second location, and a time input by the user; and The participant location information is provided by the first client computing device in the first participant state information provided to the second client computing device via the first peer-to-peer connection.

11. A non-transitory machine-readable medium storing instructions that, when executed by one or more processors, perform operations comprising: Establishing, using a first client computing device, a first peer-to-peer connection between the first client computing device and a second client computing device, wherein: The first client computing device is associated with a first participant in a coordinate grid environment provided by an application and displayed via a graphical user interface, and the second client computing device is associated with a second participant in the coordinate grid environment; receiving, with the first client computing device, second participant state information from the second client computing device via the first peer-to-peer connection, wherein the second participant state information is associated with a state change of the second participant; updating, using the first client computing device, any previously stored second participant state information using the second participant state information; receiving, with the first client computing device, from the second client computing device via the first peer-to-peer connection, at least a portion of other participant state information maintained by the second client computing device for other participants in the coordinate grid environment known to the second client computing device; and updating, by the first client computing device, previously stored other participant state information of other participants of the coordinate grid environment maintained by the first participant associated with the first client computing device using the at least a portion of the other participant state information, wherein at least one of the second participant state information or the other participant state information includes participant position information indicating a position of the participant in the coordinate grid environment.

12. The medium according to claim 11, wherein The operations also include: Causing, with the first client computing device, an update of the coordinate grid environment displayed via the graphical user interface provided on a first display on the first client computing device based on the second participant state information.

13. The medium according to claim 11, wherein The operations also include: determining, using the first client computing device, that a coordinate grid state of the first participant has undergone a state change; and Utilizing the first client computing device and in response to a state change to the coordinate grid state satisfying a state change condition, providing at least a portion of first participant state information associated with the state change of the first participant from the first client computing device to the second client computing device via the first peer-to-peer connection, wherein the at least a portion of the first participant state information associated with the state change of the first participant updates previously stored first participant state information of the first participant maintained by the second participant associated with the second client computing device.

14. The medium according to claim 13, wherein The operations also include: providing, by the first client computing device, to the second client computing device via the first peer-to-peer connection, at least a portion of other participant state information maintained by the first client computing device for other participants in the coordinate grid environment known to the first client computing device, wherein the at least a portion of the other participant state information updates any previously stored other participant state information for any of the other participants of the coordinate grid environment maintained by the second participant associated with the second client computing device.

15. The medium according to claim 11, wherein The operations also include: receiving, by the first client computing device, a peer connection initiation signal to establish the first peer connection with the second client computing device; determining, by the first client computing device, that a peer connection condition exists for the first client computing device such that the first client computing device is operable to establish the first peer connection with the second client computing device; processing, by the first client computing device, the peer connection initiation signal; and The first client computing device sends a peer connection response signal to the second client computing device in response to the peer connection initiation signal, wherein the peer connection response signal enables the first client computing device and the second client computing device to establish the first peer connection.

16. The medium according to claim 15, wherein The operations also include: The peer connection initiation signal is obtained by the first client computing device from a signal cache associated with the first client computing device, wherein the peer connection initiation signal is stored in the signal cache when the first client computing device determines that the peer connection condition is not satisfied.

17. The medium according to claim 11, wherein The second participant status information includes second participant location information indicating a location of the second participant in the coordinate grid environment, wherein the operations further include: A participant location payload is received by the first client computing device from a coordinate grid state server computing device, the participant location payload comprising participant location information for at least a portion of a plurality of participants in the coordinate grid environment including at least the second participant state information.

18. The medium according to claim 11, wherein The operations also include: receiving, with the first client computing device, third participant state information from a third client computing device via a second peer-to-peer connection, wherein the third participant state information is associated with a third state change for a third participant in the coordinate grid environment; and The previously stored third participant state information is updated using the first client computing device using the third participant state information.

19. The medium according to claim 18, wherein The operations also include: receiving, with the first client computing device, fourth participant state information from the third client computing device via the second peer-to-peer connection, wherein the fourth participant state information is associated with a fourth state change for a fourth participant in the coordinate grid environment and associated with a fourth client computing device; and The previously stored fourth participant state information is updated using the first client computing device using the fourth participant state information.

Images