Peer-to-peer signal buffering system

Abstract

A process includes: receiving a peer-to-peer connection initiation signal to establish a peer-to-peer connection with a second client computing device. The process further includes: determining that peer-to-peer connection conditions are not met, rendering a first client computing device unusable for establishing a peer-to-peer connection with the second client computing device; and storing the peer-to-peer connection initiation signal in a signal buffer associated with the first client computing device.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 288,435, filed December 10, 2021, entitled “Mesh Network Signal Buffering System”; U.S. Provisional Patent Application No. 63 / 391,652, filed July 22, 2022, entitled “Hybrid Peer-to-Peer Mesh and Streaming System”; and U.S. Provisional Patent Application No. 63 / 398,485, filed August 16, 2022, entitled “Authority Status Distribution”. The entire contents of each of the aforementioned patent applications are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to peer-to-peer mesh networks, and more specifically, to a peer signal caching system for caching signals used for proximity peer connections between participants. Background Technology

[0004] Many software applications can stream media such as video, audio, or other data between participants within the application. Video and audio streaming can enhance the user experience of an application by allowing for more personal interaction between participants within the application instance. For example, in some gaming applications, video and audio can be streamed between participants within the game instance. However, as the number of streams between a particular participant and other participants increases, the quality of audio and video streams degrades and may cause streaming sessions to fail.

[0005] Therefore, systems have been developed to limit the number of streams per participant in order to maintain acceptable streaming quality. One such system comprises a peer-to-peer mesh topology in which a subset of participants are interconnected for streaming video and / or audio. However, peer-to-peer mesh topologies present the problems discussed below. Summary of the Invention

[0006] The following is a non-exhaustive list of some aspects of this technology. These and other aspects are described in the following disclosures.

[0007] Some aspects include a process comprising: receiving a peering connection initiation signal to establish a peering connection with a second client computing device; determining that peering connection conditions are not met, such that a first client computing device cannot be used to establish a peering connection with the second client computing device; and storing the peering connection initiation signal in a signal buffer associated with the first client computing device.

[0008] Some aspects include a process comprising: a first client computing device determining that a proximity condition is met, wherein the proximity condition includes determining whether a first participant associated with the first client computing device and located at a first position in a coordinate grid environment provided by an application is within a distance of a second participant associated with a second client computing device and located at a second position in the coordinate grid environment; the first client computing device sending a first peer-to-peer connection initiation signal to establish a first peer-to-peer connection with the second client computing device in response to the proximity condition being met; and the first client computing device establishing the first peer-to-peer connection with the second client computing device in response to receiving a first response to the first peer-to-peer connection initiation signal.

[0009] Some aspects include tangible, non-transitory machine-readable media storing instructions that, when executed by a data processing device, cause the data processing device to perform operations including the processes mentioned above.

[0010] Some aspects include a system comprising: one or more processors; and a memory storing instructions, wherein when executed by the processor, these instructions cause the processor to perform the operations of the processes mentioned above. Attached Figure Description

[0011] The foregoing and other aspects of this technology will be better understood when this application is read with reference to the following figures, in which similar numbers represent similar or identical elements:

[0012] Figure 1 This illustrates a computing environment in which the present technology can be implemented according to some embodiments of the present disclosure;

[0013] Figure 2 This illustrates some embodiments according to the present disclosure. Figure 1 A block diagram illustrating an example of a client computing device within a computing environment;

[0014] Figure 3 This illustrates some embodiments according to the present disclosure. Figure 1 A block diagram illustrating an example of a server computing device in a computing environment;

[0015] Figure 4 The following are some embodiments of the invention illustrated in this disclosure. Figure 1 A flowchart illustrating the process of participant state distribution executed by the computing environment;

[0016] Figure 5 The following are some embodiments of the invention illustrated in this disclosure. Figure 1 A flowchart illustrating the peer-to-peer signal buffering process performed by the computing environment;

[0017] Figure 6Some embodiments according to this disclosure are shown. Figure 4 and Figure 5 A flowchart illustrating an example of the process;

[0018] Figure 7 The following are some embodiments of the invention illustrated in this disclosure. Figure 1 A flowchart illustrating the process by which a computing environment provides participant state information via a peer-to-peer connection.

[0019] Figure 8 Some embodiments according to this disclosure are shown. Figure 7 A flowchart illustrating an example of the process;

[0020] Figure 9 The following are some embodiments of the invention illustrated in this disclosure. Figure 1 A flowchart of the process of hybrid peer-to-peer and selective forwarding unit media streaming performed in a computing environment;

[0021] Figure 10 Some embodiments according to this disclosure are shown. Figure 9 A flowchart illustrating an example of the process;

[0022] Figure 11 The following are some embodiments of the invention illustrated in this disclosure. Figure 1 A flowchart illustrating the process of peer-to-peer connection bit rate throttling performed in the computing environment;

[0023] Figure 12A , Figure 12B and Figure 12C Some embodiments according to this disclosure are shown. Figure 11 A flowchart illustrating an example of the process;

[0024] Figure 13 An example computing device is shown that can implement the present technology according to some embodiments of the present disclosure.

[0025] Although this technology is readily available for various modifications and alternative forms, specific embodiments thereof are illustrated by way of example in the accompanying drawings and will be described in detail herein. The drawings may not be drawn to scale. However, it should be understood that the drawings and their detailed description are not intended to limit the technology to the specific forms disclosed, but rather, the invention is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the technology as defined by the appended claims. Detailed Implementation

[0026] To alleviate the problems described herein, the inventors must devise solutions and, in some cases, equally importantly, recognize problems that have been overlooked (or not yet foreseen) by others in the fields of media streaming, virtual coordinate grid environments, and peer-to-peer mesh networks. In fact, the inventors wish to emphasize the difficulty of recognizing these emerging problems, and that these problems will become even more apparent in the future if industry trends continue as the inventors anticipate. Furthermore, because multiple problems are addressed, it should be understood that some embodiments are specific to particular problems, and not all embodiments solve every problem of the conventional systems described herein or provide every benefit described herein. That is, improvements to various permutations for addressing these problems are described below.

[0027] As discussed above, the use of streaming media (e.g., video, audio, and / or other data) between participants is becoming increasingly common in application environments such as gaming environments, communication environments, social media environments, virtual reality environments, applications incorporating coordinate grids, or any other application environments that are obvious to those skilled in the art upon which this disclosure is known. As the clustering of application instances grows, reliably maintaining connections between subsets of participants becomes increasingly difficult. Typically, it is almost unnecessary to maintain streaming between all participants, but in many cases, proximity connections are highly valuable.

[0028] Some common streaming architectures include peer-to-peer (P2P) architecture (also known as mesh or P2P mesh), selective forwarding unit (SFU) architecture, multipoint conferencing unit (MCU) architecture, and experience delivery network (XDN) architecture.

[0029] P2P mesh topology is perhaps the easiest to set up and often the most cost-effective architecture for real-time communication applications, including WebRTC (Web Real-Time Communication). However, it is also the least scalable. In a P2P mesh topology, two or more peers (clients) communicate directly with each other, or, when on opposite sides of a firewall, via a relay server that relays audio, video, and other data streams to them.

[0030] P2P applications can be resource-intensive because the burden of encoding and decoding media streams is offloaded to each peer, which is why P2P performs best when there are only a small number of concurrent users (e.g., 10 or fewer). While some degree of scalability can be achieved on P2P applications by configuring a P2P mesh network, P2P applications are resource-intensive and inefficient. However, P2P applications provide optimal end-to-end encryption because they do not rely on a centralized server for encoding / decoding the stream.

[0031] Peer-to-peer mesh topologies can adequately connect subsets of application instance clusters, but quickly and accurately negotiating connections between two individual peers remains a challenge.

[0032] For these reasons, real-time communication application developers often rely on alternative architectures, such as the SFU architecture, which includes a direct-path routing system designed to offload some of the streaming processing from each client to a server. Each participant sends their encrypted media stream once to a central server, which then forwards these streams to other participants in the application without further processing. Other real-time communication applications use an MCU architecture where each client connects to a centralized MCU server, which decodes, rescales, and blends all incoming streams into a single new stream, and then encodes the signal stream and sends it to all clients. Still other real-time communication applications use the XDN architecture, which uses a cloud-based cluster architecture instead of a centralized server to address the scalability issues of WebRTC.

[0033] Most video games, or other coordinate grid-based applications involving real-time communication, utilize a central state server to resolve the accuracy of events—or aspects of realism—that each participant can agree on, and report these updates to the client. These updates can be for participant movement and other events (e.g., movement of objects in the environment, participant attributes such as health, strength, speed, or other participant attributes). The server typically confirms actions before reporting to the client, allowing the client to trust the updates provided. This is important for establishing agreements between participants regarding the occurrence of outcomes. However, confirming actions before reporting to the client requires additional processing on the server to resolve the results of interactions between participants. Thus, the MCU architecture provides both real-time communication and realism requirements. Limitations exist regarding the scalability of such systems, making the trade-off between accuracy in participant location and the efficiency of the central state server crucial as the virtual world population grows. As the number of participants increases, the consensus becomes a world-scale limiting factor. Increasing processing power and network bandwidth are needed to distribute these changes and updates to all other participants. The sheer number of confirmed updates controlled by a single server or process limits the number of participants in a world (such as a specific coordinate grid instance). However, some state-based or coordinate grid-based applications do not require real-time consensus among participants. Therefore, a centrally controlled state server is unnecessary. In such cases, a P2P mesh topology is ideal, eliminating the need for additional hardware in the State Function Unit (SFU). Distributing this work among customers is crucial for building services capable of accommodating hundreds, thousands, or even tens of thousands of participants simultaneously.

[0034] Peer signal caching and server authority state management

[0035] Embodiments of this disclosure provide an application platform in which each participant can determine its own state or the coordinate grid state of its position relative to other participants within a coordinate grid-based environment currently provided by the application. The only important fact is each participant's position relative to other participants. If two participants perceive a slight difference in their relative positioning to each other (e.g., in actual physical locations or within a virtual application environment), it has no impact on the application platform. This is ultimately resolved when the participants stop moving.

[0036] In distributing state management to participants, the application platform implements a highly scalable server. The server's sole function is to receive user input events, such as click events or other user input from participants. User input captures the participant's current location in the virtual environment (e.g., coordinates in a coordinate grid, such as a 3D or 2D grid), the user-input location (e.g., coordinates in a coordinate grid where the user instructs the participant (e.g., an avatar) to move within the virtual environment), and the time of the user input. The central server can obtain these three data points and rewrite any existing location descriptions or states of the moving participant. This is the participant's current state (e.g., location description) and will remain so until a new movement instruction is received and distributed. This payload is distributed to all participants in the application platform, completing the server's work. This implementation incurs less network overhead compared to the scenario where the server determines all participant states itself. Each client computing device associated with a participant in the application environment's coordinate grid is then responsible for updating the positions of (n) other participants at whatever frame rate each client can run.

[0037] Distributing state computations alleviates the processing required by the central server to perform O(N) updates per frame to all locations of all participants in the world and then transmit all increments to all participants every frame. This significantly reduces processing overhead and network bandwidth across the system and allows thousands of participants to engage in a single world.

[0038] However, distributing state introduces other issues regarding peer-to-peer architectures. Connecting two or more participants in a single hall or peer-to-peer connection requires relaying an "offer" signal from the "initiator" to the "receiver" who will respond with a "response" signal. If these participants correctly exchange P2P connection initiation signals, a peer-to-peer connection (such as a WebRTC connection) can be opened between them.

[0039] However, there are challenges in transmitting these exchanged P2P connection initiation signals back and forth between the two participants. For example, the first challenge is the need for a relay server. This is typically accomplished using WebSockets, which establish persistent connections from each participant to the server node. Each initiating participant can send its P2P connection initiation signal to the relay server, which then relays the signal to the corresponding receiving participant. The second challenge is the ability of the "receiving" participant to respond with a "response" signal after receiving a "proposal" signal from the "initiator." In some scenarios, the "receiving" participant may be unavailable due to an inability or unwillingness to connect. Several potential reasons exist for the "receiving" participant to be in this unavailable state. One reason could be that the "receiving" participant has exhausted its available connection slots and has not processed any incoming "proposals."

[0040] Another reason could be that the "receiver" is unaware of the "initiator's" connection intent (e.g., the participants haven't reached a consensus on their states). As discussed above, distributed state management introduces new problems in peer-to-peer networks, where participants can determine they will connect to each other asynchronously. In applications without a centrally controlled state server, because each participant controls its own state, each participant can conclude that they are close enough to connect to another participant at different times via WebRTC or other peer-to-peer protocols. For example, participant A (i.e., the "initiator") can determine they are within the connection radius of participant B (i.e., the "receiver"), while participant B may only confirm this fact some time later. Therefore, there is a possibility of missing peer-to-peer connection initiation signals (such as a WebRTC "proposal" to the "receiver") and peer-to-peer connection response signals (such as a WebRTC "response" to the "initiator"), making the P2P architecture less reliable.

[0041] Therefore, another embodiment of this disclosure addresses these problems by including a local signal buffer for each participant, such that the client computing device associated with that participant stores any peer connection initiation signals. The "receiver" should use the incoming "proposal" signal to do one of two things. If the "receiver" is ready to process the "proposal" because it knows of the "initiator's" valid incoming connection attempt, then it should process the "proposal" and respond with a "response" (e.g., both participants agree). If the "receiver" cannot process the incoming "proposal" upon receiving it from the "initiator," then the "receiver" should store these incoming "proposals" in its local signal buffer. The local signal buffer allocated to the "initiator" that sent the "proposal" holds the "proposal" for later processing.

[0042] If the receiver knows the valid conditions for connecting to the initiator and its local signal buffer is filled with a valid proposal from the initiator, the receiver can process the proposal and immediately send a response without triggering the generation of a new proposal. In this case, the initiator can still wait for the receiver's response to process. If the initiator has already moved and determines that the receiver is no longer "approaching" the initiator, there is nothing more to do, as the initiator will no longer accept responses.

[0043] Figure 1An example of a computing environment 10 in which the present technology may be implemented is shown. In some embodiments, computing environment 10 is a distributed computing environment implementing a client / server architecture, although other architectures are also conceivable, including a monolithic architecture running on a single computing device. In some embodiments, computing environment 10 includes a server 12, client computing devices 14, a relay server 16, a selective forwarding unit 18, and a network 20 (such as the Internet) through which these components can communicate.

[0044] In some embodiments, the client computing device 14 is a desktop computer, laptop computer, in-store kiosk, tablet computer, mobile phone, head-mounted display, game console, set-top box, or any other computing device that runs an operating system and a web browser or a native application in which the described user interface is presented, as will be obvious to those skilled in the art. Three client computing devices 14 are shown, but embodiments can support far more concurrent sessions, such as more than 100 or more than 1,000 sessions geographically distributed in the United States or around the world.

[0045] In some embodiments, server 12 is a non-blocking web server or application interface server configured to serve multiple concurrent sessions (e.g., implementing a model-view-controller architecture or other design) with different client computing devices 14. In some embodiments, server 12 may dynamically generate assets, markup language instructions, and scripting language instructions in response to requests from client computing devices 14 to send user interfaces to or update user interfaces on those client computing devices 14. The user interface may evolve over time (e.g., in web applications), and in some cases, new resources (e.g., images and other data) sent from server 12 may be displayed in response to user input on the user interface.

[0046] The configuration engine and rendering engine included on the server can be used to generate image files and metadata used by the server 12 to generate user interfaces, such as providing a virtual two-dimensional or virtual three-dimensional coordinate grid environment, augmented reality environment, virtual reality environment, metaverse environment, or any other coordinate grid environment or application environment that is obvious to those skilled in the art who have mastered and benefited from the teachings of this disclosure.

[0047] In some embodiments, relay server 16 is a server configured to create websocket or other persistent connections with each client 14. Each client 14 can then send its initiation and response signals via relay server 16 to establish a real-time communication session (e.g., a WebRTC connection). Selective forwarding unit 18 may be a server configured to receive media from participants and determine which media streams should be forwarded to other participants and then perform the forwarding. Although server 12, relay server 16, and selective forwarding unit 18 are shown as separate servers, various embodiments of this disclosure may include a single server providing application, relay, or media forwarding functionality for server 12, relay server 16, or selective forwarding unit 18.

[0048] Figure 2 An embodiment of a client computing device 200 is shown, which may be the above-described reference. Figure 1 Any of the client computing devices 14 discussed. In the illustrated embodiment, client computing device 200 includes a chassis 202 housing the components of client computing device 200. Several of these components are in Figure 2 As shown in the diagram. For example, chassis 202 may house a processing system (not shown) and a non-transitory memory system (not shown) including instructions that, when run by the processing system, cause the processing system to provide a peer-to-peer (P2P) application controller 204, which is configured to perform the functions of a peer-to-peer application controller or client computing device discussed below. Figure 2 In the specific example shown, the peer application controller 204 is configured to provide a local application or browser application (e.g., a peer application), such as an internet web browser, to communicate with application instances operating on server 12 or relay server 16. The peer application controller 204 may be configured to invoke a peer application programming interface (API) (e.g., the WebRTC API) or other peer programs to facilitate peer communication. The peer application controller 204 may also include a signal caching engine 204a configured to cache peer signals (e.g., initiator "proposals"), determine the existence of peer connection conditions based on coordinate grid state or other information, or other functions discussed in further detail below. The peer application controller 204 may also include a coordinate grid state engine 204b configured to capture participant or object position change inputs or other participant state changes in the environment, send position changes to a state server (e.g., server 12), receive participant state payloads for all participants in the coordinate grid environment, and determine the coordinate grid state of all participants in the environment based on the participant state payloads.

[0049] The chassis 202 may also house a communication system 210 coupled to the peer-to-peer application controller 204 (e.g., coupling between the communication system 210 and the processing system). The communication system 210 may include software or instructions stored on a computer-readable medium that allow client computing devices 200 to send and receive information via the communication networks discussed above. For example, the communication system 210 may include a communication interface 212 to provide communication over network 20 as detailed above (e.g., multiple first (e.g., remote) transceivers). In embodiments, the communication interface 212 may include a wireless antenna configured to provide communication using the IEEE 802.11 protocol (Wi-Fi), cellular communication, satellite communication, other microwave radio communication, and / or other communication. The communication system 210 may also include a communication interface 214 (e.g., multiple second (e.g., short-range) transceivers) configured to provide communication with… Figure 1 Other client computing devices 14, sensors, storage devices or the above reference Figure 1 Direct communication with other devices discussed. For example, communication interface 214 may include a wireless antenna configured to communicate with devices such as Bluetooth, Bluetooth Low Energy (BLE). Low Energy (BLE), Near Field Communication (NFC), Infrared Data Association (IrDA) It operates using the IEEE 802.11 protocol (Wi-Fi) and / or other wireless communication protocols that allow direct communication between devices.

[0050] Chassis 202 may house a storage device (not shown) providing storage system 216, which is coupled to peer application controller 204 via a processing system. Storage system 216 may be configured to store local signal buffer 216a (e.g., a buffer for peer-initiated signals) and coordinate grid state 216b, which stores one or more coordinate grid states indicating the current position of a participant in a coordinate grid environment provided by an application. The coordinate grid state may include a calculated coordinate grid state determined based on participant status information collected by server 12. The participant status information may include raw participant status information collected by client computing devices 14 / 200 or by server 12. While specific instructions and data are described as being stored in storage system 216, those skilled in the art will recognize that other instructions may be used to perform the functions described herein, as described below.

[0051] In various embodiments, chassis 202 also houses a user input / output (I / O) system 218 coupled to peer application controller 204 (e.g., via coupling between the processing system and user I / O system 218). In embodiments, user I / O system 218 may be provided by a keyboard input subsystem, mouse input subsystem, trackpad input subsystem, touch input display subsystem, microphone, audio system, haptic feedback system, and / or any other input subsystem. Chassis 202 also houses a display system 220 coupled to peer application controller 204 (e.g., via coupling between the processing system and display system 220) and may be included in user I / O system 218. In embodiments, display system 220 may be provided by a display device integrated into client computing device 200 and including a display screen (e.g., a display screen on a laptop / notebook computing device, tablet computing device, mobile phone, or wearable device), or by a display device directly coupled to client computing device 200 (e.g., a display device coupled to a desktop computing device via a wired or wireless connection). Although Figure 2 A specific client computing device 200 is shown, but those skilled in the art will recognize that the client computing device 200 may include more or fewer components and is still within the scope of this disclosure.

[0052] Figure 3 An embodiment of a server computing device 300 is shown, which may be the above-referenced Figure 1 The server 12 is discussed. However, in other embodiments, the server computing device 300 may be a selective forwarding unit 18 or a relay server 16. In the illustrated embodiment, the server computing device 300 includes a chassis 302 housing the components of the server computing device 300. Figure 3Only some of these components are shown. For example, chassis 302 may house a processing system (not shown) and a non-transitory memory system (not shown) including instructions that, when run by the processing system, cause the processing system to provide a peer-to-peer application controller 304, which is configured to perform the functions of the peer-to-peer application controller and / or server device discussed below. Peer-to-peer application controller 304 may be configured to provide a peer-to-peer application 304a that provides one or more coordinate grid environments to client 14 / 200, enabling client 14 / 200 to participate in peer-to-peer application 304a hosted by peer-to-peer application controller 304. In various embodiments, peer-to-peer application controller 304 may include a coordinate grid state engine 304b that receives participant state information for each participant in each coordinate grid environment instance hosted by peer-to-peer application controller 304, compiles these participant states into a participant state payload 308a, and provides the participant state payload to various participants (e.g., client 14 / 200), as discussed in further detail below.

[0053] The chassis 302 may also house a communication system 306, which is coupled to the peer application controller 304 (e.g., via coupling between the communication system 306 and the processing system) and is configured to communicate via Figure 1 Network 20 provides communication, as detailed below. Communication system 306 allows server computing device 300 to communicate via... Figure 1 Network 20 sends and receives information. Chassis 302 may also house a storage device (not shown) that provides a storage system 308 coupled to the peer application controller 304 via the processing system. Storage system 308 may be configured to store participant state payload 308a or other data or instructions that are obvious to those skilled in the art. In various embodiments, storage system 308 may be located on server computing device 300 or on a database accessible via communication system 306. Furthermore, in some embodiments, server computing device 300 may host a relay engine (not shown) to perform the functions of relay server 16. In other embodiments, server computing device 300 may be a distributed server computing device in which functionality is performed by more than one server. Although in Figure 3 A specific server computing device 300 is shown, but those skilled in the art will recognize that the server computing device 300 may include more or fewer components and is still within the scope of this disclosure.

[0054] 1. Server authoritative coordinate grid status system

[0055] In some embodiments, server 12, relay server 16, or client 14 may be executed, for example, by running program code stored on a tangible, non-transitory, machine-readable medium. Figure 4 The process 400. The operations shown can be run in different orders, omitted, copied, run concurrently, run serially, insert additional operations, be fully automated, involve human intervention, or otherwise modified relative to the depicted arrangement, without implying that any other description herein is limiting.

[0056] In some embodiments, process 400 in Figure 4 As shown in the diagram. Process 400 may begin at decision box 402, where it is determined whether user input conditions are met. In an embodiment, at decision box 402, the client computing device 14 / 200, and in particular the coordinate grid state engine 204b, may monitor any user input from participants (e.g., avatars, objects) provided by applications (e.g., peer application 304a hosted by server computing device 12 / 300 or peer application 204c provided on client computing device 14 / 200) that instruct the application to change the state (e.g., move to a different location in the coordinate grid, change appearance or shape, or any other state change that is obvious to those skilled in the art who possess this disclosure) provided by the coordinate grid environment. User input can be a click on a location in the application's coordinate grid environment using a mouse on a graphical user interface, touch input on the touchscreen of the client computing device 14 / 200, input of coordinate commands, voice commands, gestures, or any other user input that can change the state of a participant in peer application 304a (such as their position in the coordinate grid environment provided by peer application 304a). If no user input affecting a change in a participant's state is detected, process 400 continues to monitor user input at decision box 402.

[0057] If user input is detected, process 400 proceeds to box 404, where participant state information is captured. In an embodiment, at box 404, coordinate grid state engine 204b may capture participant state information associated with a participant in the coordinate grid environment of peer applications 204c / 304a. For example, coordinate grid state engine 204b may capture participant location information associated with a participant. Participant location information may include the participant's current location in the coordinate grid environment (e.g., two-dimensional coordinates, three-dimensional coordinates, or any other information used to describe the participant's current location), the participant's user-input location (e.g., coordinates in the coordinate grid environment entered by the user from decision box 402 indicating where the participant moved), and the time of the participant's user input. While specific participant state information has been discussed, other state information that will be apparent to those skilled in the art to whom this disclosure is known may be used.

[0058] Process 400 can then proceed to block 406, where the client computing device provides participant status information to the server computing device. In an embodiment, at block 406, the peer-to-peer application controller 204 may provide the participant status information to the server computing device 12 / 300. For example, the peer-to-peer application controller 204 may use communication systems 210 and 306 to send the participant status information to the server 12 via network 20. The participant status information may include a client computing device identifier, an identifier of the application instance on which the participant is provided (e.g., in some embodiments, the client computing device 200 may enable multiple application instances, such as peer application 204c, to be open, such that each application instance can have different participants in the coordinate grid environment provided by peer application 204c), or any other identifier that associates a participant in the coordinate grid environment with the participant status information of that participant. The participant status information may also include participant location information, such as the participant's current location in the coordinate grid, the location in the coordinate grid identified by user input, and the time when the user input was received. However, the participant status information provided to the server 12 / 300 may include other participant status information.

[0059] The process can proceed to box 408, where the coordinate grid state is updated based on participant state information. In an embodiment, at box 408, the coordinate grid state engine 204b can update the coordinate grid state 216b based on participant state information, which may include participant location information of participants associated with the application instance operating on the peer application 204c. For example, coordinate grid state 216b may include the distance between the participant in the coordinate grid environment and those other participants based on their participant state information. For example, participant location information may include the participant's current location in the coordinate grid environment, the participant's user input location, and the time of that user input. The client computing device 14 / 200 may include participant state information 216c for all participants in the coordinate grid environment. As discussed in more detail with reference to box 416 below, the client computing device 14 / 200 may have previously received a participant state payload from the server computing device 12 / 300, which may include at least a portion of the participant state information for the participants in the coordinate grid environment. Participant state information for the participants in the client computing device 14 / 200 and participant state information 216c for the participants in the coordinate grid environment can be used to determine the distance between participants in the coordinate grid environment and other participants (e.g., using a distance formula), and this distance is stored as coordinate grid state 216b. Coordinate grid state 216b can be used by the client computing device 14 / 200 for other determinations, such as... Figure 5 As discussed in process 500.

[0060] Process 400 may include a decision box 410, where the server computing device determines whether it has received participant status information from the client computing device. In an embodiment, at decision box 410, the coordinate grid state engine 304b may determine whether any updated participant status information has been received. The server computing devices 12 / 300 may communicate with the client computing devices 14 / 200, which may provide participant status information for their respective participants. If not, the coordinate grid state engine 304b may continue monitoring for participant status information at decision box 410.

[0061] If the server computing device has received participant status information, such as participant status information from box 406, process 400 can proceed to box 412, where the server computing device updates the participant status payload. In an embodiment, at box 412, the coordinate grid state engine 304b can use any received participant status information to update the participant status payload 308a. As discussed above, participant status information may include participant location information, such as the participant's current location in the coordinate grid environment, the participant's user input location, the time of the user input, and an identifier for the participant or client computing device 14 / 200. The server computing device 12 / 300 may compile all participant status information for participants in the coordinate grid environment provided by the peer application 304a running on the peer application controller 304 into the participant status payload 308a. Thus, the server computing device 12 / 300 collects participant status for each participant, which may include three data points: the participant's current location in the coordinate grid environment, the participant's user input location, and the time of the user input. The server computing device 12 / 300 simply uses the new participant state information in the participant state payload 308a to update any old participant state information. The server computing device 12 / 300 may not need to compute the participant's coordinate grid state. As discussed above, this allows the coordinate grid environment to scale in terms of participants by reducing the processing power and network bandwidth required to distribute state changes and updates to all other participants. So many confirmed updates controlled by a single server or process limit the number of participants in a world (such as a specific instance of a coordinate grid environment).

[0062] Process 400 then proceeds to block 414, where a participant state payload is distributed to at least a portion of the participants in the coordinate grid environment. In an embodiment, at block 414, server computing devices 12 / 300 may distribute participant state payload 308a to at least a portion of client computing devices 14 / 200 in the coordinate grid environment. The distribution of participant state payload 308a may occur in response to an update of participant state payload 308a. As discussed above, the participant state payload may include participant state information for at least a portion of the participants in the coordinate grid environment. For example, participant state information may include participant location information, which includes the participant's current location in the coordinate grid environment, the participant's user-input location, and the time of the participant's user input. However, as discussed herein, participant state information may include many parameters that are obvious to those skilled in the art.

[0063] Process 400 then proceeds to block 416, where the client computing devices receive participant state payloads from the server computing devices. In an embodiment, at block 416, client computing devices 14 / 200 may receive participant state payload 308a from server computing devices 12 / 300. The participant state payload 308a can be received by at least a portion of the client computing devices 14 / 200 in computing environment 10 to compute their own coordinate grid states 216b.

[0064] Process 400 then proceeds to block 418, where the client computing device updates its coordinate grid state based on the participant state information included in the participant state payload. In an embodiment, at block 418, when comparing the participant state information received in the participant state payload with a previous version of the participant state information received in a previous participant state payload, the coordinate grid state engine 204b can determine whether there has been a change in the participant state information. The coordinate grid state engine 204b can update the participant state information 216c and then recalculate the coordinate grid state 216b. Thus, the processing of the coordinate grid state 216b is distributed, allowing each client computing device 14 / 200, rather than the server computing devices 12 / 300, to perform the computation. This reduces the use of processing and other resources of the server computing devices 12 / 300 and the communication requirements on network 20, allowing the coordinate grid environment to scale while still maintaining satisfactory consensus on the coordinate grid state among the participants.

[0065] As discussed above, when peer-to-peer communication initiators rely on proximity between participants, offloading coordinate grid state calculations to individual client computing devices introduces problems in peer-to-peer communication establishment. This is because the coordinate grid state can be calculated by the client computing devices at different times. Consequently, a client computing device that has already calculated the coordinate grid state and determined that another participant is within its range can initiate a peer connection without that other participant needing to confirm it can initiate a peer connection, may cause the other participant to ignore a peer connection initiation signal sent from that client computing device. Therefore, the systems and methods of this disclosure provide a signal buffer that allows client computing devices to store peer connection initiation signals until the client computing device updates its coordinate grid state and can determine that it meets the conditions for negotiating a peer connection.

[0066] 2. Peer-to-peer mesh network signal buffering

[0067] In some embodiments, server 12, relay server 16, or client 14 may be executed, for example, by running program code stored on a tangible, non-transitory, machine-readable medium. Figure 5The process 500. The operations shown can be run in different orders, omitted, copied, run concurrently, run serially, insert additional operations, be fully automated, involve human intervention, or otherwise modified relative to the depicted arrangement, without implying that any other description herein is limiting.

[0068] In some embodiments, process 500 in Figure 5 As shown in the diagram. Process 500 may begin at decision block 502, where it is determined whether the peering connection conditions for initiating a peering connection between client computing devices associated with participants are met. In an embodiment, at decision block 502, the peering application 204c of the first client computing device 14a / 200a may determine whether the conditions for establishing a peering connection with the second client computing device 14b / 200b are met. For example, peering application 204c may determine whether the participant associated with the second client computing device 14b / 200b is adjacent to the participant associated with the first client computing device 14a / 200a. Peering application 204c may determine the distance between the participant of client computing device 14a / 200a and other participants in the coordinate grid environment provided by peering application 204c. For this purpose, peering application 204c may reference the coordinate grid state 216b determined by client computing device 14a / 200a using process 400 described above. While peering conditions can be based on proximity or a predetermined distance, it is also conceivable that asynchronicity between participants may occur, resulting in other peering conditions where one participant is ready to peer while another is not. In another embodiment, peering conditions for initiating a peering connection may include user input for connecting a participant to another participant. Although some conditions for initiating peering connections have been discussed, those skilled in the art will recognize that other conditions may cause a client computing device to initiate a peering connection with another participant.

[0069] Process 500 may proceed to block 504, where the first client computing device sends a peer-to-peer connection initiation signal to the second client computing device, and the second client computing device receives the connection initiation signal at block 506. In various embodiments, the first client computing device 14a / 200a may determine that it will be the initiator of a peer-to-peer connection with the second client computing device 14b / 200b. For example, the first client computing device 14a / 200a may determine that it is the initiator based on its participant identifier and the participant identifier associated with the second client computing device 14b / 200b. Specifically, the first client computing device 14a / 200a may determine that its participant identifier is greater than the participant identifier associated with the second client computing device 14b / 200b, which satisfies the participant identifier condition. However, in other embodiments, the participant identifier assigned the lowest participant identifier may be the initiator of the peer-to-peer connection, or other information besides the participant identifier may be used to determine the initiator status.

[0070] Thus, the first client computer device 14a / 200a generates a peer-to-peer connection initiation signal and sends it to the second client computer device 14b / 200b. For example, the peer-to-peer application 204c can communicate with a Real-Time Communication (RTC) application programming interface (API) (e.g., the WebRTC API). The WebRTC protocol may include an interactive connectivity establishment (ICE) protocol for establishing peer-to-peer connections. Therefore, the peer-to-peer initiation signal may include an SDP proposal according to a Session Description Protocol (SDP) protocol. In various embodiments, the peer-to-peer initiation signal is provided via network 20 and relay server 16. For example, the peer-to-peer initiation signal is provided through a persistent connection (e.g., a web socket) established between the first client computing device 14a / 200a and relay server 16, and a persistent connection established between the second client computing device 14b / 200b and relay server 16. The second client computing device 14b / 200b can then receive the peer-to-peer initiation signal.

[0071] Process 500 can then proceed to decision block 508, where the second client computing device determines whether conditions for processing a peer-to-peer initiation signal are met. In an embodiment, at decision block 508, the second client computing device 14b / 200b can determine whether conditions for processing a peer-to-peer initiation signal exist. For example, the second client computing device 14b / 200b can determine whether it is available to process a peer-to-peer initiation signal. In some embodiments, the second client computing device 14b / 200b may be available when it determines that peer-to-peer connection conditions are met, such as proximity conditions with the first client computing device 14a / 200a. As discussed above, due to network connectivity issues arising from factors such as the maximum number of peer connections already established by the second client computing device 14b / 200b, differences in processing speed between the second client computing device 14b / 200b and the first client computing device 14a / 200a when determining coordinate grid state 216b, or any other reason why the first client computing device 14a / 200a determines that a peer connection with the second client computing device 14b / 200b should be established before the second client computing device 14b / 200b determines that a peer connection should be established, the second client computing device 14b / 200b may determine that this condition is met at a different time than the first client computing device 14a / 200a. For example, the number of peer connections may depend on whether the second client computing device 14b / 200b has any available peer connections. Specifically, each client computing device 14 / 200 may be assigned a maximum number of peer connections (e.g., 10 peer connections, 8 peer connections, and / or any other number of peer connections that are obvious to those skilled in the art possessing this disclosure). The number of peer connections can be based on the quality requirements of the media streams (audio streams, video streams, audiovisual streams, or other data streams) that will be provided via the peer connections.

[0072] In a conventional system, if the second client computing device 14b / 200b has determined that the peering conditions have not been met, the second client 14b / 200b can ignore the peering connection signal it receives. Consequently, the first client computing device 14a / 200a may be waiting for a response (e.g., an SDP reply to an SDP proposal) without receiving one each time.

[0073] If it is determined in decision block 508 that the second client computing device is unavailable, process 500 can proceed to block 510, where the peer connection initiation signal is stored in local signal cache 216a. In an embodiment, in block 510, relay server 16 or the second client computing device 14b / 200b can store the peer connection initiation signal in local signal cache 216a via signal cache engine 204a. For example, local signal cache 216a may include a data structure that includes an identifier of the initiator (e.g., the first client computing device 14a / 200a or a participant associated with the first client computing device 14a / 200a), the peer connection initiation signal, and the time the peer connection initiation signal was received. Thus, local signal cache 216a may include an entry for each initiator of the second client computing device 14b / 200b. If a peer-to-peer initiation signal is already associated with the initiator (e.g., the first client computing device 14a / 200a) in the local signal buffer 216a, the second client computing device 14b / 200b can update the peer-to-peer initiation signal using the latest peer-to-peer initiation signal and remove any previously received peer-to-peer initiation signals. The second client computing device 14b / 200b can then return to decision box 508 to continue monitoring for the existence of the peer-to-peer condition.

[0074] In various embodiments, relay server 16 may provide a local signal buffer 216a for each of the client computing devices 14. For example, relay server 16 may buffer peer-to-peer connection initiation signals and relay those signals between the first client computing device 14a / 200a and the second client computing device 14b / 200b when requested. If no signal is present in the server-side signal buffer, the participant / client may enter a waiting state in which either side will process the signal in real time while bypassing the server-side signal buffer.

[0075] In various embodiments of this disclosure, a receiving client (e.g., a second client computing device 14b / 200b) can determine that it is available to receive a peer initiation signal before the initiating client (e.g., a first client computing device 14a / 200a) sends a proposal signal. In this scenario, the second client computing device 14b / 200b can first check its local signal buffer 216a for the peer initiation signal and then wait for the peer initiation signal from the first client computing device 14a / 200a. When the peer initiation signal arrives, the second client computing device 14b / 200b can process the peer initiation signal immediately according to block 512 of process 500 without storing the proposal signal in its local signal buffer.

[0076] If, at decision block 508, it is determined that the peering condition has been met, process 500 can proceed to block 512, where a peering connection initiation signal is processed. In an embodiment, at block 512, the second client computing device 14b / 200b can process the peering connection initiation signal, and at block 514, it responds to the first client computing device 14a / 200a via the relay server 16 using a peering connection response signal. In various embodiments, once the second client computing device 14b / 200b becomes available, it can check its local signal buffer 216a to determine whether the initiator identifier associated with the first client computing device 14a / 200a includes a peering connection initiation signal, and if so, proceed to block 514. For example, when the peering condition has been met, the second client computing device 14b / 200b will know to listen for peering initiation signals from the first client computing device 14a / 200a, which can be identified by an identifier. Therefore, the second client computing device 14b / 200b checks whether any peer connection initiation signals have been cached in the local signal cache 216n, and processes these signals when peer conditions are met (e.g., within the proximity range of the client computing device associated with the cached signal).

[0077] In various embodiments, at block 514, the peer-to-peer response signal may include an SDP response or any other response signal that is obvious to those skilled in the art possessing this disclosure. At block 516, the first client computing device 14a / 200a may receive and process the peer-to-peer response signal to establish a peer-to-peer connection (e.g., a real-time connection using WebRTC). At block 518, one or more media streams (e.g., video streams, audio streams, AV streams, or other media content streams) may then be provided on the peer-to-peer connection between the first client computing device 14a / 200a and the second client computing device 14b / 200b.

[0078] In some embodiments, the first client computing device 14a / 200a can determine that the peer connection condition no longer exists while it is waiting for a peer connection response signal. Therefore, the first client computing device 14a / 200a can ignore the peer connection response signal received from the second client computing device 14a / 200a, or disconnect the peer connection after it has been established. In some embodiments, once a peer connection is established, it can be disconnected when the peer connection condition is not met (e.g., once participants move away from each other, causing the proximity value to no longer meet the proximity value condition).

[0079] Figure 6An example of a peer-to-peer signal caching system 600, which can be provided by computer system 10, is shown, along with examples of the processes 400 and 500 discussed above. Each participant 602, 604, and up to 606 can communicate location information to a coordinate grid state server 608 (e.g., server 12 / 300). The location information may include the current location, the next location input by the user, and the time input by the user. The coordinate grid state server 608 may store the location information of each participant 602 to 606 in a cache database 610 and provide the current participant state (location) payload from the cache database 610 to participants 602 to 606. The cache database 610 may require only a single data point to distribute information to each other participant via the coordinate grid state server 608. Given a constant velocity, each participant 602 to 606 can calculate the current position of each other participant based on a line drawn through two coordinates of the participant (e.g., participant 602 and participant 604) and the time input by the user.

[0080] Because each participant 602, 604, and 606 independently determines their coordinate grid state using a participant state payload provided by a coordinate grid state server 608 (e.g., server 12) that stores updatable participant state payloads in a cache database 610, participant 602 can determine that they are within the connectable distance of participant 604 faster than participant 604. This means that participant 602 can send the peer connection initiation signal via a peer signaling relay service (e.g., provided by relay server 16) before participant 604 is ready to consume the peer connection initiation signal. Participant 604 can store the peer connection initiation signal in its local signaling cache 604a. Similarly, if participant 602 or 606 receives the peer connection initiation signal before they are ready to consume it from the participant who sent the peer connection initiation signal, participant 602 can store the peer connection initiation signal in its local signaling cache 602a, and participant 606 can store the peer connection initiation signal in its local signaling cache 606a.

[0081] Therefore, the systems and methods of this disclosure provide peer caching when the coordinate grid state for participants in a coordinate grid environment is determined by each participant's associated computing device rather than a state server, and this coordinate grid state is used to determine whether a peer connection should be established between participants. As discussed above, participants can determine the coordinate grid state at different times, which may cause client computing devices to miss peer connection initiation signals (e.g., if a client computing device does not determine that it is close to another client computing device that has determined that the two client computing devices are adjacent to each other). This allows for mesh networks of coordinate grid environments with peer connections, such that the coordinate grid environment can be sized to accommodate hundreds, thousands, or tens of thousands of participants and allows them to communicate with each other when they are adjacent, while limiting the number of participants that can communicate with each other.

[0082] Client-authoritative coordinate grid state distributed via peer-to-peer connection

[0083] As discussed above, participant state information can be provided to the coordinate grid state server. The coordinate grid state server can compile at least a portion of the participant state information and distribute it in the participant state payload. Participant state information may include participant location information, based on which each participant determines its coordinate grid state within the coordinate grid. Participant location information may include the location of the participant, which can determine the distance between participants or the coordinate grid state, the user input location, and the time of user input. Distance can be used to determine whether a peer connection should be established by satisfying proximity conditions. Local signal buffering is used to cache peer connection initiation signals because participants may generate their own coordinate grid states at different times, allowing participants to facilitate peer connections. In addition to providing participant location information, other participant state information can be provided to the coordinate grid state server. However, continuously providing participant state information via the coordinate grid server can be inefficient in terms of network, processing, and memory resources.

[0084] In various embodiments of this disclosure, once participants in a coordinate grid environment provided by the application are connected to each other via peer-to-peer channels or connections, each participant can transmit participant state information or participant state changes to connected participants when something changes about their own state or when that participant connects to another new participant. The participant receiving the state change can relay this state change to other peer-to-peer participants. If another connected participant has already received a state change from the initial participant and has already received the relay message, the participant will ignore the relay message. In some embodiments, participants may ignore the relay message with the most hops, the last received relay message, or any other condition obvious to those skilled in the art.

[0085] In various embodiments, multiple independent participant meshes can be formed within a coordinate grid environment, where there are no peer connections between two independent participant meshes. When a new participant enters a mesh, this participant provides the participant state it is tracking to participants already in the mesh. Participants in the mesh can then provide the participant state they are tracking to the new participant in the mesh. Thus, the new participant and existing mesh participants merge their exchanged state databases. Existing mesh participants relay the new participant's participant state to participants in the meshes they are connected to, and so on, until all participants in the mesh have the new participant's participant state information.

[0086] In some embodiments, not all state changes are communicated to all participants. In some embodiments, a participant's state change may only affect one participant and not the others. For example, if participants are in a paintball game, a first participant can shoot a second participant. The shooting action by the first participant can provide a state change to the world. For example, the shooting action can generate a visualization in the paintball world, which would need to be communicated to all other participants in the grid so that these participants can see the paintball shot in the visualization. However, in other embodiments, the paintball shot may not be associated with a visualization. Thus, only the second participant who is shot and will receive a deduction in health or life as a result of the shot will need to receive the state change generated by the first participant. In response to being shot, the second participant may experience a state change of reduced health or life and generate state change communication to send to the connected participants. Thus, peer applications can be configured to communicate state changes based on the fulfillment of state change conditions (e.g., a global visualization associated with the state change or local knowledge only). Otherwise, in order to save network and processing resources, if there is no impact on other participants, the state change can be not transmitted, or the state change can be sent only to the first group of participants and not to the second group of participants, where the state change of the first group of participants affects the first group of participants but not the second group of participants.

[0087] In various embodiments, a participant maintains all participant states known to it, even if the participant or another participant disconnects from the mesh. A participant sends the participant states of other participants until it receives new participant states from those other participants. When a participant receives participant states, it checks the timestamp to determine if the received participant state is newer than its stored participant states. If the received participant state is newer, the participant updates the participant states of other participants using the received participant state. Otherwise, if the received participant state is older, the participant retains its already stored participant states.

[0088] In some embodiments, server 12, relay server 16, or client 14 may be executed, for example, by running program code stored on a tangible, non-transitory, machine-readable medium. Figure 7 The process 700. The operations shown can be run in different orders, omitted, copied, run concurrently, run serially, insert additional operations, be fully automated, involve human intervention, or otherwise modified relative to the depicted arrangement, without implying that any other description herein is limiting.

[0089] In some embodiments, process 700 in Figure 7As shown in the diagram. Process 700 may begin at block 702, where a peer-to-peer connection is established between a first client computing device and a second client device. In an embodiment, at block 702, a peer-to-peer connection may be established between the first client computing device 14a / 200a and the second client computing device 14b / 200b. The establishment of the peer-to-peer connection can be based on the above... Figure 4 or Figure 5 The peer connections are established using processes 400 and 500 described herein. However, in other embodiments, peer connections may be established based on other processes that are obvious to those skilled in the art upon which this disclosure is known. In various embodiments, the first client computing device 14a / 200a or the second client computing device 14b / 200b may each establish multiple peer connections with their respective client computing devices. Thus, for example, the first client computing device may establish a second peer connection with a third client computing device, a third peer connection with a fourth client computing device, a fourth peer connection with a fifth client computing device, and so on, until a maximum peer connection limit is reached, which may be based on performance, system definition, or client computing device capabilities.

[0090] Referring to the embodiments described in procedures 400 and 500, the establishment of a peer-to-peer connection can occur within a coordinate grid environment provided by peer application 204c or 304a, and when a first participant associated with a first client computing device 14a / 200a satisfies a proximity condition with a second participant associated with a second client computing device 14b / 200b in the coordinate grid environment. Thus, the state server computing device 12 / 300 can provide participant state information (e.g., participant location information) to each participant in the coordinate grid environment, and each participant can calculate its distance and location or other coordinate grid state based on this participant state payload. However, in other embodiments of this disclosure and those discussed in detail below, each participant may belong to the same mesh network of peer-to-peer participants. For example, the first client computing device 14a / 200a and the second client computing device 14b / 200b may be connected to each other via one or more intermediate client computing devices. Based on the shared participant status information provided by the peer-to-peer connection through the mesh network, the first client computing device 14a / 200a and the second client computing device 14b / 200b can obtain each other's participant location information and determine that they are adjacent to each other, so that a peer-to-peer connection can be established.

[0091] In some cases, at least the participant's position information, including their state, needs to be communicated to the state server computing device 12 / 300. For example, if the first participant on the first client computing device 14a / 200a joins the coordinate grid environment, other participants are unaware of the first participant's position within the coordinate grid environment, and the first participant is also unaware of the positions of other participants. In another case, multiple mesh networks can operate within the coordinate grid environment without intermediate participants. Consequently, as participants in each mesh move around the coordinate grid environment, those participants in each mesh network may not know the positions of participants in other mesh networks, making it difficult to determine the current position of each participant between mesh networks, thus hindering the satisfaction of proximity conditions for establishing peer-to-peer connections.

[0092] Process 700 can proceed to decision box 704, where it is determined whether a state transition condition exists. In an embodiment, at decision box 704, the first client computing device 14a / 200a can determine whether the state transition condition is met, causing the first client computing device 14a / 200a to initiate a transition of the stored participant state information 216c. The participant state information may include participant state information for a participant in the coordinate grid environment. As discussed above, the participant state information may include participant location information, but may also include other state information used to determine authenticity in the coordinate grid environment. For example, participant state information may include participant health, abilities, virtual items associated with the participant, speed, appearance, blockchain wallet identifier, token identifier, or other updatable attributes / states. In various embodiments, a state transition condition may be met when the first participant changes one or more states, causing the participant state information for the first participant to be updated. In another embodiment, a state transition condition may be met when the first client computing device 14a / 200a detects a state change or causes a state change in another participant in the coordinate grid environment. Detection of a state change may include receiving participant state information from another client computing device on a peer-to-peer connection with a client computing device that updates at least one data entry in participant state information 216c. If no state transition condition is met, in decision block 704, the first client computing device will continue to monitor for the state transition condition to be met.

[0093] If the state transition condition is met at decision block 704, process 700 can proceed to block 706, where at least a portion of the participant state information stored on the first client computing device 14a / 200a is provided on at least a portion of the peering connection already established by the first client computing device 14a / 200a. In embodiments, at block 706, the first client computing device 14a / 200a may provide at least a portion of its participant state information 216c to other client computing devices connected to the first client computing device 14a / 200a via the peering connection. For example, at least a portion of the participant state information 216c may be provided via a peering connection with the second client computing device 14b / 200b. However, in some embodiments, the participant state information may not be transitioned on a portion of the peering connection hosted by the first client computing device 14a / 200a. For example, if the reason for transmitting the participant state information is based on participant state information received via the peering connection, the participant state information may not be provided via the peering connection whose communication resulted in the transmission of the participant state information. This prevents the looping of state information via the peering connection. However, in other embodiments, participant state information can be provided through all peer connections, and there can be other mechanisms to prevent the circulation of participant state information across peer connections.

[0094] In some embodiments, all participant status information 216c can be transferred via peer-to-peer connections. In other embodiments, only incremental updates to participant status information can be transferred via peer-to-peer connections. In other embodiments, participant status information based on a specific information type can be transferred without transferring participant status information of another type. For example, participant health information can be transferred without transferring participant location information. In other embodiments, participant status information associated with one group of participants can be transferred without transferring participant status information associated with another group of participants.

[0095] Process 700 proceeds to decision box 708, where the second client computing device is monitoring whether it has received any participant status information via its peer connection. In an embodiment, at decision box 708, the second client computing device 14b / 200b (although the first client computing device 14a / 200a could also perform this step) can monitor for any participant status information received on its established peer connection. If no participant status information has been received, process 700 can return to decision box 708.

[0096] However, if participant status information has already been received at decision box 708, process 700 can proceed to box 710, where the stored participant status information is updated using any of the most recent participant status information. In various embodiments, at box 710, the second client computing device 14b / 200b can determine whether to update any of its participant status information 216c. The second client computing device 14b / 200b can compare each participant status information entry it has in the participant status information 216c with the participant status information received via a peer-to-peer connection. For example, the second client computing device 14b / 200b can compare the participant entry for the first participant in the participant status information 216c with the participant status information received for the first participant from the first client computing device 14a / 200a or other client computing devices. The second client computing device 14b / 200b can compare timestamps to determine which participant status entry is the most recent. If the received participant status information is more recent than the stored participant status information, the more recent participant status information can be used to update the portion of the participant status information 216c held by the second client computing device 14b / 200b.

[0097] Process 700 can proceed to block 712, where the updated participant status information can cause changes in I / O device output associated with the second client computing device. In an embodiment, at block 712, the updated participant status information can cause output to be generated via user I / O system 218 or display system 220. For example, the participant status information may already include updated participant location information for the first participant. Thus, peer application 204c or 304a can cause a graphical user interface including a visual representation of the first participant to move via display system 220 of client computing devices 14b / 200b. In other embodiments, the updated participant location information may be for a third participant upstream of the second client computing device 14b / 200b via the first client computing device 14a / 200a. Thus, if the location information now indicates that the third participant is entering a viewpoint of a coordinate grid environment displayed on the second client computing device 14b / 200b, this coordinate grid environment can cause display system 220 to display the third participant moving into the coordinate grid environment displayed by the second client computing device. While specific examples of variations in the provision of I / O device outputs are shown, other I / O device outputs are conceivable and remain within the scope of this disclosure.

[0098] Process 700 can proceed to decision box 714, where it is determined whether the state transition condition is met. In an embodiment, at decision box 714, the second client computing device 14b / 200b can determine whether the state transition condition is met. Decision box 714 can be operated by the first client computing device similar to decision box 704, and therefore will not be discussed in detail.

[0099] When the state transition condition is met in decision block 714, process 700 can proceed to block 716. At block 716, the second client computing device can transfer at least a portion of the updated participant state information over the peer connection. The operation of block 716 can be similar to that of block 706 and will therefore not be discussed in detail. Thus, the second client computing device 14b / 200b can provide its participant state information to the first client computing device 14a / 200a via the peer connection, and the first client computing device 14a / 200a can complete the operations in blocks 708 to 712 before returning to decision block 704. Similarly, the second client computing device 14b / 200b can provide its participant state information to another client computing device 14a / 200a via another peer connection.

[0100] Figure 8 An example coordinate grid environment 800 according to various embodiments of the present disclosure is illustrated. The coordinate grid environment 800, or virtual world, may include two multi-participant closed mesh networks 802 and 804. Mesh network 802 may include participants 1, 2, and 3, each interconnected via peer connections 802a, 802b, and 802c. Mesh network 804 may include participants 4, 5, and 6, each interconnected via peer connections 804a, 804b, and 804c. Initially, participant 7 is independent and may be considered a third “mesh network.” In both mesh networks 802 and 804, all participants interact with each other and exchange participant state information to indicate changes in participant state to each other when something about their own state changes or certain other state transition conditions are met. For example, if participant 2 updates some information about themselves, then participant 2 will send a timestamped update to every other participant they are connected to (e.g., participant 1 and participant 3). Participants 1 and 3 will relay this information to all other participants they are connected to. However, participant 3 will ignore participant 1's relay of updates to participant 2, because participant 3 is connected to participant 2 and has already received or will receive this information from participant 2. Participant 1 will perform the same operation on participant 3's relay. Participants 1 and 3 can distinguish the relayed information from other relays based on timestamps and the location of the information source.

[0101] In the example shown, participant 7 can move independently between mesh networks 802 and 804. At some point, participant 7 and participant 3 can establish a peer-to-peer connection 806a as described above. Participant 3 remains part of mesh network 802 but has a peer-to-peer connection 806a to participant 7. Participant 7 can join mesh network 802. Upon connection, the following may occur: (1) Participant 7 can provide participant 3 with at least a portion of the participant state information that participant 7 is tracking. Participant 3 can provide participant 7 with at least a portion of the participant state information that participant 3 is tracking. Participants 7 and 3 will merge their exchanged state databases together. Now participant 7 will know everything that participant 3 knows, and participant 3 will also know everything that participant 7 knows.

[0102] Subsequently, if participant 7, connected to participant 3, decides to connect to participant 4, and participant 4 responds via peer connection 806b, participants 4 and 7 can exchange their participant state databases (e.g., participant state information) in the manner already described above for participants 7 and 3. Thus, participant 7 may now know information known by participant 4, and vice versa. Furthermore, participant 7 can relay any new information received from participant 4 to participant 3. Participant 3 will perform the same operation on new participant state information received from participant 7, causing participants 1 and 2 to receive the new information. Similarly, participant 4 can relay some or all of the new participant state information received from participant 7 to participants 6 and 5.

[0103] At the end of the merging and relaying, all participants (e.g., participants 1 through 7) across mesh networks 802 and 804 via participant 7 now know and agree on the location information of the participants or the coordinate grid status of all other participants. This information exchange can be carried out without the need for a central server.

[0104] Therefore, the peer-to-peer mesh technology of this disclosure allows for seamless entry and exit of peer connections between peers. This peer-to-peer state exchange mechanism allows for the sharing of game-like state among all participants in a large world. The use of peer connections allows coordinate grid state like this to propagate throughout a very large-scale world / coordinate grid without requiring the overhead of state server computing devices for at least a portion of the participant state information.

[0105] Hybrid peer-to-peer mesh and SFU flow distribution

[0106] As discussed above, there is an increasing prevalence of applications for streaming video and audio between participants in certain environments. As application instances grow in clusters, reliably maintaining connections between subsets of participants becomes increasingly difficult. Maintaining streaming between all participants is rarely necessary, but in many cases, proximity connections are invaluable.

[0107] Peer-to-peer mesh topologies can adequately and cheaply connect subsets of instance clusters, but challenges remain in distributing a single stream to a larger number of people. A world event might require announcing it globally. Peer-to-peer has its limitations. Due to encoding processing and upload bandwidth constraints, a single client can only maintain more than 10 streams simultaneously. Streaming servers that blend with peering (e.g., SFU) can help bridge this gap.

[0108] Running world events and utilizing peer-to-peer meshes for most smaller group interactions is efficient and inexpensive. However, there are situations where distributing one or more streams to all users in a world or a region of a world is necessary or valuable. Peer-to-peer mesh streaming may not be sufficient to stream instances to a large number of participants (e.g., more than 10 users, at least 15 users, at least 20 users, at least 50 users, at least 100 users, at least 1000 users, at least 10000 users, or any other number of users that meet a threshold quality degradation level that is obvious to those skilled in the art with this disclosure). For example, most client computing devices are not designed to stream to only a few peers simultaneously via peer-to-peer connections. Viewers among the hundreds or thousands of viewers arriving at a single stream benefit from Selective Forwarding Units (SFUs).

[0109] Therefore, compared to establishing peer-to-peer connections and streaming through these connections, the systems and methods disclosed herein provide the ability to know when to utilize SFUs to publish streams and when to retrieve streams from SFUs. Knowing when to utilize SFUs and peer-to-peer networks can improve network efficiency for distributing streams in mobile environments.

[0110] For example, each participant in a coordinate grid environment may be in: (1) a base state, (2) a broadcast state, (3) a screen-sharing state, or any other state that is obvious to those skilled in the art possessing this disclosure. A participant in the base state can only connect to the participants around them. This participant can connect to each of its neighboring participants via peer-to-peer connections, which can be based on the above... Figure 4 Process 400 and Figure 5The signal buffering method discussed in process 500. Additionally, participants in the base state can upload their audio or video streams via this peer-to-peer connection. In various embodiments, a participant in the base state can read the streams of each participant around them based on the state of each other participant. If a nearby participant is also in the base state, the participant will read the stream from the peer-to-peer connection with the nearby participant. If a nearby participant is in a broadcast or screen-sharing state, this participant can read the stream from the SFU.

[0111] As discussed above, in some cases, participants may be in a broadcast state. A participant in a broadcast state can provide a stream that will be seen / heard more often than those around them. In some embodiments, if a participant is broadcasting, that participant can upload their stream to the SFU. In various embodiments, a broadcast participant can also establish peer-to-peer connections with each nearby participant, which can be performed according to the signal buffering methods described above in processes 400 and 500. Participant state information, which may include the communication mode in which the participant is located (e.g., basic state, broadcast state, or screen-sharing state), can be computed via a state server computing device or according to the methods discussed above. Figure 7 The process 700 is transferred to each participant. In various embodiments, since the broadcast participants are uploading their streams to the SFU, it may not be necessary for the broadcast participants to upload their streams to the peer connection, and each nearby participant will know that it is getting the broadcast participant's stream from the SFU. If a nearby participant is not in a broadcast or screen-sharing state, as discussed in further detail below, the nearby participant will become a broadcast participant through the established peer stream.

[0112] In some cases, participants may be in a screen-sharing state. In some embodiments, the stream can be distributed if the stream of a participant in a screen-sharing state is of sufficiently high quality to make it readable. If any participant shares their screen, the stream will be uploaded to and distributed from the SFU, similar to broadcast mode.

[0113] In various embodiments, each participant can know the current participant status of every other participant in the world through the participant state information exchange discussed herein.

[0114] In some embodiments, server 12, relay server 16, client 14, or selective forwarding unit server 18 may be operated, for example, by running program code stored on a tangible, non-transitory, machine-readable medium. Figure 9The process 900. The operations shown can be run in different orders, omitted, copied, run concurrently, run serially, insert additional operations, be fully automated, involve human intervention, or otherwise modified relative to the depicted arrangement, without implying that any other description herein is limiting.

[0115] In some embodiments, process 900 in Figure 9 As shown in the diagram. Process 900 may begin at block 902, where a peer-to-peer connection is established between a first client computing device and a second client device. In an embodiment, at block 902, a peer-to-peer connection may be established between the first client computing device 14a / 200a and the second client computing device 14b / 200b. The establishment of the peer-to-peer connection can be based on the above... Figure 4 or Figure 5 The peer connections are established using processes 400 and 500 described herein. However, in other embodiments, peer connections may be established using other processes that are obvious to those skilled in the art upon which this disclosure is known. In various embodiments, the first client computing device 14a / 200a or the second client computing device 14b / 200b may each establish multiple peer connections with their respective client computing devices. Thus, for example, the first client computing device may establish a second peer connection with a third client computing device, a third peer connection with a fourth client computing device, a fourth peer connection with a fifth client computing device, and so on, until a maximum peer connection limit is reached, which may be based on performance, predetermined or dynamic system limitations, or client computing device capabilities.

[0116] Referring to the embodiments described in processes 400 and 500, the establishment of a peer connection can occur in a coordinate grid environment provided by peer application 204c or 304a, and when a first participant associated with a first client computing device 14a / 200a satisfies a proximity condition with a second participant associated with a second client computing device 14b / 200b in the coordinate grid environment. Thus, the state server computing device 12 / 300 can provide participant state information (e.g., participant location information) to each participant in the coordinate grid environment, and each participant can calculate its distance and position or other coordinate grid state based on this participant state payload. However, in accordance with this disclosure and the foregoing references... Figure 7In other embodiments discussed in detail in process 700, each of the participants may belong to the same mesh network of peer-to-peer participants. For example, the first client computing device 14a / 200a and the second client computing device 14b / 200b may be connected to each other via one or more intermediate client computing devices. Based on the shared participant state information provided by the peer-to-peer connection through the mesh network, the first client computing device 14a / 200a and the second client computing device 14b / 200b can obtain each other's participant location information and determine that they are adjacent to each other, enabling the establishment of a peer-to-peer connection.

[0117] Process 900 can proceed to decision block 904, where it is determined whether the selective forwarding unit condition is met. In an embodiment, at decision block 904, the first client computing device 14a / 200a can determine whether the selective forwarding unit condition is met, such that when the selective forwarding unit condition is met, the first client computing device 14a / 200a provides the media stream via selective forwarding unit 18, or when the selective forwarding unit condition is not met, the first client computing device 14a / 200a provides the media stream via peering connection. For example, as discussed above, compared to peering connection, selective forwarding unit 18 can better transmit media streams that require high quality or certain other conditions to reach a larger group of participants in the coordinate grid environment. Thus, at decision block 904, the first client computing device 14a / 200a can monitor the media stream and the transmission information associated with the media stream to determine whether the number of participants to which the media stream is destined exceeds a participant threshold, whether the media stream exceeds a quality threshold that cannot be met by peering connection, or some other selective forwarding unit condition that is obvious to those skilled in the art.

[0118] If it is determined in decision box 904 that the media stream does indeed meet the selective forwarding unit condition, process 900 can proceed to box 906, where the first client computing device enters the selective forwarding unit state. In embodiments, in box 906, the first client computing device 14a / 200a can update the communication state to the selective forwarding unit state. In some embodiments, the selective forwarding state may include, for example, a broadcast state when the media stream exceeds a participant threshold. In other embodiments, the selective forwarding state may include, for example, a screen-sharing state when the media stream meets a quality threshold. In various embodiments, changes in the communication state identified in the participant state information can lead to... Figure 7 The state transition condition is met in decision box 704 of process 700. Therefore, the communication state change of the first client computing device 14a / 200a can be determined according to... Figure 7The process 700 is communicated to other participants and their associated client computing devices via a peer-to-peer connection. However, in other embodiments, participant state information can be distributed to participants in the coordinate grid environment via the state server computing device in the participant state payload 308a, as described above. Figure 4 As discussed in the process.

[0119] Process 900 may proceed to block 908, where the first client computing device provides a media stream via a selective forwarding unit connection to the selective forwarding unit. In an embodiment, at block 908, the first client computing device 14a / 200a may provide a media stream to the selective forwarding unit 18 via communication system 210. The selective forwarding unit 18 may provide a media stream to participants in the coordinate grid environment identified in the broadcast. In some embodiments, a participant may be a subset of multiple participants in the coordinate grid environment. For example, when the first client computing device 14a / 200a is in broadcast mode, the selective forwarding unit 18 may provide a media stream to all participants in the coordinate grid environment or a subset of participants in the coordinate grid environment that exceeds the maximum peer-to-peer connection value. In another example, when the first client computing device 14a / 200a is in screen-sharing mode, the selective forwarding unit 18 may provide a media stream to all participants in the coordinate grid environment or a subset of participants in the coordinate grid environment that may exceed or not exceed the maximum peer-to-peer connection value (because screen-sharing mode is quality-based).

[0120] Returning to reference decision box 904, if the selective forwarding unit condition is not met, process 900 can proceed to box 910, where the first client computing device can enter the peer-to-peer communication state. In an embodiment, at box 910, when the selective forwarding unit condition is not met, the first client computing device 14a / 200a can update its communication state to the peer-to-peer communication state. The peer-to-peer communication state can be referred to as the base state and can be the default state of the first client computing device. Thus, the communication state can only be updated if the first client computing device 14a / 200a is not in the base state to be started. In various embodiments, in Figure 7 In the decision box 704 of process 700, a change in the communication state indicated by the participant's state information can lead to the satisfaction of the state transition condition. Therefore, the change in the communication state of the first client computing device 14a / 200a can be determined according to... Figure 7 The process 700 is communicated to other participants and their associated client computing devices via a peer-to-peer connection. However, in other embodiments, participant state information can be distributed to participants in the coordinate grid environment via the state server computing device in the participant state payload 308a, as described above. Figure 4 As discussed in process 400.

[0121] Process 900 may proceed to block 912, where the first client computing device provides a media stream via one or more peer connections. In an embodiment, at block 912, the first client computing device 14a / 200a may provide a media stream to a second client computing device 14b / 200b via a peer connection through communication system 210, or provide a media stream to other participants in the coordinate grid environment via peer connections established between the first client computing device 14a / 200a and other client computing devices.

[0122] Blocks 914 to 924 of process 900 are described from the perspective of the second client computing device 14b / 200b. However, the functionality of the second client computing device 14b / 200b can be performed by the first client computing device 14a / 200a, and vice versa. In decision block 914, the second client computing device can monitor participant status information. In various embodiments, in decision block 914, the second client computing device 14b / 200b can monitor participant status information from a state server computing device or another client computing device such as the first client computing device 14a / 200a. If no participant status information is received, the second client computing device 14b / 200b can continue to monitor the participant status information. Decision block 914 can operate similarly to decision block 708 of process 700 discussed above. If no participant status information is received, process 900 can return to decision block 914.

[0123] However, if participant status information has already been received at decision box 914, process 900 can proceed to box 916 (which may be similar to box 710 of process 700 discussed above), where the stored participant status information is updated using any recently updated participant status information. In various embodiments, at box 916, the second client computing device 14b / 200b can determine whether to update any of its participant status information 216c. The second client computing device 14b / 200b can compare each participant status information entry it has in the participant status information 216c with the participant status information received by it via a peer-to-peer connection or a status server computing device. For example, the second client computing device 14b / 200b can compare the participant entry for the first participant in the participant status information 216c with the participant status information it received from the first client computing device 14a / 200a for the first participant when the first client computing device 14a / 200a updated its communication status. The second client computing device 14b / 200b can compare timestamps to determine which participant status entry is more recent. If the received participant status information is more recent than the stored participant status information, the more recent participant status information can be used to update that portion of the participant status information 216c held by the second client computing device 14b / 200b. Thus, if the second client computing device 14b / 200b has received a communication status of the first client computing device 14a / 200a that is more recent than the one held by the second client computing device 14b / 200b in the participant status information 216c, the second client computing device 14b / 200b can use the new status (e.g., basic status, broadcast status, screen sharing status, or any other communication status) to update the participant status information 216c.

[0124] Process 900 may proceed to block 918, where the second client computing device can monitor one or more communication channels for media streaming based on the communication status of each participant in the coordinate grid environment. In an embodiment, at block 918, the second client computing device 14b / 200b can monitor the communication channels as indicated by the communication status included in the participant status information 216c. For example, when the communication status of the first client computing device 14a / 200a is in a basic state, the second client computing device 14b / 200b can monitor the peer-to-peer connection for media streaming between the first client computing device 14a / 200a and the second client computing device 14b / 200b. In another embodiment, when the communication status of the first client computing device 14a / 200a is in a screen-sharing state or a broadcast state, the second client computing device 14b / 200b can monitor the communication channel with the selective forwarding unit 18 for media streaming from the first client computing device 14b / 200b. In some embodiments, when the communication state of the first client computing device 14a / 200a is in a broadcast state or a screen sharing state, the second client computing device 14b / 200b may block the media stream from the first client computing device 14a / 200a.

[0125] Process 900 can proceed to decision box 920, where it is determined whether the media stream has been detected by the second client computing device. If no media stream has been detected, the process can return to box 918.

[0126] However, if a media stream is detected at the communication channel being monitored by the second client computing device at decision block 920, process 900 can proceed to block 922, where the media stream is received via the communication channel indicated by the communication state of this participant. In an embodiment, at block 922, the second client computing device 14b / 200b can receive the media stream based on the communication state of each participant it has. For example, if the communication state of a participant in the coordinate grid environment associated with the first client computing device 14a / 200a indicates a broadcast state or a screen-sharing state, the second client computing device 14b / 200b can receive the media stream via the communication channel communicating with the selective forwarding unit 18. If the communication state of a participant in the coordinate grid environment associated with the first client computing device 14a / 200a indicates a basic state, the second client computing device 14b / 200b can receive the media stream via the peer-to-peer connection between the first client computing device 14a / 200a and the second client computing device 14b / 200b.

[0127] Process 900 may proceed to block 924, where the media stream is reproduced or output by the second client computing device. In an embodiment, at block 924, the second client computing device 14b / 200b may output the media stream via display system 220 or user I / O system 218. For example, an audio stream may be provided via a speaker included on user I / O system 218. In another example, a video stream may be provided via a display included in display system 220. However, other media streams may be output via other output devices or combinations of output devices included in display system 220 or user I / O system 218 that are obvious to those skilled in the art.

[0128] Figure 10 This is a flowchart illustrating an example stream exchange in a hybrid use of peer-to-peer and selective forwarding unit (SFU) streaming environment 1000. In this example, it is assumed that all participants in the coordinate grid environment are close enough to be interconnected via peer-to-peer connections. This means that each participant will negotiate a peer-to-peer connection with each of the other participants, creating a mesh network among the four participants, as indicated by peer connections 1002a, 1002b, 1002c, 1002d, 1002e, and 1002f. Additionally, in this example, participant 2 is in screen-sharing mode, participant 4 is in broadcast mode, and participants 1 and 3 are in base mode. This means that participants 2 and 4 are streaming their audio, video, or other media content to SFU 1004, as indicated by arrows 1006 and 1008, respectively. Neither participant 2 nor participant 4 is streaming their media streams on the peer connections 1002a through 1002f that exist between them and the other three participants.

[0129] However, participants 2 and 4 each download media streams from participants 1 and 3 from peer connections 1002a, 1002c, 1002d, and 1002f to these participants. No bytes flow in either direction on peer connection 1006e between participants 2 and 4. Participants 1 and 3 each upload and download media streams to each other on peer connection 1002b. Participants 1 and 3 also upload their media streams to 1002a, 1002c, 1002d, and 1002f.

[0130] In various embodiments, participants 1, 3, and 4, which consume streams from participant 2, download these streams from SFU 1004 as indicated by arrows 1010, 1012, and 1014, respectively. Participants 1, 2, and 3 consume streams from participant 4 as indicated by arrows 1016, 1018, and 1020, respectively.

[0131] When participant 2 or participant 4 stops broadcasting or screen sharing, they will stop uploading streams to SFU 1004 and instead upload streams to their established peer connections 1002a, 1002c, 1002d, 1002e, or 1002f. If participant 1 or participant 3 wants to start broadcasting or screen sharing, they will stop uploading their streams to peer connections 1002a, 1002b, 1002c, 1002d, and 1002f and instead upload them to SFU 1004. All other participants in the range will then retrieve their new streams from SFU 1004.

[0132] Therefore, the systems and methods disclosed herein provide a peer-to-peer environment for normal proximity interaction, but still utilize SFU when higher quality (screen sharing) is required, or when a larger number of streaming participants must access a participant's stream (broadcast), or when non-local participants need a stream.

[0133] Peer-to-peer mesh network bit rate throttling

[0134] As discussed in this paper, in large-scale clustered application coordinate grid environments, individual mesh networks are used to connect participants whose distances are close enough to warrant direct peer-to-peer connections. As peer-to-peer participants in a coordinate grid environment connect, a mesh network is formed. If these participants stream something like video between each other, then processing and bandwidth consumption increases with each new participant added to the grid. Because each participant in a mesh or peer pair has different computational and bandwidth capabilities, balancing the quality of the stream with the overall experience is crucial, as the network size increases and decreases, to throttle the maximum bandwidth between each participant.

[0135] Therefore, as the size of a mesh network increases, the processing and bandwidth requirements for maintaining the mesh also increase. With more participants joining the mesh, the pressure on each individual participant's client computing devices and network increases. Each participant must assess their capabilities and throttle their output to maintain all necessary connections.

[0136] In various embodiments of this disclosure, bitrate throttling can be performed when participants enter and leave any given peer-to-peer mesh. Each participant knows how many other participants they are currently connected to. For example, if participant A and participant B are connected to each other, they enter a mesh of two participants. Since they are only two participants, each of them can stream at high quality. Specifically, participants A and B can stream video at a high quality of 1 MB / s. If participant C enters the range of both A and B, the mesh will evolve into a network of three participants. Participant C can detect the peer-to-peer connection with the two other participants. Participant C can keep its maximum bandwidth consumption below 1 MB / s and thus limit the bitrate to both participants A and B to 500 kb / s. Participants A and B will detect that another participant has entered the mesh and can limit participant C's bitrate to 500 kb / s. They will also renegotiate their connection and limit the bitrate to 500 kb / s. After this adjustment, a peer-to-peer mesh network with three participants exists, each streaming to each other at a maximum of 500kb / s, and no participant individually consumes more than 1MB / s. If participant B wants to leave the mesh, participants A and C can renegotiate their connection and raise the limit to 1MB / s, allowing for further quality improvements.

[0137] In some embodiments, server 12, relay server 16, client 14, or selective forwarding unit server 18 may be operated, for example, by running program code stored on a tangible, non-transitory, machine-readable medium. Figure 11 The process 1100. The operations shown can be run in different orders, omitted, copied, run concurrently, run serially, insert additional operations, be fully automated, involve human intervention, or otherwise modified relative to the depicted arrangement, without implying that any other description herein is limiting.

[0138] In some embodiments, process 1100 in Figure 11As shown in the diagram. Process 1100 may begin at decision block 1102, where it is determined whether the peering connection conditions for initiating a peering connection between client computing devices associated with participants are met. In an embodiment, at decision block 1102, the peering application 204c of the first client computing device 14a / 200a may determine whether the conditions for establishing a peering connection with the second client computing device 14b / 200b are met. For example, peering application 204c may determine whether the participant associated with the second client computing device 14b / 200b is adjacent to the participant associated with the first client computing device 14a / 200a (e.g., within a predetermined distance, or within a distance determined based on the congestion of the coordinate grid environment). Peering application 204c may determine the distance between the participant of client computing device 14a / 200a and other participants in the coordinate grid environment provided by peering application 204c. For this purpose, peering application 204c may reference the distances described above by client computing device 14a / 200a. Figure 4 The process 400 determines the coordinate grid state 216b. Although peer-to-peer connectivity conditions can be based on proximity or a predetermined distance, other peer-to-peer connectivity conditions can also be envisioned.

[0139] Process 1100 may proceed to block 1104, where the maximum bit rate for peering connections with participants is determined. In an embodiment, at block 1104, the peering application controller 204 may determine the maximum bit rate for peering connections. The maximum bit rate for peering connections may be based on the number of peering connections to which the first client computing device 14a / 200a is connected, and the maximum bit rate or bandwidth of the first client computing device 14a / 200a. An administrator may set the maximum bit rate for a coordinate grid environment such that the client computing device can have an actual maximum bit rate greater than the set maximum bit rate. Specifically, the maximum bit rate for peering connections may be calculated as a function of the maximum bit rate of all peering connections established by the client computing device and the number of connections. In one example, the bit rate for peering connections may be the maximum bit rate for the client computing device divided by the number of peering connections. However, in other embodiments, as discussed below, it may be determined based on the maximum bit rate of the connected client computing devices. For example, if participant A has two peer connections and a maximum bit rate of 1 Mbps, and participant B has four peer connections and a maximum bit rate of 1 Mbps, then the connection between participant B and participant A will be limited to a bit rate of 250 kbps per connection for participant B. Thus, the first peer connection between participant A and, for example, participant C could have a bit rate of 750 kbps, while the peer connection between participant A and participant B would be at 250 kbps, even though participant A could have a bit rate of 500 kbps with participant B. In other embodiments, the maximum bit rate for peer connections can be calculated based on other factors (e.g., the priority of a particular participant, attributes in the participant state information, or any other factor).

[0140] Process 1100 may proceed to block 1106, where a peer initiation signal including the maximum bit rate for the connection is generated. In an embodiment, at block 1106, the first client computing device 14a / 200a may generate a peer initiation signal announcing that the first client computing device 14a / 200a must propose a maximum bit rate to the second client computing device 14b / 200b with which it will establish a peer connection. The peer initiation signal may include a bandwidth or maximum bit rate entry or announcement. Continuing with the WebRTC example herein, the SDP proposal packet provided by the first client computing device 14a / 200a to the second client computing device 14b / 200b to initiate a peer connection may include a bandwidth entry setting the maximum bit rate for the peer connection. Typically, the SDP proposal maintains a bandwidth that allows WebRTC to consume as much bandwidth as possible. The peering application 204c of this disclosure may modify the content of the SDP proposal such that the maximum bit rate for this peer connection is announced in the bandwidth entry.

[0141] Process 1100 may proceed to block 1108, where the first client computing device sends a peer-to-peer connection initiation signal to the second client computing device, and the second client computing device receives the peer-to-peer connection initiation signal block 1110. In various embodiments, the first client computing device 14a / 200a may determine that it will be the initiator of a peer-to-peer connection with the second client computing device 14b / 200b. For example, the first client computing device 14a / 200a may determine that it is the initiator based on its participant identifier and the participant identifier associated with the second client computing device 14b / 200b. Specifically, the first client computing device 14a / 200a may determine that its participant identifier is greater than the participant identifier associated with the second client computing device 14b / 200b, which satisfies the participant identifier condition. However, in other embodiments, the participant identifier assigned the lowest participant identifier may be the initiator of the peer-to-peer connection.

[0142] Thus, the first client computer device 14a / 200a sends a peer-to-peer connection initiation signal and a maximum bit rate announcement for the peer-to-peer connection to the second client computer device 14b / 200b. For example, the peer-to-peer application 204c may communicate with a Real-Time Communication (RTC) application programming interface (API) (e.g., the WebRTC API). The WebRTC protocol may include an interactive connectivity establishment (ICE) protocol for establishing peer-to-peer connections. Therefore, the peer-to-peer initiation signal may include an SDP proposal including a maximum bit rate or bandwidth announcement according to a Session Description Protocol (SDP). More specifically, the SDP protocol may include a video bandwidth announcement. In various embodiments, the peer-to-peer initiation signal is provided via network 20 and relay server 16. For example, the peer-to-peer initiation signal is provided through a persistent connection (e.g., a web socket) established between the first client computing device 14a / 200a and relay server 16, and a persistent connection established between the second client computing device 14b / 200b and relay server 16. The second client computing device 14b / 200b can then receive peer-initiated signals.

[0143] Process 1100 may proceed to block 1112, where the second client computing device determines the maximum bit rate for the peer-to-peer connection. In an embodiment, at block 1112, the second client computing device 14b / 200b may determine its maximum bit rate for the peer-to-peer connection. The second client computing device 14b / 200b may determine its maximum bit rate based on the operations performed by the first client computing device 14a / 200a as described in block 1104, and therefore will not be repeated in detail. In various embodiments, if the maximum bit rate determined by the second client computing device 14b / 200b is greater than the maximum bit rate announced in the peer-to-peer initiation signal, then the second client computing device 14b / 200b may determine that its maximum bit rate for the peer-to-peer connection is the maximum bit rate announced in the peer-to-peer initiation signal.

[0144] Process 1100 can then proceed to box 1114, where the second client computing device generates a peering response signal using its determined maximum bit rate announcement. In an embodiment, at box 1114, the second client computing device 14b / 200b can generate a peering response signal announcing the maximum bit rate that the second client computing device 14a / 200a must propose to the first client computing device 14a / 200a with which it will establish a peering connection. The peering response signal may include a bandwidth or maximum bit rate entry or announcement. Continuing with the WebRTC example here, the SDP response packet provided by the second client computing device 14b / 200b to the first client computing device 14a / 200a in response to an SDP proposal may include a bandwidth entry setting the maximum bit rate for the peering connection. Typically, the SDP response maintains bandwidth that allows WebRTC to consume as much bandwidth as possible. The peering application 204c of this disclosure can modify the content of the SDP response such that the maximum bit rate used for this peering connection is announced in the bandwidth entry.

[0145] Process 1100 may proceed to block 1116, where the second client computing device sends a peer-to-peer response signal. In an embodiment, at block 1116, the second client computing device 14b / 200b may respond to the peer-to-peer initiation signal via relay server 16 with a peer-to-peer response signal to the first client computing device 14a / 200a. At block 1118, the first client computing device 14a / 200a may receive and process the peer-to-peer response signal to establish a peer-to-peer connection (e.g., a real-time connection using WebRTC). At block 1120, one or more media streams (e.g., video streams, audio streams, AV streams, or other media content streams) may then be provided on the peer-to-peer connection between the first client computing device 14a / 200a and the second client computing device 14b / 200b, such that these media streams do not exceed the negotiated maximum bit rate.

[0146] Process 1100 can then proceed to decision block 1122, where the first client computing device determines whether a bit rate renegotiation condition is met on the peering connection. In an embodiment, at decision block 1122, the first client computing device 14a / 200a can determine whether a bit rate renegotiation condition exists. For example, a bit rate renegotiation condition may be met when the first client computing device 14a / 200a determines that it will establish a subsequent peering connection with another client computing device, requiring the release of bandwidth. In another example, a bit rate renegotiation condition may be met when another peering connection on the first client computing device 14a / 200a is removed, allowing additional bandwidth to be used on the peering connection with the second client computing device 14b / 200b. In another embodiment, a bit rate renegotiation condition may be met when a renegotiation signal is received from the second client computing device. In other embodiments, other bit rate renegotiation conditions that are obvious to those skilled in the art are possible to monitor.

[0147] If the bit rate renegotiation condition is not met, then in block 1120, process 1100 can continue to provide the media stream via the peer connection at a bit rate not exceeding the maximum bit rate. If the bit rate renegotiation condition is met in decision block 1122, then the first client computing device 14a / 200a can renegotiate the maximum bit rate with the second client computing device 14b / 200b. For example, in block 1104, the first client computing device 14a / 200a can determine the maximum bit rate for the peer connection and provide the updated maximum bit rate to the second client computing device 14b / 200b to renegotiate and update the maximum bit rates on both client computing devices. For example, when a connection needs to be negotiated via a signaling channel, a "negotiationneeded" event can be sent to the peer connection (e.g., RTCPeerConnection). This can occur during the initial establishment of the peer connection and at any time when changes in the communication environment require reconfiguration of the peer connection. Although decision box 1122 is described from the perspective of the first client computing device 14a / 200a, the second client computing device 14b / 200b can also monitor the bit rate renegotiation signal.

[0148] Figure 12A It is a coordinate grid environment 1200 with participants 1202 and 1204. Participants 1202 and 1204 can establish a peer connection 1206 when they are adjacent to each other, such that the peer connection initiation conditions are met, which may include... Figure 5Any of the operations in process 500. Participants 1202 and 1204 can negotiate a maximum bit rate of 1 Mbps, which is likely a limitation that any client computing device can have in a coordinate grid environment, even if the client computing device is configured to have a higher bit rate. Subsequently, as Figure 12B As shown, participant 1208 can enter coordinate grid environment 1200, enabling it to establish peering connection 1210 with participant 1202 and peering connection 1212 with participant 1204. The maximum bit rate used for peering connection 1206 can be renegotiated to 500 kbps. Furthermore, peering connections 1210 and 1212 can be throttled to 500 kbps. While the example shows halving the bit rate during throttling, other bit rate allocations can be implemented. Figure 12C It was shown that participant 1204 had left by Figure 12B The mesh network formed by participants 1202, 1204, and 1208 eliminates the need for peer connections 1206 and 1212. Participants 1202 and 1208 can renegotiate the maximum bit rate on peer connection 1210 to 1 Mbps. As those skilled in the art, having mastered this disclosure and these examples, will understand, as various participants enter and leave the mesh network and form various mesh topologies, peer connections can be renegotiated throughout the mesh network based on the number of participants and the mesh network topology.

[0149] Therefore, the systems and methods of this disclosure provide peer-to-peer bitrate throttling. When a peer-to-peer connection is established, the client computing device announces the determined maximum bitrate and negotiates the maximum bitrate used for this peer-to-peer connection. When participants in a coordinate grid environment enter and leave a grid network including a peer-to-peer connection, the maximum bitrate used for this connection can be re-determined and renegotiated by the client computing device. Thus, as the network size grows and shrinks, the bitrate throttling of this disclosure allows for a balance between the quality of the media stream and the overall experience.

[0150] This patent filing is one of three patent applications filed on the same day by the same applicant, a group whose members include: Peer-to-Peer Signal Buffering System; Hybrid Peer-to-Peer Mesh and Forwarding System; and Client Authority Status Distribution System. The entire contents of each of the other patent filings are incorporated herein by reference.

[0151] Figure 13This is a diagram illustrating an exemplary computing system 1300 according to an embodiment of the present technology. Various parts of the systems and methods described herein may include or be executed on one or more computer systems similar to computing system 1300. For example, client computing devices 14 / 200, server computing devices 12 / 300, relay server 16, or selective forwarding unit 18 may be provided by computing system 1300. Furthermore, the processes and modules described herein may be run by one or more processing systems similar to computing system 1300.

[0152] The computing system 1300 may include one or more processors (e.g., processors 1310a to 1310n) coupled to system memory 1320, input / output I / O device interface 1330, and network interface 1340 via input / output (I / O) interface 1350. The processor may include a single processor or multiple processors (e.g., a distributed processor). The processor may be any suitable processor capable of running or otherwise executing instructions. The processor may include a central processing unit (CPU) that executes program instructions to perform arithmetic, logical, and input / output operations of the computing system 1300. The processor may execute code that creates an execution environment for the program instructions (e.g., processor firmware, protocol stack, database management system, operating system, or a combination thereof). The processor may include a programmable processor. The processor may include a general-purpose or special-purpose microprocessor. The processor may receive instructions and data from memory (e.g., system memory 1320). The computing system 1300 may be a single-processor system including one processor (e.g., processor 1310a), or a multiprocessor system including any number of suitable processors (e.g., 1310a to 1310n). Multiple processors can be employed to provide parallel or sequential execution of one or more portions of the techniques described herein. Processes such as logical flows described herein can be executed by one or more programmable processors running one or more computer programs to perform functions by manipulating input data and generating corresponding outputs. The processes described herein can be executed by dedicated logic circuit systems, and the apparatus can also be implemented as dedicated logic circuit systems, such as FPGAs (field-programmable gate arrays) or ASICs (application-specific integrated circuits). Computing system 1300 may include multiple computing devices (e.g., a distributed computer system) to implement various processing functions.

[0153] I / O device interface 1330 can provide an interface for connecting one or more I / O devices 1360 to computer system 1300. I / O devices may include devices that receive input (e.g., from a user) or output information (e.g., to a user). I / O device 1360 may include, for example, a graphical user interface presented on a display (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor), a pointing device (e.g., a computer mouse or trackball), a keyboard, keypad, touchpad, scanning device, voice recognition device, gesture recognition device, printer, audio speaker, microphone, camera, etc. I / O device 1360 can be connected to computer system 1300 via wired or wireless connections. I / O device 1360 can be connected to computer system 1300 from a remote location. For example, I / O device 1360 located on a remote computer system can be connected to computer system 1300 via a network and network interface 1340.

[0154] Network interface 1340 may include a network adapter that provides a connection between computer system 1300 and a network. Network interface 1340 may facilitate data exchange between computer system 1300 and other devices connected to the network. Network interface 1340 may support wired or wireless communication. The network may include electronic communication networks such as the Internet, local area network (LAN), wide area network (WAN), cellular communication network, etc.

[0155] System memory 1320 may be configured to store program instructions 1301 or data 1302. Program instructions 1301 may be executed by a processor (e.g., one or more processors 1310a to 1310n) to implement one or more embodiments of the present technology. Instructions 1301 may include computer program instruction modules for implementing one or more technologies described herein with respect to various processing modules. Program instructions may include computer programs (which are referred to in some forms as programs, software, software applications, scripts, or code). Computer programs may be written in programming languages, including compiled or interpreted languages, or declarative or procedural languages. Computer programs may include units suitable for use in a computing environment, including as standalone programs, modules, components, or subroutines. Computer programs may or may not correspond to files in a file system. Programs may be stored as a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file storing one or more modules, subroutines, or code portions). Computer programs can be deployed to execute on one or more computer processors located locally at a single site or distributed across multiple remote sites interconnected by a communication network.

[0156] System memory 1320 may include a tangible program carrier on which program instructions are stored. The tangible program carrier may include a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may include a machine-readable storage device, a machine-readable storage substrate, a memory device, or any combination thereof. A non-transitory computer-readable storage medium may include non-volatile memory (e.g., flash memory, ROM, PROM, EPROM, EEPROM memory), volatile memory (e.g., random access memory (RAM), static random access memory (SRAM), synchronous dynamic RAM (SDRAM)), mass storage memory (e.g., CD-ROM and / or DVD-ROM, hard disk drive), etc. System memory 1320 may include a non-transitory computer-readable storage medium on which program instructions executable by a computer processor (e.g., one or more of processors 1310a to 1310n) are stored to cause operation of the subjects and functions described herein. Memory (e.g., system memory 1320) may include a single memory device and / or multiple memory devices (e.g., distributed memory devices). Instructions or other program code that provide the functionality described herein may be stored on a tangible, non-transitory computer-readable medium. In some cases, the entire instruction set may be stored on the medium simultaneously, or in other cases, different parts of the instructions may be stored on the same medium at different times.

[0157] I / O interface 1350 can be configured to coordinate I / O traffic between processors 1310a to 1310n, system memory 1320, network interface 1340, I / O devices 1360, and / or other peripheral devices. I / O interface 1350 can perform protocol, timing, or other data transformations to convert data signals from one component (e.g., system memory 1320) into a format suitable for use by another component (e.g., processors 1310a to 1310n). I / O interface 1350 may include support for devices attached via various types of peripheral buses, such as the Peripheral Component Interconnect (PCI) bus standard or variants of the Universal Serial Bus (USB) standard.

[0158] Embodiments of the techniques described herein can be implemented using a single instance of computer system 1300 or multiple computer systems 1300 configured to host different portions or instances of the embodiments. Multiple computer systems 1300 can provide parallel or sequential processing / execution of one or more portions of the techniques described herein.

[0159] Those skilled in the art will understand that computer system 1300 is merely illustrative and not intended to limit the scope of the techniques described herein. Computer system 1300 may include any combination of devices or software capable of performing or otherwise providing the performance of the techniques described herein. For example, computer system 1300 may include cloud computing systems, data centers, server racks, servers, virtual servers, desktop computers, laptop computers, tablet computers, server equipment, client devices, mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, in-vehicle computers, or Global Positioning Systems (GPS), or combinations thereof. Computer system 1300 may also be connected to other devices not shown, or may operate as a stand-alone system. Furthermore, in some embodiments, the functionality provided by the illustrated components may be combined into fewer components or distributed across additional components. Similarly, in some embodiments, the functionality of some of the illustrated components may not be provided, or other additional functionality may be available.

[0160] Those skilled in the art will also understand that although various items are shown as being stored in memory or on storage devices during use, these items, or portions thereof, may be transferred between memory and other storage devices for memory management and data integrity purposes. Alternatively, in other embodiments, some or all of the software components may be executed in memory on another device and communicate with the illustrated computer system via inter-computer communication. Some or all of the system components or data structures may also be stored (e.g., as instructions or structured data) on a computer-accessible medium or portable item for retrieval by a suitable drive, various examples of which are described above. In some embodiments, instructions stored on a computer-accessible medium separate from computer system 1300 may be transmitted to computer system 1300 via a transmission medium or signal (such as an electrical, electromagnetic, or digital signal), transmitted via a communication medium such as a network or wireless link. Various embodiments may also include receiving, transmitting, or storing instructions or data implemented according to the foregoing description on a computer-accessible medium. Therefore, this technology can be practiced using other computer system configurations.

[0161] In the block diagram, the components shown are depicted as discrete functional blocks, but the embodiments are not limited to systems in which the functions described herein are organized as shown. The functionality provided by each of the components may be provided by software or hardware modules that are organized differently from those currently described; for example, such software or hardware may be mixed, combined, replicated, decomposed, distributed (e.g., distributed within a data center or geographically), or otherwise organized differently. The functions described herein may be provided by one or more processors of one or more computers executing code stored on a tangible, non-transitory machine-readable medium. In some cases, although the singular term "medium" is used, instructions may be distributed across different storage devices associated with different computing devices, for example, where each computing device has a different subset of instructions, consistent with the use of the singular term "medium" herein. In some cases, a third-party content delivery network may host some or all of the information transmitted over the network, in which case the information (e.g., content) may be provided by sending instructions to retrieve it from the content delivery network, to the extent that the information is supplied or otherwise provided.

[0162] Readers should understand that this application describes several independently useful techniques. Instead of separating these techniques into multiple independent patent applications, the applicant has grouped them into a single document because their related subject matter is useful for the economy of the application process. However, the different advantages and aspects of these techniques should not be confused. In some cases, embodiments resolve all the deficiencies mentioned herein; however, it should be understood that these techniques are independently useful, and some embodiments resolve only a subset of these problems or provide other unmentioned benefits that are obvious to those skilled in the art upon reading this disclosure. Due to cost constraints, some techniques disclosed herein may not be claimed at present but may be claimed in a later application, such as a continuation or by amending these claims. Similarly, due to space limitations, the abstract and summary of the inventive section of this document should not be considered a comprehensive list containing all such techniques or all aspects of these techniques.

[0163] It should be understood that the specification and drawings are not intended to limit the invention to the specific forms disclosed, but rather, the invention is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention as defined by the appended claims. In view of this description, further modifications and alternative embodiments of various aspects of the invention will be apparent to those skilled in the art. Therefore, this specification and drawings are to be interpreted only as illustrative and to teach those skilled in the art the general manner of practicing the invention. It should be understood that the forms of the invention shown and described herein will be considered as examples of embodiments. Elements and materials may be substituted for those shown and described herein, parts and processes may be reversed or omitted, and certain features of the technology may be used independently, all of which will be apparent to those skilled in the art who benefit from the description of the technology. Changes may be made to the elements described herein without departing from the spirit and scope of the technology as described in the appended claims. The headings used herein are for organizational purposes only and are not intended to limit the scope of the description.

[0164] As used throughout the application, the word “may” is used in a permissive sense (i.e., meaning possible), not in a mandatory sense (i.e., meaning must). The words “include,” “including,” “includes,” etc., mean including but not limited to. As used throughout the application, the singular forms “a,” “an,” and “the” include plural indicators unless explicitly stated otherwise. Thus, for example, a reference to “element” includes a combination of two or more elements, although other terms and phrases, such as “one or more,” are used for one or more elements. Unless otherwise stated, the term “or” is non-exclusive, i.e., includes both “and” and “or.” Terms describing conditional relationships, such as “in response to X, Y,” “when X, Y,” “if X, Y,” “when X, Y,” etc., encompass causal relationships where the antecedent is a necessary causal condition, a sufficient causal condition, or a contributing causal condition to the result; for example, “state X occurs when condition Y is acquired” is a class of “X occurs only when Y” and “X occurs when Y and Z.” This conditional relationship is not limited to results obtained by directly following antecedent conditions, as some results may be deferred, and in conditional statements, antecedent conditions are linked to their results; for example, antecedent conditions relate to the probability of a result occurring. Statements that map multiple attributes or functions to multiple objects (e.g., one or more processors perform steps A, B, C, and D) encompass all such attributes or functions being mapped to all such objects, and subsets of attributes or functions being mapped to subsets of attributes or functions (e.g., all processors each perform steps A through D, and a case where processor 1 performs step A, processor 2 performs a portion of steps B and C, and processor 3 performs a portion of step C and step D), unless otherwise stated. Similarly, references to a “computer system” performing step A and a “computer system” performing step B can include the same computing device within the computer system performing both steps or different computing devices within the computer system performing steps A and B. Furthermore, unless otherwise stated, statements that a value or action is “based on” another condition or value encompass the case where the condition or value is the only factor and the case where the condition or value is one of multiple factors. Unless otherwise stated, statements that “each” instance of a certain set has a certain property should not be construed as excluding the possibility that certain identical or similar members of a larger set do not have that property; that is, each does not necessarily mean every. Limitations regarding the sequence of stated steps should not be construed as claims unless explicitly stated, for example using explicit language such as “after X is performed, Y is performed,” which, in contrast to statements that might be improperly interpreted as implying a sequence limitation (such as “X is performed on item X, Y is performed on item X”), are used for the purpose of making the claims more readable, rather than specifying a sequence.Statements involving "at least Z of A, B, and C" (e.g., "at least Z of A, B, or C") refer to at least Z of the listed categories (A, B, and C), and do not require at least Z units in each category. Unless otherwise stated, it is obvious from the discussion that it should be understood that throughout the discussion of this specification, the use of terms such as "processing," "computing," and "determining" refers to the action or process of a particular device, such as a dedicated computer or similar dedicated electronic processing / computing equipment. Features described with reference to geometric constructions (such as "parallel," "perpendicular / orthogonal," "square," "cylindrical," etc.) should be interpreted as encompassing items that substantially embody the properties of the geometric construction; for example, a reference to a "parallel" surface encompasses substantially parallel surfaces. The permissible range of deviations from the Platonic ideal of these geometric constructions is determined by referring to the scope in the specification, and where such ranges are not defined by industrial specifications in the field of use, and where such ranges are not defined by industrial specifications in the field of manufacture of the specified features, and where such ranges are not defined, the features that substantially embody the geometric construction should be interpreted as including those features within 15% of the defined properties of the geometric construction. The terms “first,” “second,” “third,” “given,” etc. (if used in the claims) are used to distinguish or otherwise identify, not to indicate an order or numbering limitation. As is the case in ordinary use in this art, data structures and formats described with reference to human-significant uses do not need to be presented in a human-understandable format to constitute the described data structure or format; for example, text does not need to be presented in Unicode or ASCII or even encoded to constitute text; images, maps, and data visualizations do not need to be displayed or decoded to constitute images, maps, and data visualizations respectively; speech, music, and other audio do not need to be emitted through a speaker or decoded to constitute speech, music, or other audio respectively. Computer-implemented instructions, commands, etc., are not limited to executable code and can be implemented in the form of data that causes a function to be invoked, for example, in the form of function arguments or API calls. The use of custom-designed noun phrases (and other inventive terms) in claims, and the lack of self-evident construction, means that the definition of such phrases can be stated in the claims themselves. In such cases, the use of such custom-designed noun phrases should not be considered as imposing additional limitations by reviewing the specification or external evidence.

[0165] In this patent, to the extent that any U.S. patent, U.S. patent application, or other material (e.g., an article) is incorporated herein by reference, the text of such material is incorporated herein only to the extent that there is no conflict between such material and the statements and figures set forth herein. In the event of such conflict, the text of this document shall prevail, and the terminology in this document shall not be interpreted as narrower due to how those terms are used in other materials incorporated by reference.

[0166] The present invention will be better understood by referring to the following examples:

[0167] 1. A non-transitory machine-readable medium storing instructions that, when executed by one or more processors, implement operations including: receiving a peer-to-peer connection initiation signal by a first client computing device to establish a peer-to-peer connection with a second client computing device; determining that peer-to-peer connection conditions are not met, such that the first client computing device is unavailable for establishing a peer-to-peer connection with the second client computing device; and storing the peer-to-peer connection initiation signal in a signal buffer associated with the first client computing device.

[0168] 2. The medium as described in Example 1, wherein these operations further include: determining, by a first client computing device, that a peer-to-peer connection condition exists for the first client computing device, such that the first client computing device can be used to establish a peer-to-peer connection with a second client computing device; determining, by the first client computing device, that a peer-to-peer connection initiation signal is in a signal buffer; processing, by the first client computing device, the peer-to-peer connection initiation signal from the signal buffer; and sending, by the first client computing device, a response to the peer-to-peer connection initiation signal to the second client computing device, wherein the response causes the first client computing device and the second client computing device to establish a peer-to-peer connection.

[0169] 3. The medium as described in any one of Embodiments 1 to 2, wherein determining whether a peer-to-peer connection is satisfied includes determining whether a proximity condition is satisfied, wherein the proximity condition includes determining whether a first participant associated with a first client computing device and located at a first position in a coordinate grid environment provided by the application is within a distance of a second participant associated with a second client computing device and located at a second position in a coordinate grid environment.

[0170] 4. The medium as described in Example 3, wherein these operations further include: determining the distance between a first participant and a second participant by a first client computing device using a coordinate grid state generated by the first client computing device and generated using a participant location payload provided by a state server computing device, wherein the participant location payload includes participant location information associated with each participant in the coordinate grid environment.

[0171] 5. The medium as described in Example 4, wherein the participant location information for each participant includes the participant's current location in the coordinate grid environment, the location indicated in the user input for the participant, and the time of the user input for the participant.

[0172] 6. The medium as described in any one of Embodiments 1 to 5, wherein these operations further include: receiving user input from a first client computing device to move a first participant associated with the first client computing device and located at a first position in a coordinate grid environment provided by an application to a second position; and sending participant location information for the first participant to a state server computing device from the first client computing device, wherein the participant location information includes the first position, the second position, and the time input by the user.

[0173] 7. The medium as described in any one of Embodiments 1 to 6, wherein these operations further include: the first client computing device sending a second peer connection initiation signal to establish a second peer connection with the third client computing device; and the first client computing device establishing a second peer connection with the third client computing device in response to receiving a second peer connection response signal.

[0174] 8. The medium as described in Example 7, wherein these operations further include: determining by the first client computing device that the second peer connection conditions are met; and in response, performing the transmission of a second peer connection initiation signal to establish the second peer connection.

[0175] 9. The medium as described in Example 8, wherein determining whether a second peer connection is satisfied includes determining whether a proximity condition is satisfied, wherein the proximity condition includes determining whether a first participant associated with a first client computing device and located at a first position in a coordinate grid environment provided by the application is within a distance of a third participant associated with a third client computing device and located at a third position in a coordinate grid environment.

[0176] 10. The medium as described in any one of Examples 1 to 9, wherein the peer connection initiation signal is a Session Description Protocol (SDP) proposal based on the Interactive Connection Establishment (ICE) protocol provided by the Real-Time Communication (RTC) Application Programming Interface (API).

[0177] 11. A system comprising: one or more processors; and one or more computer-readable media storing instructions that, when executed by the one or more processors, implement operations including the operations according to any one of embodiments 1 to 10.

[0178] 12. A method comprising: operations according to any one of embodiments 1 to 10.

[0179] 13. A non-transitory machine-readable medium storing instructions, which, when executed by one or more processors, implement operations including: determining, by a first client computing device, that a proximity condition is satisfied, wherein the proximity condition includes determining whether a first participant associated with the first client computing device and located at a first position in a coordinate grid environment provided by an application is within a distance of a second participant associated with a second client computing device and located at a second position in the coordinate grid environment; the first client computing device, in response to the satisfaction of the proximity condition, sending a first peer-to-peer connection initiation signal to establish a first peer-to-peer connection with the second client computing device; and the first client computing device establishing the first peer-to-peer connection with the second client computing device in response to receiving a first response to the first peer-to-peer connection initiation signal.

[0180] 14. The medium as described in Example 11, wherein these operations further include: a first client computing device using a coordinate grid state generated by the first client computing device to determine the distance between a first participant and a second participant, wherein the coordinate grid state is generated using a participant location payload provided by a state server computing device, wherein the participant location payload includes participant location information associated with each participant in the coordinate grid environment.

[0181] 15. The medium as described in Example 14, wherein the participant location information for each participant includes the participant's current location in the coordinate grid environment, the location indicated by user input for the participant, and the time of user input for the participant.

[0182] 16. The medium as described in any one of Embodiments 13 to 15, wherein the operations further include: receiving a second peer-to-peer connection initiation signal by a first client computing device to establish a second peer-to-peer connection with a third client computing device; determining that the second peer-to-peer connection conditions are not met by the first client computing device, such that the first client computing device cannot be used to establish a peer-to-peer connection with the third client computing device; and storing the second peer-to-peer connection initiation signal in a signal buffer associated with the first client computing device by the first client computing device.

[0183] 17. The medium as described in Example 16, wherein these operations further include: determining, by the first client computing device, that a second peer-to-peer connection condition exists for the first client computing device, such that the first client computing device can be used to establish a second peer-to-peer connection with the third client computing device; determining, by the first client computing device, that a second peer-to-peer connection initiation signal is in a signal buffer; processing, by the first client computing device, the second peer-to-peer connection initiation signal; and sending, by the first client computing device, the second peer-to-peer connection initiation signal to the third client computing device, wherein the response causes the first client computing device and the third client computing device to establish a second peer-to-peer connection.

[0184] 18. The medium as described in Example 16, wherein determining whether a second peer connection is satisfied includes determining whether a proximity condition is satisfied, wherein the proximity condition includes determining whether a first participant associated with a first client computing device and located at a first position in a coordinate grid environment provided by the application is within a distance of a third participant associated with a third client computing device and located at a third position in a coordinate grid environment.

[0185] 19. The medium as described in Example 18, wherein these operations further include: determining the distance between a first participant and a third participant by a first client computing device using a coordinate grid state generated by the first client computing device and generated using a participant location payload provided by a state server computing device, wherein the participant location payload includes participant location information associated with each participant in the coordinate grid environment.

[0186] 20. The medium as described in Example 17, wherein the participant location information for each participant includes the participant's current location in the coordinate grid environment, the location indicated by user input for the participant, and the time indicated by user input for the participant.

[0187] 21. The medium as described in any one of Embodiments 13 to 20, wherein these operations further include: receiving user input from a first client computing device to move a first participant associated with the first client computing device and located at a first position in a coordinate grid environment provided by an application to a second position; and sending participant location information for the first participant to a state server computing device from the first client computing device, wherein the participant location information includes the first position, the second position, and the time input by the user.

[0188] 22. A system comprising: one or more processors; and one or more computer-readable media storing instructions that, when executed by the one or more processors, implement operations including the operations according to any one of embodiments 13 to 21.

[0189] 23. A method comprising: the operation according to any one of embodiments 13 to 21.

Claims

1. A non-transitory machine-readable medium storing instructions that, when executed by one or more processors, perform operations including: The first client computing device receives a peer-to-peer connection initiation signal to establish a peer-to-peer connection with the second client computing device; The first client computing device determines whether the proximity condition is met; When the proximity condition is not met, the first client computing device is not available to establish the peer connection with the second client computing device, and the first client computing device stores the peer connection initiation signal in a signal buffer associated with the first client computing device. as well as When the proximity condition is met, the first client computing device sends the peer connection initiation signal to establish the peer connection with the second client computing device.

2. The medium according to claim 1, wherein, The operation also includes: The first client computing device determines that the proximity condition exists for the first client computing device, so that the first client computing device can be used to establish the peer-to-peer connection with the second client computing device; The first client computing device determines that the peer connection initiation signal is in the signal buffer; The peer connection initiation signal from the signal buffer is processed by the first client computing device; and The first client computing device sends a response to the second client computing device initiating the peer connection, wherein the response causes the first client computing device and the second client computing device to establish the peer connection.

3. The medium according to claim 1, wherein, The proximity condition includes determining whether a first participant, associated with the first client computing device and located at a first position within a coordinate grid environment provided by the application, is within a distance of a second participant, associated with the second client computing device and located at a second position within the coordinate grid environment.

4. The medium according to claim 3, wherein, The operation also includes: The distance between the first participant and the second participant is determined by the first client computing device using a coordinate grid state generated by the first client computing device and using a participant location payload provided by the state server computing device, wherein the participant location payload includes participant location information associated with each participant in the coordinate grid environment.

5. The medium according to claim 4, wherein, The participant location information for each participant includes the participant's current location in the coordinate grid environment, the location indicated in the participant's user input, and the time indicated in the participant's user input.

6. The medium according to claim 1, wherein, The operation also includes: The first client computing device receives user input to move a first participant, associated with the first client computing device and located in a coordinate grid environment provided by the application, from a first position to a second position; and The first client computing device sends participant location information for the first participant to the state server computing device, wherein the participant location information includes the first location, the second location, and the time input by the user.

7. The medium according to claim 1, wherein, The operation also includes: The first client computing device sends a second peer-to-peer connection initiation signal to establish a second peer-to-peer connection with the third client computing device; and In response to receiving a second peer connection response signal, the first client computing device establishes a second peer connection with the third client computing device.

8. The medium according to claim 7, wherein, The operation also includes: The first client computing device determines that the second proximity condition is met; and in response, it sends the second peer connection initiation signal to establish the second peer connection.

9. The medium according to claim 8, wherein, Determining whether the second peer connection is satisfied includes determining whether a proximity condition is satisfied, wherein the proximity condition includes determining whether a first participant associated with the first client computing device and located at a first position in a coordinate grid environment provided by the application is within a distance of a third participant associated with the third client computing device and located at a third position in the coordinate grid environment.

10. The medium according to claim 1, wherein, The peer connection initiation signal is based on the Session Description Protocol (SDP) proposal of the Interactive Connection Establishment (ICE) protocol provided by the Real-Time Communication (RTC) Application Programming Interface (API).

11. A non-transitory machine-readable medium storing instructions that, when executed by one or more processors, perform the following operations: The proximity condition is determined by the first client computing device, whereby... The proximity condition includes determining whether a first participant, associated with the first client computing device and located at a first position within a coordinate grid environment provided by the application, is within a distance of a second participant, associated with the second client computing device and located at a second position within the coordinate grid environment; When the proximity condition is not met, the first client computing device is not available to establish a first peer connection with the second client computing device, and the first client computing device stores the first peer connection initiation signal in a signal buffer associated with the first client computing device. When the proximity condition is met, the first client computing device sends a first peer connection initiation signal in response to the satisfaction of the proximity condition to establish the first peer connection with the second client computing device; as well as In response to receiving a first response signal initiating the first peer connection, the first client computing device establishes the first peer connection with the second client computing device. The operation further includes: The first client computing device receives a second peer connection initiation signal to establish a second peer connection with the third client computing device; The first client computing device determines that the second proximity condition is not met, making the first client computing device unusable for establishing a peer-to-peer connection with the third client computing device; and The first client computing device stores the second peer connection initiation signal in a signal buffer associated with the first client computing device.

12. The medium according to claim 11, wherein, The operation also includes: The first client computing device determines that there is a second proximity condition for the first client computing device, so that the first client computing device can be used to establish a second peer-to-peer connection with the third client computing device; The first client computing device determines that the second peer connection initiation signal is in the signal buffer; The second peer connection initiation signal is processed by the first client computing device; and The first client computing device sends a second peer connection initiation signal to the third client computing device, wherein the response causes the first client computing device and the third client computing device to establish the second peer connection.

13. The medium according to claim 11, wherein, Determining whether the second peer connection is satisfied includes determining whether the proximity condition is satisfied, wherein the proximity condition includes determining whether the first participant, associated with the first client computing device and located at the first position in the coordinate grid environment provided by the application, is within a distance of the third participant, associated with the third client computing device and located at the third position in the coordinate grid environment.

14. The medium according to claim 13, wherein, The operation also includes: The distance between the first participant and the third participant is determined by the first client computing device using a coordinate grid state generated by the first client computing device and using a participant location payload provided by the state server computing device, wherein the participant location payload includes participant location information associated with each participant in the coordinate grid environment.

15. The medium according to claim 14, wherein, The participant location information for each participant includes the participant's current location in the coordinate grid environment, the location indicated by user input for the participant, and the time indicated by user input for the participant.

16. A method comprising: The first client computing device receives a peer-to-peer connection initiation signal to establish a peer-to-peer connection with the second client computing device; The first client computing device determines whether the proximity condition is met; When the proximity condition is not met, the first client computing device is not available to establish the peer connection with the second client computing device, and the first client computing device stores the peer connection initiation signal in a signal buffer associated with the first client computing device. as well as When the proximity condition is met, the first client computing device sends the peer connection initiation signal to establish the peer connection with the second client computing device.

Images