Console of game publisher, cloud service client, invitation of remote player via remote play application, or its connection control

Abstract

To provide a console of a game publisher, a cloud service client, an invitation of remote player via a remote play application, or its connection control.SOLUTION: A method and a system are provided so as to improve a remote computing experience. The method utilizes a corrected PC or console architecture for validating both of local and remote console connection. The system is linked to multiple devices via peer-to-peer connection. In game context, the system is engineered so as to cope with a low up stream rate and restricted bandwidth, or an obstacle, such as the quality of services. An AI trained regarding a play style is used as a substitute for a low-bandwidth client.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present disclosure relates to sending and receiving content, content collaboration between local and / or remote users, peer-to-peer communications, gaming, peer-to-peer gaming, peer-to-peer game streaming, peer-to-peer remote play, and the like. Summary of the Invention [Means for solving the problem]

[0002] Remote play over mobile or Wi-Fi has numerous issues and drawbacks. Remote play requires a minimum bandwidth of approximately 5 Mbps, but greater than or equal to approximately 15 Mbps is preferred for optimal performance, regardless of network type. This bandwidth requirement applies to mobile, including wide area networks (WANs) covering mobile, digital subscriber lines (DSL), data services interface specifications over cable (DOCSIS), mobile fixed lines, and the like. Existing remote play operates through a mobile phone application (or "app") that connects to a given console. The app first searches for a local area network (LAN) and, if it cannot find one, looks for an internet connection. Once a connection is made, the app checks for a network.

[0003] However, several problems exist with this approach. For example, if the bandwidth falls below the minimum required by a given console or game (e.g., below about 5 Mbps), remote play is terminated. This can lead to high latency (e.g., observed on existing PS5 systems) and undesirable data consumption, especially for those with limited data on their mobile plans. Even the simplest games, such as golf, are hindered by high latency. Furthermore, if the battery charge of a remote player's controller or mobile phone connected via existing remote play runs out, the game simply ends on the other side. For disc-based games, the disc must be in the console for validation. Multiple players are not allowed to play the game remotely, leading to an unsatisfactory overall playing experience.

[0004] Steam, a platform for remote play, contributes to the problem. Originally, it was limited to LANs, and all encoding configuration was manual; that is, bitrates were hard-coded. Resolutions were set for streaming to connected monitors on personal computers (PCs), and bitrates could not be adapted. Steam Remote Play enables a multiplayer experience, but it comes with significant drawbacks, including an inability to adjust for uplink bandwidth. Steam forces users to establish bitrates. Furthermore, Steam streams games at the display resolution set by the host machine's game settings, which can cause problems at lower uplink bitrates. This forces the host user to play the game at the same resolution required for the encoding resolution, leading to a substandard user experience for the host machine's user. Steam first streams to a cloud server and then to the remote player. The stream is then transcoded to match the remote player's downlink bitrate, introducing significant latency.

[0005] Additionally, Steam's port binding is unreliable; there is no indication of whether Steam is able to bind the discovery port, i.e., port 27036. If it fails, the other computer will not appear in the remote computer list in Remote Play settings. Games may time out but still launch. If the game loses focus, Steam streams the desktop screen. There is limited support for non-Steam games, and surround sound is converted to stereo. Some features, such as audio recording, are not fully supported, and older DirectX controllers and games are not supported. Streaming may not perform as well when streaming to older systems with single- or dual-core CPUs and without hardware-accelerated H264 decoding. Windows XP streaming is not available, and a User Account Control (UAC) dialog blocks streaming. On Mac OS®, streaming is only supported from Mac OS 10.8 (2012) onwards. On SteamOS and Linux® builds, device files that allow userspace processes to generate and send input events to the kernel input subsystem (e.g., / dev / uinput (older interface) or / dev / input / uinput (newer interface)) must be readable and writable by Steam. Linux rumble support is non-existent; therefore, existing multiplayer streaming is unsatisfactory.

[0006] Another approach, Self-Clocked Rate Adaptation for Multimedia (SCReAM), does not associate encoding properties with bitrate change requests to the encoder.

[0007] Several improvements to remote play are provided to enhance the remote play experience. For example, the system (including modified PC or console architecture) supports local and / or remote console play with local and / or remote friends. Throughout this disclosure, the term "friend" or "friends," when used, should be assumed to refer more broadly to users or participants and the like, who are not necessarily friends in the strict sense, and more specifically, to one or more devices operated by users and participants, as the context requires. That is, the term "friend" or "friends" is used colloquially and for convenience and should not be construed as limiting.

[0008] In some embodiments, the system supports multiple relatively "light" devices (e.g., controllers) or smartphones connected via apps with a relatively "heavy" PC or console that handles processing, rendering, and / or gameplay loads, variable and / or modulated resolutions, frame rates, High Efficiency Video Coding (also known as H.265, HEVC) at relatively low bitrates (low Mbs), and / or HEVC at relatively high bitrates (high Mbs). Cloud services for console game providers and PC game publishers are improving. In some embodiments, device improvements may not be necessary as the system provides satisfactory remote gameplay for multiple "light" clients.

[0009] For example, one method is provided for playing a game with a friend using a game console, either locally or remotely. The game console acts as a server and can directly connect with one or more remote devices (such as the friend's console or computer). The game console sends a set of instructions (or QoE definition) for the game being played. The game console then processes this QoE definition into a map that links encoding bitrates (the speed at which data is transmitted) to encoding properties (how the data is formatted). A controller in the game console, which manages video encoding rates and properties, receives this map. The controller also receives information about the length of queues of data packets waiting to be transmitted, the bitrate of the audio being encoded, and bitrates from a multiplexer (which combines multiple signals into one) to determine audio encoding and multiplexing overhead bitrates. The controller processes all of this information and determines a target video encoding bitrate and target video encoding properties. The controller then sends these targets to the game console's video encoder, which is connected to a transmitter that transmits the data, for example, using the Real-Time Transport Protocol (RTP). Finally, the transmitter transmits the multiplexed audio and video, encoded with the target video encoding properties at the target video encoding bitrate, to one or more receivers and decoders on a remote client device. In short, this is a way for a game console to manage how game data is sent to other devices, aiming to provide a good gaming experience whether playing on the console or remotely on another device. It takes into account various factors such as the speed of data transmission and the format of the data, ensuring that the game runs smoothly.

[0010] Systems and methods are provided to overcome the relatively low upstream rate as a limiter. Other types of implementations include security systems, extended reality environments, and servers, where there is one encoder and one network transmitter that transmits multimedia streams with video to multiple remote clients with ultra-low latency. Overall remote play and remote operability are improved for users.

[0011] In some embodiments, an improved game console is provided as a robust device designed to accommodate not only four native controllers, but also one, two, three, or multiple lightweight remote devices, whether they are remote or not. These lightweight remote devices are controllers that send and receive human input. The game console achieves ultra-low latency, which ensures a smooth and responsive gaming experience. The encoding process is designed to accommodate all clients. The encoding process uses encoder encoding at a single bitrate, with a frame rate and resolution optimized for the bitrate selected to send to the encoder, as opposed to adaptive bitrate (ABR) encoding, for example. For example, in some embodiments, only one encoder and only one RTP streamer distribute to multiple RTP receivers and handle bitrate control. The system determines what to encode based on the requirements of all clients.

[0012] In some embodiments, one client has very low bandwidth or low Quality of Service (QoS). In such cases, the system determines what to do when the bandwidth drops below a certain threshold, for example, about 5 Mbps. There are some games, such as team hockey, that are completely ruined for all players by this kind of drop in bandwidth. To address this issue, the system proposes alternatives. One option is for another player to take over the controller. Another option is for the low-bandwidth client to be relegated to "spectator mode" and replaced with artificial intelligence (AI) that has been trained on the low-bandwidth client's playing style.

[0013] In some embodiments where QoS is low, the system switches the player to an ABR server / encoder. This may introduce delays and cause the player to fall behind real time, but the player can still watch the game, although they cannot play it. For example, the system switches to an ABR server / encoder for one or more players whose bandwidth is below a threshold. The system operates normally with respect to other players. In this example, the particular game session also uploads video to a third-party distribution service (e.g., a content delivery network) so that the particular player (currently in spectator mode) can still watch the game.

[0014] In some embodiments, a cloud is provided to establish a peer-to-peer connection from lightweight devices to heavyweight game consoles. This peer-to-peer connection helps reduce latency for streaming to and from intermediate cloud services. This approach ensures that all players can participate in the gaming experience regardless of their bandwidth or QoS.

[0015] In some embodiments, a process for local and / or remote play via a game console is provided, the process including accessing a game title Quality of Experience (QoE) remote play definition, determining a bitrate / encoding property mapping, and determining a target video encoding bitrate and target video encoding properties.

[0016] In some embodiments, processes are provided for encoding and transmitting video and audio via RTP packets, processing received Real-Time Control Protocol (RTCP) packets and new or removed Source Internet Protocol (SRC IP) connections, and managing the congestion window (CWND) and round trip time (RTT) for each source RTP packet.

[0017] In some embodiments, a process is provided for connecting a single encoder and RTP sender to multiple client devices via a User Datagram Protocol (UDP) delivery port, involving starting and rendering video at a video source, encoding video and audio at a defined bitrate, receiving audio and video packetized elementary streams (PES) at a multiplexer, and adjusting the video encoding bitrate based on changes in the size of the packet priority queue as a result of a transmission scheduler sending RTP packets. In some embodiments, rate and quality are expected to spike, assuming all clients have adequate QoS. For example, if a client joins with poor QoS or begins to experience poor QoS, the video encoded bitrate will drop for all connected clients in some embodiments.

[0018] In some embodiments, a process is provided for sending a new SRC IP address connection to a network congestion controller and adding it to a new packet response data store for the SRC IP address.

[0019] In some embodiments, a process is provided for inviting remote players and controlling remote player connections via a game publisher's console device or cloud service client. The process includes using a credential to log a profile associated with the console or game owner into the console device or PC Game Publisher Cloud Service (CD / CS), and sending a user-authenticated response along with the user identification (shown in the figures as "User_ID") to the CD / CS as a User Authentication and / or User Search and / or Friend Connection (UA / US / FC) response to the UA / US / FC controller of the CD / CS. In some embodiments, the cloud service is a PC Game Publisher Cloud Service (PCGPCS), which is shown in some of the figures.

[0020] In some embodiments, a process is provided for local and / or remote play via a game console, where the console acts as a server and forms a peer-to-peer connection with a remote client device. The process includes at least one of the following steps in a game engine: accessing a game title QoE remote play definition; determining a bitrate / encoding property mapping; accessing the mapping; determining a queue length based on a priority queue of RTP packets; determining an audio and / or video bitrate and a multiplexer bitrate; determining a target video encoding bitrate and target video encoding properties; accessing the target video encoding bitrate and target video encoding properties; encoding video with the target video encoding properties according to the target video encoding bitrate; or transmitting the encoded video to one or more RTP receivers and / or decoders of one or more remote client devices. The game title QoE remote play definition, in some examples, includes a frame rate and resolution for a range of bandwidth conditions. The process also involves encoding and transmitting video and audio RTP packets, processing received RTCP packets, and managing a CWND and RTT for each source packet. In response to determining that the packet response data store is waiting for an RTCP response packet, the SRC IP address is saved in the network congestion controller as the worst-case client device, and the CWND and RTT are then sent to the game console's transmission scheduler for the worst-case client device or last client device to respond with an RTCP packet.The method further includes, in some embodiments, determining and transmitting (e.g., from the transmission scheduler to a UDP socket) the RTP-multiplexed encoded video and audio packets, and then transmitting them to the remote client device. The process also includes, in some embodiments, receiving RTCP packets at the UDP socket from the remote client device along with the SRC IP address.

[0021] In some embodiments, a process is provided for local and / or remote play via a game console that includes at least one of the following steps: receiving and processing RTCP packets and existing, new, or removed SRC IP connections in a network congestion controller; sending the RTCP packets to a packet response data store; processing, for each SRC, a synchronization source (SSRC), a transmission timestamp (TS(TX)), a Real-time Transport Protocol sequence number (RTP(SN)), an RTP packet (RTP(size)), and the RTCP packets into a CWND and RTT; or sending the CWND and RTT for each source packet to a transmission scheduler. In some embodiments, the process also involves at least one of the following steps: transmitting a game title QoE remote play definition from the game engine, processing the definition into a bitrate / encoding property mapping, receiving in a rate and video encoding property controller the mapping, a queue length from an RTP packet priority queue, an audio bitrate, and a multiplexer bitrate, and processing them into a target video encoding bitrate and target video encoding properties, transmitting the target video encoding bitrate and target video encoding properties to a video encoder of the game console operatively connected to an RTP transmitter of the game console, encoding video according to the target video encoding bitrate with the target video encoding properties, or transmitting the encoded video via the RTP transmitter to one or more RTP receivers and decoders of one or more remote client devices. In some embodiments, the game title QoE remote play definition includes a frame rate and a resolution for a range of bandwidth conditions.The process also involves, in some embodiments, encoding and transmitting video and audio, and sending the RTP-multiplexed encoded video and audio packets to an RTP packet priority queue and a transmission scheduler. The method further involves, in some embodiments, sending the RTP-multiplexed encoded video and audio packets from the transmission scheduler to a UDP socket and then to a remote client device. The process also includes, in some embodiments, receiving RTCP packets at the UDP socket from the remote client device along with the SRC IP address.

[0022] In some embodiments, a process is provided for connecting a single encoder and RTP sender to multiple client devices via a UDP delivery port, the process including the steps of: starting and rendering video at a video source; encoding video and audio at a defined bit rate; adjusting the video encoding bit rate based on a change in size of a packet priority queue as a result of a transmission scheduler transmitting RTP packets; sending a new SRC IP address connection to a network congestion controller; adding a new packet response data store for the SRC IP address; transmitting the RTP-multiplexed encoded video and audio packets to the packet priority queue and the UDP delivery port for transmission; receiving RTCP packets at a UDP socket; removing the RTCP packets from the packet response data store for the SRC IP address of the RTCP packets; and, if the packet response data store receives the RTCP response of the last client device, removing the SRC IP address from the packet response data store. The process includes at least one of the following steps: saving the IP address as a worst-case client device, sending the CWND and RTT to a transmission scheduler, monitoring the queue length of a priority queue of packets, calculating a new video encoding rate based on the audio rate and the multiplexed bit rate, if rate adjustment is required, accessing encoding properties based on the multiplexer bit rate and the audio bit rate, sending a new target video encoding bit rate and new target video encoding properties to a video encoder, or sending the multiplexed audio and video PES streams to an RTP sender and sending the multiplexed bit rate to a rate and video encoding property controller. In some embodiments, the process includes processing an RTCP response from an RTP receiver on one of the remote client devices in network congestion control.In some embodiments, the process includes, in response to the last remote client device responding with an RTCP packet with the expected RTP sequence number, removing the RTCP packet from the packet response data store for the RTCP packet's SRC IP address. In some embodiments, the multiplexed bit rate is automatically adjusted based on the audio encoded bit rate and the video encoded bit rate. The multiplexed audiovisual stream, for example, has additional overhead. In some embodiments, the additional overhead is taken into account when requesting a bit rate change from the encoder.

[0023] In some embodiments, a process is provided for inviting remote players and controlling remote player connections via a game publisher's console device or cloud service client. The process includes the steps of logging a profile associated with the console or game owner into the CD / CS using a certificate, sending a user authenticated response with the user identification to the CD / CS and a UA / US / FC response to the UA / US / FC controller of the CD / CS, sending a friend list request with the user identification to the console game provider or cloud service (CGP / CS;In response to a user selecting an option to invite a remote player, sending a user list to the UA / US / FC controller along with the username and user identification response; in response to a user selecting one or more friends from the user list to invite, adding the selected user identification to an invitation list for the user to invite in the CD / CS; in response to the user inputting a name and the user selecting a search, sending a search request along with the username and user identification. and sending the invitation to the UAAS; in response to the user adding the retrieved user to the invite list, at the CD / CS, adding the selected user identification to an invite list for the user to invite; in response to the user selecting the send invitation selection option, sending a remote session request to the remote player management controller of the CGP / CS along with the user identification and remote session identifier ("remote_session_ID"); sending the invitation to the remote player management controller of the CGP / CS along with the user identification list, the requesting user name and user identification, the RTP streaming connection endpoint (shown in the drawing as "address:port"), and the controller connection UDP the steps of sending a user invite request to each invited user along with the requester username and user identification, the RTP streaming connection endpoint, and the controller connection UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and remote session identifier to the remote player management controller of the CGP / CS; or sending a user authentication, user lookup, and UA / US / FC management controller user login along with the certificate to the UAAS of the CGP / CS;

[0024] In some embodiments, a process is provided for inviting remote players and controlling remote player connections via a remote play application, the process includes the steps of logging a user into a remote client application and sending credentials to a UAAS, authenticating the user at the UAAS and sending a user identification to the remote play application, requesting a user device from a CGP / CS at the remote play controller, initiating a remote service at the CGP / CS in response to a user selection of remote player management and exchanging connection details with the remote play controller, prompting the user to invite friends by adding the user identification to an invite list, and in response to a user selection of a name lookup, sending a request to the UAAS, which responds with the user list. The method includes at least one of: adding a user identification to an invitation list in response to a user selection to add the searched user to the invitation list; sending an invitation request from the remote play controller to the CGP / CS in response to a user selection to send an invitation, which transmits the invitation request to each invited user; connecting the remote play controller to the console device and initiating interaction through a remote display; sending a remote session request from the remote play controller to the CGP / CS, which stores the remote connection IP address; or sending a user invitation request from the CGP / CS to each invited user.

[0025] In some embodiments, a process is provided for a remote player to accept a remote play invitation, the process including at least one of the following steps: logging the user into a remote client application and sending user credentials to a UAAS of a CGP / CS; logging the remote user into the CGP / CS in response to the remote user selecting to accept the remote play invitation; sending an invitation acceptance response from a remote play controller to the CGP / CS, which stores the remote user's IP address; attempting to connect to a controller port in the remote play application; sending a message to the remote user's device in response to an unsuccessful connection to the controller port; sending controller input from the remote application to the console device in response to a successful connection to the controller port; receiving tactile feedback in the remote application from the console device; connecting in the remote play controller to an RTP streaming UDP connection endpoint of the console device; or decoding and rendering content for display.

[0026] In some embodiments, a process is provided for initial controller connection for local and / or remote users. The process includes the steps of locally powering on the console and controllers, assigning the locally powered-on controller as "Controller 1" (i.e., the console's primary controller), restricting primary control at the console to Controller 1, determining whether an additional controller has been powered on, and, if applicable, setting the console to a state in which it will not accept additional controller connections if all controller slots are in use, or, if all controller slots are not in use, assigning the additional controller to the next available controller spot, if an additional controller has not been powered on, determining whether a remote user has joined the controller UDP port, and, if applicable, determining whether all controller slots are in use, and, if all controller slots are in use, determining whether the remote user has a service account. the remote application, if applicable, unlocking an option to forcibly disconnect a connected local or remote user from a controller slot and an option to forcibly grant console control to controller 1; or, if all controller slots are not in use, determining whether the remote user has joined the CD / CS as a service account owner, and, if applicable, unlocking an option in the remote application to forcibly grant console control to controller 1 if other remote or local game controllers are connected; or, if other remote or local game controllers are not connected, sending controller 1 force mapping and controller number input mapping requests to a local or remote UDP port mapping controller input-output mapper.

[0027] In some embodiments, a process is provided for transferring a controller for local and / or remote play via a game console, the process including at least one of the following steps: determining whether all controller slots are filled and whether any remote users are connected to the remote play game console session; verifying whether the connected remote users are saved in the remote player management controller for the remote session identifier; determining whether the local player paused the game and selected a “controller transfer” option and the type of game in progress; sending a user list in a remote session request and receiving a user list remote session response based on the type of game in progress; displaying a list of remote users for user selection at the console; receiving a user selection to “transfer the controller”; sending a local controller transfer request and a “controller transfer” request; sending the “controller transfer” request to the remote player controller of the remote player application; connecting the remote player management controller of the remote playing application to a controller UDP connection endpoint; or sending a controller connected response from the remote playing controller of the remote playing application to the remote player management controller.

[0028] In some embodiments, processes and systems are configured for the particular type of game being played. For example, remote play is configured for multiplayer games, where all players in the game share the same video screen and control all characters present on the screen at all times. Remote play is also configured for multiplayer in the sense of games (e.g., "Call of Duty") where a multiplayer server exists and allows players to connect to a multiplayer session where all players are playing on their own devices or through a cloud subscription. Furthermore, remote play is configured for multiplayer games, which can be played locally without an internet connection, where all multiplayer aspects are handled locally within the game engine.

[0029] In some embodiments, a process is provided for the console owner to claim main control for a user or force a disconnect when the controller port is full. The process includes at least one of the following steps: determining whether the console owner is logged in to the remote application and whether the slot for controller 1 on the CD / CS is occupied by a local or remote user; sending a controller 1 forced mapping request from the remote application to the remote player management controller of the CD / CS; in a controller input-output mapper, disconnecting the connected remote user from the mapped controller UDP socket and connecting it to the console owner's mapped controller UDP socket; if all UDP sockets are occupied, displaying a message to the remote user to be disconnected and providing the console owner with a list of remote users connected to the controller; if all UDP sockets are unoccupied, in the controller input-output mapper, remapping the remote user's connected controller UDP socket to an unused controller connection and mapping the console owner's connected controller UDP socket connection to controller input 1; receiving a user's console owner selection to remove control; sending a forced controller mapping request for the remote user; disconnecting the connected remote user from the controller UDP socket mapped to the controller input for the remote user; or connecting the console owner's controller to the mapped controller UDP socket.

[0030] In some embodiments, a process is provided for low-bandwidth ABR for local and / or remote play via a game console. The process includes the steps of: sending a low-bandwidth notification to an RTCP reporting system if the target video encoding bitrate is below a minimum; sending a client device disconnect request to a remote play controller of the game console; forcing, by the remote play controller, a disconnect from the received address and port; if an ABR delivery controller session has not yet been initiated for the remote session identifier, sending the disconnect force request to the remote play client application; and determining whether the reported bitrate is below a minimum value based on the peer-to-peer RTP broadcast.If the bandwidth is high enough to reconnect to the UDP streaming session, sending a force disconnect request to the remote player management controller along with the remote session identifier, IP address, and minimum bit rate requirement request; sending a session reconnect request to the remote play client application; if a low-bandwidth client is not determined, sending an ABR session termination request to the ABR delivery controller; if a low-bandwidth client is determined, sending an ABR session start request to the ABR delivery controller; starting an ABR session instance using an RTP transmission receiver, a demultiplexer, an ABR transcoder, and an ABR segmenter; establishing an RTP connection via the RTP transmission receiver; receiving an RTP-multiplexed audiovisual packet stream at the RTP transmission receiver and dividing it into video and audio PES packets. The method includes at least one of the following steps: demultiplexing the video and audio PES streams into streams; sending the video and audio PES streams to an ABR transcoder and an ABR segmenter; generating a live manifest and writing segments to an ABR audiovisual segment and manifest storage device in the ABR segmenter; sending an ABR live manifest Uniform Resource Locator (URL) notification to a remote play client application; starting an instance of an ABR video player in the remote play client application and sending the ABR live manifest to the ABR video player; downloading and playing ABR segments and receiving live manifest updates in the ABR video player; or sending the calculated bitrate from the ABR video player to a remote play controller of a remote player management controller.

[0031] In some embodiments, a process is provided for providing local and remote play via a game console, prompting a user selection for inviting a remote player to a game session, and forming a peer-to-peer connection between the game console and a remote client device. The prompt, in some examples, is provided at the main game screen or main console interface after a pause command during the game session. The process also, in some examples, includes prompting a user selection for transferring control to a remote user during the game session, and determining whether all controller slots are filled, whether a remote user is connected to the remote play game console session, whether a local player has paused the game and selected the "controller transfer" option, and / or the type of game in progress. The process involves sending a user list in a remote session request based on the type of game in progress, receiving a user list remote session response, displaying a list of remote users at the console for user selection, receiving a user selection to “hand over the controller,” sending a local controller handover request, sending a “controller handover” request, connecting the remote player management controller to the controller UDP connection endpoint, and sending a controller connected response. The process also includes, in some embodiments, suspending the remote player's gameplay mode and placing the remote player in a spectator mode in response to detecting a low bandwidth connection associated with the remote player, modifying the appearance of displayed game elements associated with the remote player in the spectator mode, replacing game play by the remote player with artificial intelligence trained on the remote player's playing style, and / or terminating the remote player's spectator mode and returning the remote player to a gameplay mode in response to detecting a sufficient bandwidth connection associated with the remote player.

[0032] Associated systems for performing one or more of the above processes as detailed herein are also provided. Associated system architectures are also provided. Associated packets, manifests, tables, and data stores are also provided. Associated graphical user interfaces (GUIs) are also provided. Associated non-transitory computer-readable media having non-transitory computer-readable instructions encoded thereon that, when executed by control circuitry, cause the control circuitry to perform one or more of the above processes are also provided. Associated devices for performing one or more of the above processes are also provided. Means for performing one or more steps of one or more of the above processes are also provided.

[0033] The present invention is not limited to combinations of elements as listed herein, but may be assembled in any combination of elements as described herein.

[0034] These and other capabilities of the disclosed subject matter will be more fully understood after review of the following figures, detailed description, and claims. The present invention provides, for example, the following items. (Item 1) 1. A method for inviting remote players and controlling remote player connections via a game publisher's console device or cloud service client, the method comprising: Logging in a profile associated with the console or game owner to the console device or cloud service (CD / CS) using the certificate; Sending a user authenticated response, along with the user identification, from a user account and authentication system (UAAS) of a console game provider or cloud service (CGP / CS) to said CD / CS, and sending a user authentication, user lookup, or user connection (UA / US / UC) response to a UA / US / UC controller of said CD / CS; In response to a user selection of an option to invite a remote player from at least one of the console device, the game service client, or the paused game via a user interface, sending a participant list request from a UA / US / UC controller of the CD / CS to a UAAS of the CGP / CS, along with the user identification; The UAAS sends a user list to the UA / US / UC controller together with a user name and a user identification response. A method comprising: (Item 2) in response to a user selection of one or more participants to invite from the user list, adding, at the CD / CS, the selected user identities to an invitation list for the user to invite; In response to the user's name entry and the user's selection of a search, sending a search request from the UA / US / UC controller to the UAAS, along with the user's name and user identification; sending the user list, along with the user name and a null response, to the UAAS, and sending the machine user name and user identification to the UA / US / UC controller; The method according to the above item, comprising: (Item 3) In response to identifying the user identity within the UAAS, transmitting from the UAAS the user list along with a username and user identity response to the UAAS and transmitting the machine username and the user identity to the UA / US / UC controller; adding, at the CD / CS, the selected user identification to an invitation list for the inviting user in response to the user adding the retrieved user to the invitation list; The method according to any one of the preceding items, comprising: (Item 4) In response to a user selection of a send invitation option, sending a remote session request from a remote play controller of said CD / CS to a remote player management controller of said CGP / CS together with said user identification and remote session identifier; sending a remote session response, together with the remote session identifier, from the remote player management controller of the CGP / CS or the remote play controller of the CD / CS to the remote play controller of the CD / CS; The method according to any one of the preceding items, comprising: (Item 5) The method of any one of the preceding items, comprising sending an invitation from the remote play controller of the CD / CS to one or more client devices of one or more users to a remote player management controller of the CGP / CS, along with the user identification list, requester username and user identification, RTP streaming connection endpoints, and controller connection UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and remote session identifier. (Item 6) The method of any one of the preceding items, comprising sending a user invite request from the remote player management controller of the CGP / CS to each invited user, together with the requester username and user identification, the RTP streaming connection endpoint, and the controller connection UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and a remote session identifier. (Item 7) The method of any one of the preceding items, comprising sending the user authentication, user search, and UA / US / UC management controller user login, together with a certificate, from the CD / CS to the UAAS of the CGP / CS. (Item 8) 1. A method for inviting remote players and controlling remote player connections via a remote play application, the method comprising: Logging a user device into a remote client application and transmitting a credential associated with the user device to a User Account and Authentication System (UAAS); At the UAAS, authenticating the user device and sending a user identification of the user device to the remote play application; requesting, at the remote play controller, user devices from a console game provider or cloud service (CGP / CS), which responds with a list of device identifiers; initiating a remote service at said CGP / CS and exchanging connection details with said remote player controller in response to a user selection of remote player management; prompting the user device to invite participants by adding the user identification to an invitation list; In response to a user selection to retrieve a name, sending a request to the UAAS, which responds with a list of users; adding the user identification to the invitation list in response to a user selecting to add the retrieved user to the invitation list; In response to a user selection to send an invitation, sending an invitation request from said remote play controller to said CGP / CS, which transmits the invitation request to each invited user; connecting the remote play controller to a console device and initiating interaction through a remote display; sending a remote session request from said remote play controller to said CGP / CS, which stores a remote connection IP address; The CGP / CS sends a user invitation request to each user to be invited. A method comprising: (Item 9) 1. A system for inviting remote players and controlling remote player connections via a game publisher's console device or cloud service client, the system comprising: memory and communication ports; control circuitry communicatively coupled to the memory and the communication port, the control circuitry executing instructions; Logging in a profile associated with the console or game owner to the console device or cloud service (CD / CS) using the certificate; Sending a user authenticated response, along with the user identification, from a user account and authentication system (UAAS) of a console game provider or cloud service (CGP / CS) to said CD / CS, and sending a user authentication, user lookup, or user connection (UA / US / UC) response to a UA / US / UC controller of said CD / CS; In response to a user selection of an option to invite a remote player from at least one of the console device, the game service client, or the paused game via a user interface, sending a participant list request from a UA / US / UC controller of the CD / CS to a UAAS of the CGP / CS, along with the user identification; The UAAS sends a user list to the UA / US / UC controller together with a user name and a user identification response. control circuitry configured to A system comprising: (Item 10) the control circuitry executes the instructions; in response to a user selection of one or more participants to invite from the user list, adding, at the CD / CS, the selected user identities to an invitation list for the user to invite; In response to the user's name entry and the user's selection of a search, sending a search request from the UA / US / UC controller to the UAAS, along with the user's name and user identification; sending the user list, along with the user name and a null response, to the UAAS, and sending the machine user name and user identification to the UA / US / UC controller; 2. The system of claim 1, configured to: (Item 11) the control circuitry executes the instructions; In response to identifying the user identity within the UAAS, transmitting from the UAAS the user list along with a username and user identity response to the UAAS and transmitting the machine username and the user identity to the UA / US / UC controller; adding, at the CD / CS, the selected user identification to an invitation list for the inviting user in response to the user adding the retrieved user to the invitation list; 2. The system of claim 1, configured to: (Item 12) the control circuitry executes the instructions; In response to a user selection of a send invitation option, sending a remote session request from a remote play controller of said CD / CS to a remote player management controller of said CGP / CS together with said user identification and remote session identifier; sending a remote session response, together with the remote session identifier, from the remote player management controller of the CGP / CS or the remote play controller of the CD / CS to the remote play controller of the CD / CS; 2. The system of claim 1, configured to: (Item 13) the control circuitry executes the instructions; The system of any one of the preceding items is configured to send an invitation from the remote play controller of the CD / CS to one or more client devices of one or more users to the remote player management controller of the CGP / CS, along with the user identification list, requester username and user identification, RTP streaming connection endpoint, and controller connection UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and remote session identifier. (Item 14) the control circuitry executes the instructions; The system of any one of the preceding items is configured to send a user invite request from the remote player management controller of the CGP / CS to each invited user, along with the requester username and user identification, the RTP streaming connection endpoint, and the controller connection UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and remote session identifier. (Item 15) the control circuitry executes the instructions; The system of any one of the preceding items, configured to send the user authentication, user search, and UA / US / UC management controller user login, along with a certificate, from the CD / CS to the UAAS of the CGP / CS. (Item 16) 1. A system for inviting remote players and controlling remote player connections via a remote play application, the system comprising: memory and communication ports; control circuitry communicatively coupled to the memory and the communication port, the control circuitry executing instructions; Logging a user device into a remote client application and transmitting a credential associated with the user device to a User Account and Authentication System (UAAS); At the UAAS, authenticating the user device and sending a user identification of the user device to the remote play application; requesting, at the remote play controller, user devices from a console game provider or cloud service (CGP / CS), which responds with a list of device identifiers; initiating a remote service at said CGP / CS and exchanging connection details with said remote player controller in response to a user selection of remote player management; prompting the user device to invite participants by adding the user identification to an invitation list; In response to a user selection to retrieve a name, sending a request to the UAAS, which responds with a list of users; adding the user identification to the invitation list in response to a user selecting to add the retrieved user to the invitation list; In response to a user selection to send an invitation, sending an invitation request from said remote play controller to said CGP / CS, which transmits the invitation request to each invited user; connecting the remote play controller to a console device and initiating interaction through a remote display; sending a remote session request from said remote play controller to said CGP / CS, which stores a remote connection IP address; The CGP / CS sends a user invitation request to each user to be invited. control circuitry configured to A system comprising: (Summary) A method and system are provided for improving the remote computing experience. The method utilizes a modified PC or console architecture that enables both local and remote console connections. The system interfaces with multiple devices through peer-to-peer connections. In a gaming context, the system is engineered to address obstacles such as low upstream rates and limited bandwidth or quality of service. AI trained on playing style is used as a proxy for low-bandwidth clients. Methods are also provided for inviting remote players, transferring control to remote users, managing controller connections, and using low adaptive bitrates to address bandwidth situations. Improvements in connection, single encoding, and transmission processes contribute to a better computing and / or gaming experience for all players, regardless of their bandwidth or quality of service. Related networks, models, apparatus, devices, techniques, articles, architectures, packets, manifests, tables, data stores, and graphical user interfaces are also provided. [Brief explanation of the drawings]

[0035] The present disclosure, in accordance with one or more various embodiments, will be described in detail with reference to the following figures. The drawings are provided for illustrative purposes only and merely depict non-limiting examples and embodiments. These drawings are provided to facilitate understanding of the concepts disclosed herein and should not be considered as limiting the scope, scope, or applicability of these concepts. It should be noted that for clarity and ease of illustration, these drawings are not necessarily made to scale.

[0036] The embodiments herein may be better understood by reference to the following description in conjunction with the accompanying drawings, in which like reference numbers indicate the same or similar elements.

[0037] [Figure 1]FIG. 1 depicts a system for local and / or remote play via a game console, according to some embodiments of the present disclosure, where the console acts as a server and forms a peer-to-peer connection with remote client devices.

[0038] [Figure 2] FIG. 2 depicts a security system for connecting local and / or remote devices through a server that forms a peer-to-peer connection with a remote client device, according to some embodiments of the present disclosure.

[0039] [Figure 3] FIG. 3 depicts an extended reality (XR) system for connecting local and / or remote devices via a server that forms a peer-to-peer connection with a remote client device, according to some embodiments of the present disclosure.

[0040] [Figure 4-1] FIG. 4 depicts a system for local and / or remote play via a game console, according to some embodiments of the present disclosure, where the console acts as a server and forms a peer-to-peer connection with remote client devices. [Figure 4-2] FIG. 4 depicts a system for local and / or remote play via a game console, according to some embodiments of the present disclosure, where the console acts as a server and forms a peer-to-peer connection with remote client devices. [Figure 4-3] FIG. 4 depicts a system for local and / or remote play via a game console, according to some embodiments of the present disclosure, where the console acts as a server and forms a peer-to-peer connection with remote client devices. [Figure 4-4]FIG. 4 depicts a system for local and / or remote play via a game console, according to some embodiments of the present disclosure, where the console acts as a server and forms a peer-to-peer connection with remote client devices. [Figure 4-5] FIG. 4 depicts a system for local and / or remote play via a game console, according to some embodiments of the present disclosure, where the console acts as a server and forms a peer-to-peer connection with remote client devices. [Figure 4-6] FIG. 4 depicts a system for local and / or remote play via a game console, according to some embodiments of the present disclosure, where the console acts as a server and forms a peer-to-peer connection with remote client devices. [Figure 4-7] FIG. 4 depicts a system for local and / or remote play via a game console, according to some embodiments of the present disclosure, where the console acts as a server and forms a peer-to-peer connection with remote client devices. [Figure 4-8] FIG. 4 depicts a system for local and / or remote play via a game console, according to some embodiments of the present disclosure, where the console acts as a server and forms a peer-to-peer connection with remote client devices. [Figure 4-9] FIG. 4 depicts a system for local and / or remote play via a game console, according to some embodiments of the present disclosure, where the console acts as a server and forms a peer-to-peer connection with remote client devices.

[0041] [Figure 5] FIG. 5 depicts a packet for RTCP, according to some embodiments of the present disclosure.

[0042] [Figure 6-1]FIG. 6 depicts a process for bitrate and encoding quality control for delivering a single encoded stream from a single RTP sender to multiple client devices over an unmanaged network, according to some embodiments of the present disclosure. [Figure 6-2] FIG. 6 depicts a process for bitrate and encoding quality control for delivering a single encoded stream from a single RTP sender to multiple client devices over an unmanaged network, according to some embodiments of the present disclosure. [Figure 6-3] FIG. 6 depicts a process for bitrate and encoding quality control for delivering a single encoded stream from a single RTP sender to multiple client devices over an unmanaged network, according to some embodiments of the present disclosure. [Figure 6-4] FIG. 6 depicts a process for bitrate and encoding quality control for delivering a single encoded stream from a single RTP sender to multiple client devices over an unmanaged network, according to some embodiments of the present disclosure. [Figure 6-5] FIG. 6 depicts a process for bitrate and encoding quality control for delivering a single encoded stream from a single RTP sender to multiple client devices over an unmanaged network, according to some embodiments of the present disclosure. [Figure 6-6] FIG. 6 depicts a process for bitrate and encoding quality control for delivering a single encoded stream from a single RTP sender to multiple client devices over an unmanaged network, according to some embodiments of the present disclosure. [Figure 6-7]FIG. 6 depicts a process for bitrate and encoding quality control for delivering a single encoded stream from a single RTP sender to multiple client devices over an unmanaged network, according to some embodiments of the present disclosure.

[0043] [Figure 7-1] FIG. 7 depicts a system that focuses on user invitation by console owners, management of peer-to-peer connections from remote controller inputs, and receiving game-rendered streams from game consoles, in accordance with some embodiments of the present disclosure. [Figure 7-2] FIG. 7 depicts a system that focuses on user invitation by console owners, management of peer-to-peer connections from remote controller inputs, and receiving game-rendered streams from game consoles, in accordance with some embodiments of the present disclosure. [Figure 7-3] FIG. 7 depicts a system that focuses on user invitation by console owners, management of peer-to-peer connections from remote controller inputs, and receiving game-rendered streams from game consoles, in accordance with some embodiments of the present disclosure. [Figure 7-4] FIG. 7 depicts a system that focuses on user invitation by console owners, management of peer-to-peer connections from remote controller inputs, and receiving game-rendered streams from game consoles, in accordance with some embodiments of the present disclosure. [Figure 7-5] FIG. 7 depicts a system that focuses on user invitation by console owners, management of peer-to-peer connections from remote controller inputs, and receiving game-rendered streams from game consoles, in accordance with some embodiments of the present disclosure. [Figure 7-6]FIG. 7 depicts a system that focuses on user invitation by console owners, management of peer-to-peer connections from remote controller inputs, and receiving game-rendered streams from game consoles, in accordance with some embodiments of the present disclosure. [Figure 7-7] FIG. 7 depicts a system that focuses on user invitation by console owners, management of peer-to-peer connections from remote controller inputs, and receiving game-rendered streams from game consoles, in accordance with some embodiments of the present disclosure. [Figure 7-8] FIG. 7 depicts a system that focuses on user invitation by console owners, management of peer-to-peer connections from remote controller inputs, and receiving game-rendered streams from game consoles, in accordance with some embodiments of the present disclosure.

[0044] [Figure 8-1] FIG. 8 depicts a process for a user to use a CD / CS application on a remote device to start a remote session, invite friends, and send connection information to the remote device, according to some embodiments of the present disclosure. [Figure 8-2] FIG. 8 depicts a process for a user to use a CD / CS application on a remote device to start a remote session, invite friends, and send connection information to the remote device, according to some embodiments of the present disclosure. [Figure 8-3] FIG. 8 depicts a process for a user to use a CD / CS application on a remote device to start a remote session, invite friends, and send connection information to the remote device, according to some embodiments of the present disclosure.

[0045] [Figure 9-1] FIG. 9 depicts a system according to some embodiments of the present disclosure that focuses on user invitations from a remote device by the console owner, for example, when the user is not at home. [Figure 9-2]FIG. 9 depicts a system according to some embodiments of the present disclosure that focuses on user invitations from a remote device by the console owner, for example, when the user is not at home. [Figure 9-3] FIG. 9 depicts a system according to some embodiments of the present disclosure that focuses on user invitations from a remote device by the console owner, for example, when the user is not at home. [Figure 9-4] FIG. 9 depicts a system according to some embodiments of the present disclosure that focuses on user invitations from a remote device by the console owner, for example, when the user is not at home. [Figure 9-5] FIG. 9 depicts a system according to some embodiments of the present disclosure that focuses on user invitations from a remote device by the console owner, for example, when the user is not at home. [Figure 9-6] FIG. 9 depicts a system according to some embodiments of the present disclosure that focuses on user invitations from a remote device by the console owner, for example, when the user is not at home.

[0046] [Figure 10-1] FIG. 10 depicts a process for a user to use the Remote Play application to start a remote session, invite friends, and send connection information to a remote device, according to some embodiments of the present disclosure. [Figure 10-2] FIG. 10 depicts a process for a user to use the Remote Play application to start a remote session, invite friends, and send connection information to a remote device, according to some embodiments of the present disclosure. [Figure 10-3] FIG. 10 depicts a process for a user to use the Remote Play application to start a remote session, invite friends, and send connection information to a remote device, according to some embodiments of the present disclosure. [Figure 10-4]FIG. 10 depicts a process for a user to use the Remote Play application to start a remote session, invite friends, and send connection information to a remote device, according to some embodiments of the present disclosure.

[0047] [Figure 11-1] FIG. 11 depicts a process for a remote invitation request from an account owner logged in as a remote user, according to some embodiments of the present disclosure. [Figure 11-2] FIG. 11 depicts a process for a remote invitation request from an account owner logged in as a remote user, according to some embodiments of the present disclosure.

[0048] [Figure 12-1] FIG. 12 depicts a process for initial controller connection for both local and remote users, according to some embodiments of the present disclosure. [Figure 12-2] FIG. 12 depicts a process for initial controller connection for both local and remote users, according to some embodiments of the present disclosure. [Figure 12-3] FIG. 12 depicts a process for initial controller connection for both local and remote users, according to some embodiments of the present disclosure.

[0049] [Figure 13-1] FIG. 13 depicts a process for transferring controller between a local user and a remote user according to some embodiments of the present disclosure. [Figure 13-2] FIG. 13 depicts a process for transferring controller between a local user and a remote user according to some embodiments of the present disclosure. [Figure 13-3] FIG. 13 depicts a process for transferring controller between a local user and a remote user according to some embodiments of the present disclosure. [Figure 13-4]FIG. 13 depicts a process for transferring controller between a local user and a remote user according to some embodiments of the present disclosure. [Figure 13-5] FIG. 13 depicts a process for transferring controller between a local user and a remote user according to some embodiments of the present disclosure.

[0050] [Figure 14-1] FIG. 14 depicts a process for a remote owner to request control or force a disconnect on a primary controller input (e.g., controller input 1) when all controller slots are in use, according to some embodiments of the present disclosure. [Figure 14-2] FIG. 14 depicts a process for a remote owner to request control or force a disconnect on a primary controller input (e.g., controller input 1) when all controller slots are in use, according to some embodiments of the present disclosure.

[0051] [Figure 15-1] FIG. 15 depicts a system, according to some embodiments of the present disclosure, that focuses on determining whether a user has sufficient bandwidth based on a standard value (e.g., a determined minimum value) that would cause the user to disconnect. [Figure 15-2] FIG. 15 depicts a system, according to some embodiments of the present disclosure, that focuses on determining whether a user has sufficient bandwidth based on a standard value (e.g., a determined minimum value) that would cause the user to disconnect. [Figure 15-3] FIG. 15 depicts a system, according to some embodiments of the present disclosure, that focuses on determining whether a user has sufficient bandwidth based on a standard value (e.g., a determined minimum value) that would cause the user to disconnect. [Figure 15-4]FIG. 15 depicts a system, according to some embodiments of the present disclosure, that focuses on determining whether a user has sufficient bandwidth based on a standard value (e.g., a determined minimum value) that would cause the user to disconnect. [Figure 15-5] FIG. 15 depicts a system, according to some embodiments of the present disclosure, that focuses on determining whether a user has sufficient bandwidth based on a standard value (e.g., a determined minimum value) that would cause the user to disconnect. [Figure 15-6] FIG. 15 depicts a system, according to some embodiments of the present disclosure, that focuses on determining whether a user has sufficient bandwidth based on a standard value (e.g., a determined minimum value) that would cause the user to disconnect. [Figure 15-7] FIG. 15 depicts a system, according to some embodiments of the present disclosure, that focuses on determining whether a user has sufficient bandwidth based on a standard value (e.g., a determined minimum value) that would cause the user to disconnect. [Figure 15-8] FIG. 15 depicts a system, according to some embodiments of the present disclosure, that focuses on determining whether a user has sufficient bandwidth based on a standard value (e.g., a determined minimum value) that would cause the user to disconnect. [Figure 15-9] FIG. 15 depicts a system, according to some embodiments of the present disclosure, that focuses on determining whether a user has sufficient bandwidth based on a standard value (e.g., a determined minimum value) that would cause the user to disconnect. [Figure 15-10] FIG. 15 depicts a system, according to some embodiments of the present disclosure, that focuses on determining whether a user has sufficient bandwidth based on a standard value (e.g., a determined minimum value) that would cause the user to disconnect. [Figure 15-11]FIG. 15 depicts a system, according to some embodiments of the present disclosure, that focuses on determining whether a user has sufficient bandwidth based on a standard value (e.g., a determined minimum value) that would cause the user to disconnect. [Figure 15-12] FIG. 15 depicts a system, according to some embodiments of the present disclosure, that focuses on determining whether a user has sufficient bandwidth based on a standard value (e.g., a determined minimum value) that would cause the user to disconnect.

[0052] [Figure 16-1] FIG. 16 depicts a process for determining whether sufficient bandwidth exists to support a remote client session, according to some embodiments of the present disclosure. [Figure 16-2] FIG. 16 depicts a process for determining whether sufficient bandwidth exists to support a remote client session, according to some embodiments of the present disclosure. [Figure 16-3] FIG. 16 depicts a process for determining whether sufficient bandwidth exists to support a remote client session, according to some embodiments of the present disclosure.

[0053] [Figure 17] FIG. 17 depicts the user experience (UX) after a user pauses a game with the option to invite a remote player to join the game, according to some embodiments of the present disclosure.

[0054] [Figure 18] FIG. 18 depicts a UX that includes an option for a user to invite remote players to win a spot in the game, according to some embodiments of the present disclosure.

[0055] [Figure 19]FIG. 19 depicts a UX according to some embodiments of the present disclosure, including an option for the console owner to select a remote player to view the main console as if the remote player were playing with the console owner.

[0056] [Figure 20] FIG. 20 depicts a UX that includes options that allow a user to transfer a controller to a remote user or a local user to take over control of a game controller locally, according to some embodiments of the present disclosure.

[0057] [Figure 21] FIG. 21 depicts the UX in spectator mode according to some embodiments of the present disclosure.

[0058] [Figure 22] FIG. 22 depicts the UX in gameplay mode according to some embodiments of the present disclosure.

[0059] [Figure 23] FIG. 23 depicts an artificial intelligence system according to some embodiments of the present disclosure.

[0060] [Figure 24] FIG. 24 depicts a system including a server, a communication network, and a computing device for implementing the methods and processes described herein, according to some embodiments of the present disclosure.

[0061] [Figure 25] FIG. 25 depicts an example flowchart of a process 2500 for monitoring players using an AI model, according to some embodiments of the present disclosure.

[0062] Throughout the drawings, drawing continuations are labeled as "(continued)" and continuations between sheets are labeled alphabetically sequentially with circles, i.e., "A" (circle), "B" (circle), etc. When the continuation number exceeds 26 (e.g., Sheet 7 / 77, Fig. 4 (continued)), the label continues as "AA" (circle), "AB" (circle), etc.

[0063] The drawings are intended to depict only typical aspects of the subject matter disclosed herein, and therefore should not be considered as limiting the scope of the present disclosure. Those skilled in the art will understand that the structures, systems, devices, and methods specifically described herein and illustrated in the accompanying drawings are non-limiting exemplary embodiments, and that the scope of the present invention is defined only by the claims. DETAILED DESCRIPTION OF THE INVENTION

[0064] Detailed Description

[0065] Game consoles, console game providers, and game publisher cloud services offer various features to enhance the gaming experience. These systems enable both local and remote play, enabling users to engage with friends regardless of their geographic location. Some features include the ability to play together virtually, invite remote players, share games in single-player mode, and join multiplayer games. The systems also ensure efficient delivery of game data through a single stream, adapting to varying bandwidth and game-based bitrate requirements. These features collectively create an immersive, flexible, and accessible gaming experience, encouraging social connection and shared gaming experiences.

[0066] A game console is provided for local and / or remote console play with local and / or remote friends. A CGP / CS is also provided for local and / or remote console play with local and / or remote friends. One or more of the following features may be provided with the game console and / or CGP / CS:

[0067] Play Together Virtually: This feature allows game console or PC game owners to play with friends or family who are far away, creating an experience that feels like they are playing in the same room.

[0068] Invite Remote Players: Users at home can invite others to join their game. These invited players may use a device (such as a phone, tablet, or streaming stick) with a remote play application. They themselves are not required to own a game console or PC game.

[0069] In some embodiments, console owners are provided with the ability to extend invitations to friends to participate in gaming activities via the remote play application. The invitation process can be initiated within the application itself. The system is location-independent, allowing console owners to extend these invitations regardless of their physical location. This feature increases the flexibility and accessibility of the gaming experience, enabling the enjoyment of multiplayer games with friends regardless of each individual's geographic location. The system serves as an effective medium for maintaining social connections and facilitating shared gaming experiences.

[0070] Sharing games in single-player mode: In games that are typically for one player, this feature allows users to take turns playing games with remote friends, similar to handing off game controllers within the same room. Unlike traditional approaches, this feature is achieved using a peer-to-peer connection, for example, between the console and one or more remote devices. In some embodiments, specific bandwidth management for peer-to-peer connections is leveraged to provide improved QoE.

[0071] Multiplayer Games: For games that support multiple players, this feature allows all players, regardless of their location, to join the game as if they were in the same room. They can even shuttle and virtually hand off game controllers.

[0072] Single-stream delivery: All remote devices receive game data from a single encoder and a single RTP stream, making the delivery process more efficient. In some embodiments, a direct peer-to-peer connection is provided to client devices. For example, a new adaptable bitrate and encoding quality controller is provided. The adaptable bitrate and encoding quality controller is provided, for example, with a single encoder and a single RTP transmitter. The single encoder and single RTP transmitter are configured to connect to one or more clients via an unmanaged network. Thus, remote play is provided in a peer-to-peer connection with multiple clients.

[0073] Bandwidth Adaptability: If a remote player's internet connection weakens, they can switch to a cloud-based ABR stream and continue watching the game. Once their connection improves, they can rejoin the game as a player.

[0074] Game-based bitrate adjustment: A game can adjust its bitrate requirements based on the game's activity level and the player's network quality. For example, during high-action sequences, the game may require a higher frame rate. If a player's network cannot meet the minimum requirements, they may be switched to a view-only mode until their network quality improves. As an example, bitrate requirements and encoding properties are controlled by the game engine through an application programming interface (API). Bitrate requirements and encoding properties are based, for example, on the level of action (e.g., higher action translates to a higher frame rate) or the type of scene (e.g., cutscenes are delivered at a lower frame rate and higher resolution). That is, game-controlled encoding parameters are based on the network QoS and / or bandwidth at any given time. In some examples, the minimum bandwidth required varies based on the minimum bitrate defined in the encoding profile. The varying minimum bitrate determines whether a client device whose network QoS, based on its bandwidth, is preventing it from meeting the minimum encoding properties should be either degraded or switched to a listen-only ABR delivery mode until it can meet the minimum network QoS and / or bandwidth requirements. For example, if a device's network quality is not sufficient to meet the minimum requirements for data transfer rate (bitrate), it may be unable to handle the encoding. In such a case, the device will either disconnect or switch to a "listen-only" mode that requires less data but increases latency. It may remain in this mode until network quality improves and it can meet the minimum requirements.

[0075] Other conditions may also trigger a drop or a switch to ABR, for example, a drop or a switch to ABR may be triggered when the remote device decodes at a limited frame rate or limited resolution and displays a limited refresh rate (device-specific or battery-constrained).

[0076] In some embodiments, the overall QoS of all players is assessed. For example, if a remote player has a lower bandwidth that is still playable, the system evaluates the extent to which the lower bandwidth affects the QoS for the entire game session for all remote users. If the player's slow bandwidth limits the session's QoS, the system automatically switches the player to a "listen-only" mode. This action is taken if removing the player would significantly improve the game experience for others with higher bandwidths.

[0077] These features make remote gaming more accessible, flexible, and enjoyable for all involved. They represent an improvement in the gaming experience, especially for those who wish to play remotely with others.

[0078] In some embodiments, a game controller, provided for local and / or remote console play with local and / or remote friends, is part of a system (e.g., FIG. 4) that ensures full QoS for multiple clients (e.g., three clients are shown), and in some instances requires bandwidth exceeding approximately 15 Mbps. System 400 of FIG. 4 comprises a game console that includes a game engine, a video encoder, an audio encoder, a multiplexer, an RTP sender, a transmission scheduler, and a UDP socket that can connect to one or more remote client devices. The game console also includes a rate and video encoding property controller for multiplayer low-latency gaming and a network congestion controller.

[0079] In some embodiments, a console game provider or PC game publisher cloud service offering local and / or remote console play with local and / or remote friends is part of a system (e.g., FIG. 7) configured to send and receive data to and from a game console and one or more client devices. System 700 of FIG. 7 may be similar in some or all respects to system 400. In addition, the game console of system 700 includes a UA / US / FC controller, a transport bandwidth control system, and a remote play controller. System 700 also includes a CGP / CS with a UAAS and a remote player management controller.

[0080] The system of Figure 15, which is part of a console game provider or PC game publisher cloud service process (e.g., Figure 16), is designed for low bandwidth / QoS connection conditions for at least one client. The system includes a remote session viewing adaptive bitrate (ABR) system with several novel details, such as an ABR delivery controller, UDP sockets, a transmission receiver, a demultiplexer, an ABR transcoder and segmenter, an ABR A / V segment and manifest database, and an ABR Hypertext Transfer Protocol (HTTP) delivery system.

[0081] A graphical user interface (GUI) for local and / or remote console play with local and / or remote friends is depicted in Figures 17-22.

[0082] Various methods associated with the present system are also provided, such as the process of Figure 6 for the system of Figure 4 and the process of Figure 8 for the system of Figure 7. Additional systems and methods are provided that detail processes such as a remote invitation request from an account owner logged in as a remote user, remote initial controller connection for local and remote users, controller handover, and scenarios where the owner is remote and requests control.

[0083] As described in detail herein, systems and methods are provided for enhancing local and / or remote console play with local and / or remote friends, whether through a game controller or console game provider or PC game publisher cloud service. Features of these systems and methods enable an improved gaming experience and improved remote computing.

[0084] 1 depicts a system 100 for local and / or remote play via a game console 110, according to some embodiments of the present disclosure, where the console acts as a server and forms peer-to-peer connections with remote client devices. Game console 110 can locally connect to one or more controllers, e.g., game controller 112, game controller 114, game controller 116, and game controller 118, via separate communication links (shown with double-headed arrows). Game console 110 can connect to a mobile or fixed-line network 120 via communication link 115. Game console 110 can connect to one or more remote client devices, e.g., a desktop computer 130, a mobile device 140, and a tablet 150, via mobile or fixed-line network 120 and separate communication links 125, 135, and 145. In the illustrated embodiment, desktop computer 130, mobile device 140, and tablet 150 are connected to game controller 132, game controller 142, and game controller 152 via separate communication links (shown with double arrows). In some examples, the controllers are not separate from the remote client device, and game play control is controlled by the remote client device itself (not shown). Game console 110 can connect to one or more providers or cloud services, e.g., CGP / CS 160, via mobile or fixed-line network 120 and communication link 155. Although mobile or fixed-line network 120 is shown as a single network, multiple networks may be utilized.

[0085] 2 depicts a security system 200 for connecting to local and / or remote devices via a server 210 that forms peer-to-peer connections with remote client devices, according to some embodiments of the present disclosure. Local security server 210 can locally connect to one or more devices, e.g., security camera 212, security camera 214, security camera 216, and security camera 218, via separate communication links (shown with double-headed arrows). Local security server 210 can connect to a mobile or fixed-line network 220 via communication link 215. Local security server 210 can connect to one or more remote client devices, e.g., a desktop computer 230, a mobile device 240, and a tablet 250, via mobile or fixed-line network 220 and separate communication links 225, 235, and 245. In the illustrated embodiment, desktop computer 230, mobile device 240, and tablet 250 are connected via separate communication links (shown with double-headed arrows) to security camera 232, security camera 242, and security camera 252. In some examples, the cameras are not separate from the remote client devices and / or the cameras themselves are remote client devices (not shown). Local security server 210 can connect to one or more providers or cloud services, such as security provider cloud service 260, via mobile or fixed-line network 220 and communication link 255. While mobile or fixed-line network 220 is shown as a single network, multiple networks may be utilized.

[0086] 3 depicts an extended reality (XR) system 300 for connecting to local and / or remote devices via a server 310, which forms a peer-to-peer connection with remote client devices, according to some embodiments of the present disclosure. The local XR server 310 can locally connect to one or more devices, e.g., XR device 312, XR device 314, XR device 316, and XR device 318, via separate communication links (shown with double-headed arrows). The local XR server 310 can connect to a mobile or fixed-line network 320 via communication link 315. The local XR server 310 can connect to one or more remote client devices, e.g., a desktop computer 330, a mobile device 340, and a tablet 350, via the mobile or fixed-line network 320 and separate communication links 325, 335, and 345. In the illustrated embodiment, desktop computer 330, mobile device 340, and tablet 350 are connected via separate communication links (shown with double arrows) to XR headset 332, XR glasses 342, and XR projector 352. In some examples, the XR devices are not separate from the remote client devices and / or are themselves remote client devices (not shown). Local XR server 310 can connect to one or more providers or cloud services, e.g., XR provider cloud service 360, via mobile or fixed-line network 320 and communication link 355. Although mobile or fixed-line network 320 is shown as a single network, multiple networks may be utilized.

[0087] FIG. 4 depicts a system 400 for local and / or remote play via a game console 401, according to some embodiments of the present disclosure, where the console 401 acts as a server and forms a peer-to-peer connection with remote client devices.

[0088] For example, the game console 401 may be located within a user's home. The remote client device may be a mobile phone, tablet, over-the-top (OTT) set-top box (STB), high-definition multimedia interface (HDMI) stick, personal computer, or other game console running a remote play application provided by the console company. The remote play application running on the remote device, for example, may be an extremely lightweight application requiring only an RTP receiver, a video decoder, and the ability to remotely connect to a game controller from the remote device, which sends control data to the game console via UDP, connected via a UDP socket. In an implementation for a mobile phone or tablet with a touch display, remote buttons may be shown on a screen overlay. Another UDP socket is also provided for connecting the controller back to the game console. Controller data is sent from the remote device to the game console, and controller tactile data is sent from the game console to the remote device. To implement low-latency delivery, some embodiments provide an enhanced SCReAM. The enhanced SCReAM leverages ultra-low latency RTP delivery. The enhanced SCReAM architecture utilizes the returned RTCP packets to determine when the next RTP packet should be sent. Once the transmission scheduler determines when to send a packet, the packet is removed from the priority queue RTP packet by the transmission scheduler and sent to the client via a UDP socket. Changes in the size of the priority queue of RTP packets are sent to the rate and video encoding property control. Based on the change in the priority queue size, the video encoding bitrate is adjusted to fit within the current network QoS. In some embodiments, the enhanced SCReAM architecture includes providing a data store of RTCP packets from each connected client and adjusting the encoding rate and encoding properties based on the worst-case client.

[0089] In some embodiments, system 400 is configured with SCReAM and Low Latency, Low Loss, Scalable Throughput (L4S) Internet Services. The L4S architecture is part of the Internet Engineering Task Force (IETF) standard and allows packets to be marked by the sender and received at the receiver, for example, using Explicit Congestion Notification (ECN). SCReAM has been thoroughly tested with L4S, achieving dramatic improvements in latency and packet loss. The insight on which L4S is based is that the root cause of queuing delays lies in the congestion controller at the sender, not the queue itself.

[0090] In some embodiments, system 400 is configured with an L4S architecture. With the L4S architecture, all Internet applications can transition from congestion control algorithms that cause substantial queuing delays to a new class of congestion controls that induce little queuing and are assisted by explicit congestion signaling from the network. This new class of congestion controls provides low latency for capacity-seeking flows, allowing applications to achieve both high bandwidth and low latency. This is provided as a flag that enables marking packets with the appropriate ECN L4S marker.

[0091] In some embodiments, system 400 is configured for real-time video, XR, and cloud gaming. For applications such as real-time video, XR, and cloud gaming, the extra delay caused by retransmission or forward error correction (FEC) can be significant. ECN uses the active queue management (AQM) congestion detection method, but with explicit signaling to indicate to end hosts that packets that would normally be discarded are instead marked as congested. ECN uses two bits in the IP header, and information about packets marked with ECN Congestion Experience (CE) is carried in Transmission Control Protocol (TCP) Acknowledgments (ACKs). End hosts then treat CE-marked packets in the same way as lost packets.

[0092] In some embodiments, system 400 is configured with ECN-aware AQM. ECN-aware AQM marks packets as CE when congestion is detected instead of discarding them, resulting in significant packet loss reduction, but less latency reduction compared to packet-discarding AQM. The interpretation of L4S-aware AQM is that if a packet is marked as L4S-aware, it should be marked as CE immediately when queuing delay is low. This provides immediate response to minor signs of congestion and allows end hosts to implement scalable congestion control. This means that instead of the multiplicative decrease approach used for ECN-enabled flows, scalable congestion control reduces the congestion window (or transmission rate) in proportion to the percentage of packets marked as CE. Compared to classical ECN, the L4S approach allows for denser congestion events, provides lower delay jitter, and allows for low queuing delay while maintaining high link utilization.

[0093] In some embodiments, system 400 is configured with an enhanced SCReAM architecture, which is modified so that multiple client devices can now connect to the same UDP socket and receive L4S marked multimedia packets. For each RTP packet sent to a client device, the client device responds with an RTCP packet, which contains the following: Synchronization Source Identifier (SSRC), Transmission Timestamp (TSTX), RTP Sequence Number (RTP(SN)), RTP Packet Size (RTP(Size)), Round Trip Time (RTT), Congestion Window (CWND), Source Address (SRC:<IPアドレス> ), destination address (DEST:<IPアドレス> ), and the client device IP address (SRC) sending the RTCP packet.

[0094] 5 depicts a packet 500 for RTCP according to some embodiments of the present disclosure. In some embodiments, packet 500 includes an SRC and a DEST IP address. SRC, shown in box 510, is the IP address of the RTP client / receiver sending the RTCP packet. DEST is the IP address of the RTP sender (e.g., game console 400). Packet 500 is designated as an RTCP packet in box 520.

[0095] In addition to the source and destination IP addresses, packet 500 includes ports, a timestamp, and other protocol-specific data. Frame 2069 refers to the specific packet being analyzed. The frame size is 138 bytes when sent and when captured. Ethernet II is the most common Ethernet frame type and contains source (Cisco_b7:17:0a) and destination (Cisco_76:b5:12) MAC addresses. IPv4 is the fourth version of the Internet Protocol (IP). IPv4 is used to identify devices on a network. The source IP is 14.50.201.48 and the destination IP is 172.18.110.203. User Datagram Protocol (UDP), shown in box 530, is used to establish low-latency, loss-tolerant connections between applications on the Internet. The source port is 23979 and the destination port is 8229. RTCP is used in conjunction with RTP to provide out-of-band control information for streaming media. This includes the sender report, receiver report, source description, and termination function. As shown in box 540, the sender SSRC is a synchronization source identifier that uniquely identifies the source of the stream. The sender SSRC value is here 623818024. The sender report timestamp (Most Significant Word (MSW)) is 3728732393. The sender packet count is the number of packets the sender has sent for this profile. The value is here 298. The sender octet count is the total number of payload octets (i.e., not including headers or padding) that this sender has sent in the RTP packet. The value is here 47680. Other information is provided in packet 500 as shown.

[0096] In some embodiments, network congestion control is configured to send the CWND and RTT (bytes in flight) to a transmission scheduler to determine when packets should be removed from the RTP packet priority queue and sent to devices via RTP. The packets are delivered to multiple devices over an unmanaged network with varying QoS. To accomplish this, an RTCP response router, a packet response data store per connected IP address (SRC IP), and an RTCP reporting system are provided within the architecture of system 400. When an RTCP packet is received for a sent RTP packet, the RTCP packet from the SRC IP address is placed in a packet response data store dedicated to that SRC IP address. The data store also reports the CWND, RTT (bytes in flight) for that SRC IP address. The RTCP reporting system waits until there are CWND, RTT (bytes in flight) for all connected clients. Once all connected clients have reported, the RTCP reporting system calculates the worst-case SRC:<IPアドレス> The CWND and RTT (bytes in flight) for the message are sent to the transmission scheduler. Once this message is received, the transmission scheduler moves the priority queue RTP packet into the transmission scheduler and sends the packet to connected clients via UDP sockets. Using this method, all clients receive the same quality encoding based on the worst-case client. This method can be referred to as multicast multimedia with bitrate control, which delivers a single encoding to multiple devices over a single UDP link.The advantage of multicast multimedia with bitrate control is that it involves the use of a single encoder to deliver a single stream with bitrate and quality control to multiple devices over an unmanaged, limited internet uplink connection with ultra-low latency (significantly less on a consumer unmanaged internet connection versus an unmanaged downlink connection), which is suitable for remotely rendered gaming.

[0097] In some embodiments, system 400 is configured to connect players directly using a minimal number of servers controlled by different organizations. Priority queues are configured, for example, to help manage traffic and minimize the potential for congestion-induced latency. To minimize inherent latency caused by physical distance, the number of nodes in a path and the number of connections between nodes are minimized. Additionally, to ensure the effectiveness of priority queues, in some embodiments, the server establishes the initial connection from the client device through an invitation. Thereafter, the server is not involved. Console- or PC-based systems are configured to allow a connection URL to be created and given to devices, for example, through a game streaming platform, using a custom app that connects to the console or PC system directly via a shared link.

[0098] In some embodiments, system 400 provides for local and / or remote play via game console 401. Game console 401 acts as a server and forms peer-to-peer connections with one or more remote client devices. Game console 401 may include and / or connect to video device 451 and audio device 453, which may be integrated into game console 401 or provided separately and connected to game console 401 via separate communication links. Game console 401 may locally connect to one or more controllers, e.g., game controller 1 455, game controller 2 457, game controller 3 459, and game controller 4 461, via separate communication links. Game console 401 may connect to a mobile or fixed-line network 463 via a communication link. Game console 401 is connectable to one or more remote client devices, e.g., remote client device 1 465a, remote client device 2 465b, and remote client device 3 465c, via mobile or fixed-line network 463 and separate communication links. In the illustrated embodiment, remote client device 1 465a, remote client device 2 465b, and remote client device 3 465c are connected to game controllers 491a, game controller 491b, and game controller 491c via separate communication links. In some examples, the controllers are not separate from the remote client devices, and game play control is controlled by the remote client devices themselves (not shown). Although mobile or fixed-line network 463 is illustrated as a single network, multiple networks may be utilized.

[0099] The game console 401 includes at least one of a central processing unit (CPU), a graphics processing unit (GPU), an audio chip, memory (e.g., random access memory (RAM)), storage (e.g., a hard drive or solid state drive), one or more input / output interfaces, an operating system, game media, or a network adapter (see, e.g., FIGS. 4 and 24). The CPU is the main processor of the console 401 and is responsible for executing computer program instructions. The GPU is a specialized processor that accelerates the generation of images for output to a display. The audio chip is responsible for generating sound. Memory is a memory for temporary data that is being actively processed by the CPU. The primary storage of the console 401. The storage is where data is stored long-term, including game data, user information, and so on. One or more input / output interfaces allow for the connection of controllers, external storage devices, and other peripherals. The operating system is software that manages hardware resources and provides various services for the execution of games and applications. The game media can be a cartridge, CD, DVD, or digital download. The game media contains game data and is read by the console 401 to play games. The network adapter allows the console to connect to the Internet for online gaming, updates, and so on.

[0100] In some embodiments, game console 401 includes at least one of a game engine 403 for running game titles, a rate and video encoding property control 407, a video encoder 409, an audio encoder 411, a multiplexer 413, an RTP transmitter 415, a priority queue RTP packet data store (or file) 417, a transmission scheduler 419, a UDP socket (connection endpoint 1) 421, network congestion control 423, controller handler 1 433, controller handler 2 435, controller handler 3 437, controller handler 4 439, a controller input / output mapper 441, a UDP socket 1 (connection endpoint 2) 443, a UDP socket 2 (connection endpoint 3) 445, a UDP socket 3 (connection endpoint 4) 447, or a UDP socket 4 (connection endpoint 5) 449.

[0101] In some embodiments, the game engine 403 is configured to perform at least one of the following steps: sending a game title QoE remote play definition to a data store (or file) 405; sending raw image data to a video encoder 409; sending raw image data to a video device 451; sending raw audio image data to an audio encoder 411; sending raw audio image data to an audio device 453; sending tactile data to at least one of the controller handlers (e.g., controller handler 1 433, controller handler 2 435, controller handler 3 437, controller handler 4 439); or receiving controller input from at least one of the controller handlers.

[0102] In some embodiments, the controller handlers are configured to exchange information with local and / or remote controllers or devices. For example, at least one of the controller handlers (e.g., controller handler 1 433, controller handler 2 435, controller handler 3 437, controller handler 4 439) is configured to send tactile data to one or more controllers (e.g., game controller 1 455, game controller 2 457, game controller 3 459, and game controller 4 461) and receive controller input from one or more controllers. Also, in some embodiments, at least one of the controller handlers is configured to send tactile data to a separate UDP socket and receive controller input from a separate UDP socket, e.g., UDP socket 1 (connection endpoint 2) 443, UDP socket 2 (connection endpoint 3) 445, UDP socket 3 (connection endpoint 4) 447, or UDP socket 4 (connection endpoint 5) 449. In some embodiments, either in addition to the controller handlers or instead of at least one of the controller handlers, a dedicated remote slot 440 is provided for communication with one or more of the remote clients and / or controllers. In some embodiments, four local controller handlers (such as controller handlers 433-439, respectively, as shown) are provided for the four controllers, and four remote slots (such as dedicated remote slot 440, respectively) are provided for separate UDP sockets. In some embodiments, controller input / output mapper 441 is configured to control various inputs and outputs to and / or from the local and / or remote controllers via the UDP sockets.

[0103] For example, in some embodiments, UDP socket 1 (connection endpoint 2) 443 is configured to send tactile data 443a to UDP socket 483a (at IP address 1:port 2) of remote client device 1 465a and receive controller input 483a1 from UDP socket 483a (at IP address 1:port 2) of remote client device 1 465a. UDP socket 2 (connection endpoint 3) 445 is also configured to send tactile data 445a to UDP socket 483b (at IP address 2:port 2) of remote client device 2 465b and receive controller input 483b1 from UDP socket 483b (at IP address 2:port 2) of remote client device 2 465b. Additionally, UDP Socket 3 (Connection Endpoint 4) 447 is configured to send tactile data 447a to UDP Socket 483c (at IP Address 3:Port 2) of Remote Client Device 3 465c and to receive controller input 483c1 from UDP Socket 483c (at IP Address 3:Port 2) of Remote Client Device 3 465c. Additionally, UDP Socket 4 (Connection Endpoint 5) 449 may be configured for additional remote client devices (not shown).

[0104] In some embodiments, a bitrate / encoding property mapping based on the game title QoE remote play definition stored in data storage (or file) 405 is sent to rate and video encoding property control 407. In some embodiments, rate and video encoding property control 407 is configured to perform at least one of receiving a queue length from priority queue RTP packet data storage (or file) 417, receiving a bitrate / encoding property mapping from data storage (or file) 405, receiving an audio bitrate from audio encoder 411, determining a target video encoding bitrate, sending the target video encoding bitrate to video encoder 409, determining target video encoding properties, sending the target video encoding properties to video encoder 409, or receiving a multiplexer (multiplexing) bitrate from multiplexer 413. In some embodiments, the rate and video encoding properties control 407 is configured to calculate a target rate and target video encoding properties based on the queue length, the bitrate / encoding property mapping (which itself is based on the game title QoE remote play definition), the audio bitrate, and the multiplexing bitrate.

[0105] In some embodiments, the video encoder 409 and the audio encoder 411 process information according to instructions received from the game engine 403 and / or the rate and video encoding property control 407. For example, the video encoder 409 receives raw image data, a target rate, and target video encoding properties, and encodes the raw image data, the target rate, and the target video encoding properties into encoded video, which is sent to the multiplexer 413. Also, for example, the video encoder 411 receives raw audio data and encodes the raw image data into encoded audio, which is sent to the multiplexer 413. Also, the video encoder 411 transmits the audio bitrate to the rate and video encoding property control 407.

[0106] In some embodiments, the multiplexer 413 receives the encoded video and encoded audio, processes the encoded video and encoded audio, transmits the multiplexed bitrate to the rate and video encoding properties control 407, and transmits the multiplexed encoded video and audio packets to the RTP sender 415.

[0107] In some embodiments, the RTP transmitter 415 receives the multiplexed encoded video and audio packets, processes the multiplexed encoded video and audio packets, and transmits the RTP multiplexed encoded video and audio packets to a priority queue RTP packet data store (or file) 417.

[0108] In some embodiments, the transmission scheduler 419 receives RTP-multiplexed encoded video and audio packets from the priority queue RTP packet data store (or file) 417, receives the CWND and RTT (e.g., bytes-in-flight for the worst-case source) from the network congestion control 423, processes the RTP-multiplexed encoded video and audio packets, CWND, and RTT, transmits the synchronization source (SSRC), transmission timestamp (TS(TX)), Real-time Transport Protocol sequence number (RTP(SN)), and RTP packet (RTP(size)) to the network congestion control 423, and transmits the RTP-multiplexed ("multiplexed" and "muxed" are used interchangeably herein) encoded video and audio packets to the UDP socket (connection endpoint 1) 421.

[0109] In some embodiments, UDP socket (connection endpoint 1) 421 is configured to receive RTP-multiplexed encoded video and audio packets from the transmission scheduler and process the RTP-multiplexed encoded video and audio packets. UDP socket (connection endpoint 1) 421 is configured to perform at least one of the following steps: receiving one or more RTCP packets from one or more remote client devices; transmitting RTP-multiplexed encoded video and audio packets to one or more remote client devices; sending an existing, new, or removed SRC IP connection to network congestion control 423; or sending RTCP packets along with the SRC IP to network congestion control 423. For example, if a client disconnects, the client's IP address, along with the data store, is removed from the network congestion control, and the network congestion control will no longer receive RTCP packets from the removed client. In other words, a client connected to the UDP socket connection endpoint sends RTCP packets to the network congestion control.

[0110] 4, UDP socket (connection endpoint 1) 421 receives RTCP packets from UDP socket 469a (e.g., at IP address 1:port 1) of remote client device 1 465a with SRC:<remote client device 1 IP address> 469a1 and transmits RTP-multiplexed encoded video and audio packets 421a (for SRC:<remote client device 1 IP address>) to UDP socket 469a of remote client device 1 465a. UDP socket (connection endpoint 1) 421 and UDP socket 469a (e.g., at IP address 1:port 1) may communicate, for example, over one of RTP ports 49152-64512.

[0111] In some embodiments, UDP socket (connection endpoint 1) 421 may be configured for multiple remote clients. For example, in the embodiment of Figure 4, UDP socket (connection endpoint 1) 421 receives RTCP packets with SRC:<remote client device 2 IP address> 469b1 from UDP socket 469b (e.g., at IP address 2:port 1) of remote client device 2 465b and transmits RTP-multiplexed encoded video and audio packets 421b (for SRC:<remote client device 2 IP address>) to UDP socket 469b of remote client device 2 465b. Similarly, UDP socket (connection endpoint 1) 421 receives RTCP packets from UDP socket 469c (e.g., at IP address 3:port 1) of remote client device 3 465c with SRC:<remote client device 3 IP address> 469c1 and transmits RTP-multiplexed encoded video and audio packets 421c (for SRC:<remote client device 3 IP address>) to UDP socket 469c of remote client device 3 465c. UDP socket (connection endpoint 1) 421 may communicate with one or both of UDP socket 469b (e.g., at IP address 2:port 1) and UDP socket 469c (e.g., at IP address 3:port 1) via, for example, one of RTP ports 49152-64512.

[0112] Network congestion control 423 of game console 401 manages data traffic between game console 401 and remote client devices (e.g., 465a, 465b, 465c). Network congestion control 423 receives and processes various types of data, tracks data traffic for each client device, identifies any potential problem areas (such as worst-case client devices), and adjusts its data transmissions accordingly.

[0113] For example, the network congestion control 423 of the game console 401 is configured to receive RTCP packets and changes to the SRC IP connection from a UDP socket (e.g., 421) connected to one or more remote client devices (e.g., 465a, 465b, 465c). The network congestion control 423 also receives various data from the transmission scheduler 419 of the game console 401. This data includes, for example, the SSRC, TS(TX), RTP(SN), and RTP(size). The RTCP packets, along with the SRC IP, are processed in the RTCP response router 425 of the network congestion control 423. The network congestion controller 423 processes existing, new, or removed SRC IP connections. The RTCP response router 425 sends the RTCP packets to at least one packet response data store, for example, a packet response data store 427 for SRC 1 and a packet response data store 429 for SRC n. The network congestion controller 423 processes various data by accessing one or more packet response data stores. The network congestion control 423 accesses the packet response data stores (e.g., 427, 429) for each SRC. The network congestion control 423 determines a congestion window (CWND) and round trip time (RTT) for each SRC, which are sent to the RTCP reporting system 431 of the network congestion control 423. The RTCP reporting system 431 is configured to process the CWND and RTT for each source packet.

[0114] In some embodiments, packet response data stores (e.g., 427, 429) store one or more RTCP responses as they are received from their corresponding client IP addresses. Once the last data store receives the RTCP response, the information is sent to transmission scheduler 419. The worst-case client device will respond (e.g., in operation, data stores for other clients already have the RTCP packets stored in their data stores). The last RTCP packet is used to send the information to transmission scheduler 419. All data stores that store that RTCP packet will remove it.

[0115] For the worst-case or last client device to respond with an RTCP packet, RTCP reporting system 431 sends the CWND and RTT to transmission scheduler 419 of game console 401. In some embodiments, once the RTP packet is sent with the sequence number, network congestion control 423 may wait until all (or some) client devices (identified by IP addresses) have responded with an RTCP packet. The last client to respond, for example, results in network congestion control 423 sending the CWND and RTT for that last client to respond.

[0116] Turning to remote client device 1 465a, remote client device 2 465b, and remote client device 3 465c of system 400, a description of remote client device 1 465a is provided herein. In some embodiments, such as the embodiment depicted in FIG. 4, the remote client devices are depicted identically (but not necessarily identical). Thus, like reference numbers for remote client device 1 465a have the same description and functionality as like reference numbers for remote client device 2 465b and remote client device 3 465c. Duplicate descriptions are omitted for brevity.

[0117] In some embodiments, remote client device 1 465a includes a game remote play client app 467a. The game remote play client app 467a includes at least one of a UDP socket 469a, a transmission receiver 471a, a demultiplexer 473a, a video decoder 475a, a video renderer 477a, an audio decoder 479a, an audio renderer 481a, a UDP socket 483a, or a controller for data transmission 485a. Remote client device 1 465a may also include at least one of an internal or separately connected video device 487a, an internal or separately connected audio device 489a, or an internal or separately connected controller, such as a game controller 491a.

[0118] In operation, in some embodiments, remote client device 1 465a is configured to perform at least one of the following steps: sending RTCP packets with SRC:<remote client device 1 IP address> 469a1 to game console 401, receiving RTP-multiplexed encoded video and audio packets 421a (for SRC:<remote client device 1 IP address>) from game console 401, sending rendered video frames to video device 487a, sending rendered audio frames to audio device 489a, receiving tactile data 443a from game console 401, sending controller input 483a1 to game console 401, sending tactile data to game controller 491a, or receiving controller input from game controller 491a. As described in detail herein, the RTCP packets with SRC:<remote client device 1 IP address> 469a1 are sent to game console 401 for processing (e.g., by network congestion control 423).

[0119] In some embodiments, the game remote play client app 467a includes at least one of the following: a UDP socket 469a configured to send RTP-multiplexed (multiplexed) encoded video and audio packets to a transmission receiver 471a; a transmission receiver 471a configured to generate and / or send RTCP packets to a UDP socket 469a; or a transmission receiver 471a configured to generate and / or send RTP-multiplexed encoded video and audio packets to a demultiplexer 473a, video processing, or audio processing. On the video processing side, in some embodiments, the demultiplexer 473a is configured to demultiplex the RTP-multiplexed encoded video and audio packets, the demultiplexer 473a is configured to send the encoded video PES to the video decoder 475a, the video decoder 475a is configured to decode the encoded video PES, the video decoder 475a is configured to send the decoded video frames to the video renderer 477a, the video renderer 477a is configured to render the rendered video frames, and the video renderer 477a is configured to send the rendered video frames to the video device 487a.On the audio processing side, in some embodiments, the demultiplexer 473a is configured to demultiplex the RTP-multiplexed encoded audio and audio packets, the demultiplexer 473a is configured to send the encoded audio PES to the audio decoder 479a, the audio decoder 479a is configured to decode the encoded audio PES, the audio decoder 479a is configured to send the decoded audio frames to the audio renderer 481a, the audio renderer 481a is configured to render the rendered audio frames, and the audio renderer 481a is configured to send the rendered audio frames to the audio device 489a.

[0120] In some embodiments, game remote play client app 467a is configured to process controller input data and tactile data (or other forms of input and output) between remote client device 1 465a, controller 491a, and game console 401. For example, UDP socket 483a is configured to receive tactile data 443a from game console 401, UDP socket 483a is configured to transmit tactile data to controller for data transmission 485a, and controller for data transmission 485a is configured to transmit tactile data to game controller 491a. A user operating game controller 491a operates one or more input devices (e.g., buttons, triggers, sensor data, and the like) of game controller 491a, which is converted into controller input data. Controller for data transmission 485a is configured to receive controller input from game controller 491a. Controller for data transmission 485a is configured to transmit controller input data to UDP socket 483a. UDP socket 483a is configured to send controller input data 483a1 to UDP socket 443 of game console 401. The controller input data is sent back to game engine 403 via one of the controller handlers (e.g., 433), as described in more detail herein.

[0121] In connection with the concept defined herein of a game title Quality of Experience (QoE) remote play definition (e.g., between game engine 403 and data store 405 (see also FIG. 6)), Table 1 below illustrates example bitrates for remote rendering codecs for a specific game title. These bitrates are adjusted to provide a controlled user experience based on bitrate variations. A game developer or publisher may define the quality of a game title. This allows for adjustments to the encoding quality based on the bitrate requested by the encoder. The enhanced SCReAM described herein is configured to associate varying bitrates with the encoder. The rate controller is also configured to set encoding properties based on policy definitions. In some embodiments, this control is based on codec type, such as Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), or Versatile Video Coding (VVC). In the present context, the example provided is for HEVC. This example meets the minimum requirements defined by a game console (e.g., Sony).

[0122] The game engine may also send updated requirements based on the dynamics and latency requirements of the game video. For example, during a cutscene, the rate and frame rate may change over its duration. After a cutscene, rate-related encoding properties may change according to a different encoding policy. These properties are dynamic and can change based on the content being sent to the remote rendering client device at any given time.

[0123] Table 1 provides data regarding bit rates (both low and high) for specific combinations of resolution and frame rate. [Table 1-1] [Table 1-2]

[0124] 6 depicts a process 600 for bitrate and encoding quality control for delivering a single encoded stream from a single RTP sender to multiple client devices over an unmanaged network, according to some embodiments of the present disclosure. In some embodiments, the process is dynamic in that it provides control to the encoder in real time, adjusting encoding properties based on the type of content. The encoding properties may change based on the content and the current bandwidth at any given time. The content producer, in this case, a game provider, provides encoding properties to the encoder to drive quality, for example, at any given time.

[0125] In some embodiments, process 600 includes steps involving several components of system 400, including, for example, at least one of game console 401, game engine 403, rate and video encoding property control 407, video encoder 409, audio encoder 411, multiplexer 413, RTP transmitter 415, priority queue RTP packet data store 417, transmission scheduler 419, UDP socket (connection endpoint 1) 421, network congestion control 423, RTCP response router 425, packet response data store for SRC 1 427, packet response data store for Src n 429, or RTCP reporting system 431.

[0126] The process 600 includes step 602 of starting and / or rendering video at a video source. The process 600 includes step 604 of instantiating a self-clocked rate-adaptive RTP delivery session. The process 600 includes step 606 of instantiating the RTP delivery session. The process 600 includes step 608 of instantiating a video encoder instance in a live low-latency mode. The process 600 includes step 610 of receiving encoding properties along with a bitrate range (e.g., Table 1) at a video encoder (e.g., 409). The process 600 includes step 612 of instantiating an audio encoder instance. The process 600 includes step 614 of encoding the video at the lowest bitrate set in an encoding profile table or controller at the video encoder. The process 600 includes step 616 of starting encoding the audio at a defined audio bitrate at an audio encoder (e.g., 411). The process 600 includes a step 618 of receiving the audio and video PES at a multiplexer (e.g., 413) and adjusting the multiplexing rate based on the audio and video PES. The process 600 includes a step 620 of transmitting the multiplexed audio and video PES streams from the multiplexer to an RTP sender (e.g., 415). The process 600 includes a step 622 of transmitting the multiplexed bitrate rate from the multiplexer to a rate and video encoding property control (e.g., 407). The process 600 includes a step 624 of pausing the RTP delivery session for the UDP connection.

[0127] Process 600 includes step 626, which determines whether the connection is made over a UDP delivery IP address and port number (shown in the figure as "port:address"). If the connection is made over a UDP delivery IP address and port number (626="yes"), process 600 proceeds to step 628. If the connection is not made over a UDP delivery IP address and port number (626="no"), process 600 proceeds to step 638.

[0128] Process 600 includes step 628 of sending a new SRC IP connection with the client's SRC IP address to the network congestion control (e.g., 423). Process 600 includes step 630 of adding a new packet response data store (e.g., 427, 429) for the SRC IP address in the network congestion control (e.g., 423). Process 600 includes step 632 of determining whether any existing connections exist. If any existing connections do not exist (632="no"), process 600 proceeds to step 634. If any existing connections do exist (632="yes"), process 600 proceeds to step 636.

[0129] Process 600 includes a step 634 in an RTP sender (e.g., 415) of transmitting (e.g., buffering) the RTP-multiplexed encoded video and audio packets to a priority queue RTP packet data store (e.g., 417). Process 600 includes a step 636 in a transmission scheduler (e.g., 419) of accessing a first RTP-multiplexed encoded video and audio packet from the priority queue of packets and transmitting it to a UDP socket connection endpoint (e.g., 421) for delivery.

[0130] Process 600 includes step 638 of determining whether the connection was dropped on the UDP delivery IP address and port number. If the connection was not dropped on the UDP delivery IP address and port number (638="No"), process 600 proceeds to step 636. If the connection was dropped on the UDP delivery IP address and port number (638="Yes"), process 600 proceeds to step 640.

[0131] Process 600 includes step 640 of removing the packet response data store for the SRC IP address. Process 600 includes step 642 of determining whether any client devices are still connected. If any client devices are still connected (642="yes"), process 600 proceeds to step 636. If any client devices are still connected (642="yes"), process 600 proceeds to step 644.

[0132] Process 600 includes step 644 of stopping transmission (buffering) of RTP multiplexed encoded video and audio packets in the priority queue. Process 600 includes step 646 (e.g., at 417) of flushing the priority queue RTP packets. Process 600 returns to step 614 of encoding.

[0133] After the sending step 634, the process 600 includes a step 648 of monitoring the queue length of the packet's priority queue in the rate and video encoding property control (e.g., 407). The process 600 includes a step 650 of determining whether a rate adaptation increase or decrease is needed based on the queue length. If a rate adaptation increase or decrease is needed based on the queue length (650="yes"), the method proceeds to step 652. If a rate adaptation increase or decrease is not needed based on the queue length (650="no"), the method returns to the monitoring step 648.

[0134] Process 600 includes step 652 of accessing encoding properties based on the new calculated bitrate from an encoding property definition (e.g., Table 1) in a rate and video encoding property control (e.g., 407). In some embodiments, in response to determining that a bitrate adjustment increase or decrease is needed based on the queue length, the rate and video encoding property controller accesses encoding properties based on the multiplexer bitrate and the audio bitrate. Process 600 includes step 654 of calculating a new video encoding rate based on the audio rate in the rate and video encoding property control (e.g., 407) and a multiplexed bitrate based on the calculated network QoS. Process 600 includes step 656 of determining whether an increase is needed. If an increase is needed (656="yes"), the method proceeds to step 658. If an increase is not needed (656="no"), the method proceeds to step 662.

[0135] If an increase is required (656="Yes"), process 600 includes step 658 of sending a new target rate from the rate and video encoding properties control (e.g., 407) to the video encoder (e.g., 409). Process 600 includes step 660 of sending new video encoding properties (e.g., resolution and frame rate) from the rate and video encoding properties control (e.g., 407) to the video encoder (e.g., 409).

[0136] If an increase is not required (656="No"), steps 658 and 660 are reversed. That is, process 600 includes step 662 of sending new video encoding properties (e.g., resolution and frame rate) from the rate and video encoding properties control (e.g., 407) to the video encoder (e.g., 409). Process 600 includes step 664 of sending a new target rate from the rate and video encoding properties control (e.g., 407) to the video encoder (e.g., 409).

[0137] After transmitting step 660 or transmitting step 664, process 600 includes a step 666 of receiving the audio and video PES streams at a multiplexer (e.g., 413) and adjusting the multiplexing rate based on the audio and video PES rates. Process 600 includes a step 668 of transmitting the multiplexed bitrate rate from the multiplexer (e.g., 413) to the rate and video encoding property control (e.g., 407). Process 600 returns to monitoring step 648.

[0138] If any existing connections exist (632="yes"), after access step 636, process 600 includes step 670 of waiting for an RTCP packet response at the RTP responding router (e.g., 425). Process 600 includes step 672 of determining whether the UDP socket connection endpoint (e.g., 421) has received an RTCP packet (672="yes"), process 600 proceeds to step 674. If the UDP socket connection endpoint (e.g., 421) has not received an RTCP packet (672="no"), process 600 returns to waiting step 670.

[0139] Process 600 includes step 674 of receiving an RTCP packet at an RTCP response router (e.g., 425) of a network congestion control (e.g., 423). Process 600 includes step 676 of removing the RTCP packet from a packet response data store for the RTCP packet's SRC IP address (e.g., 472, 429) at the RTCP response router (e.g., 425). Process 600 includes step 678 of determining whether one of the packet response data stores is waiting for an RTCP response. If one of the packet response data stores is waiting for an RTCP response (678="yes"), process 600 returns to waiting step 670. If one of the packet response data stores is not waiting for an RTCP response (678="no"), process 600 includes step 680 of saving the SRC IP address as the worst-case client device in the network congestion control (e.g., 423). The process 600 includes a step 682 of sending the CWND and RTT from the RTCP reporting system (431) to the transmission scheduler (e.g., 419). The process 600 returns to the access step 636.

[0140] 7 depicts a system 700 that manages peer-to-peer connections from remote controller input and focuses on user invitations by console owners to receive game-rendered streams from game consoles, according to some embodiments of the present disclosure. System 700 also includes a "controller delegation" component and interface for delegating controllers to both remote and local users.

[0141] System 700 may be similar to and / or include some or all of the features described above with respect to system 400. In some embodiments, system 700 provides for local and / or remote play via game console 701. Game console 701 acts as a server and forms peer-to-peer connections with one or more remote client devices. Game console 701 may include and / or connect to a video device (not shown) and an audio device (not shown), which may be integrated into game console 701 or provided separately and connected to game console 701 via separate communication links. Game console 701 may locally connect to one or more controllers, e.g., game controller 1 731, game controller 2 733, game controller 3 735, and game controller 4 737, via separate communication links. Game console 701 may connect to a mobile or fixed line network 739 and / or a mobile or fixed line network 761 via communication links. Game console 701 is connectable to one or more remote client devices, e.g., remote client device 1 741a, remote client device 2 741b, and remote client device 3 741c, via mobile or fixed line network 739 and separate communication links. In the illustrated embodiment, remote client device 1 741a, remote client device 2 741b, and remote client device 3 741c are connected to game controllers 759a, 759b, and 759c via separate communication links. In some examples, the controllers are not separate from the remote client devices, and game play control is controlled by the remote client devices themselves (not shown). Although mobile or fixed line network 739 and mobile or fixed line network 761 are each illustrated as a single network, multiple networks may be utilized.Although mobile or fixed line network 739 and mobile or fixed line network 761 are shown as separate networks, they may be the same network.

[0142] 4 , game console 701 of system 700 includes a transport bandwidth control system 705, a UA / US / FC controller 729, and a remote play controller 727, which are described in detail herein. System 700 also includes a CGP / CS 763 with a UAAS 769 and a remote player management controller 765. Furthermore, remote client device 1 741 a, remote client device 2 741 b, and remote client device 3 741 c each include a game remote play client app 743 a, a game remote play client app 743 b, and a game remote play client app 743 c, respectively, which are described in detail herein. Further still, game remote play client app 743a, game remote play client app 743b, and game remote play client app 743c each include a remote play controller 749a, remote play controller 749b, and remote play controller 749c, respectively, as described in detail herein. Further still, game remote play client app 743a, game remote play client app 743b, and game remote play client app 743c each include a UA / US / FC management controller, respectively, as described in detail herein.

[0143] In some embodiments, the game console 701 includes at least one of a game engine 703, a transport bandwidth control system 705, a UDP socket 1 (connection endpoint 1) or an RTP delivery socket 707, a controller handler 1 709, a controller handler 2 711, a controller handler 3 713, a controller handler 4 715, a controller input / output mapper 717, a UDP socket 2 (connection endpoint 2) 719, a UDP socket 3 (connection endpoint 3) 721, a UDP socket 4 (connection endpoint 4) 723, a UDP socket 5 (connection endpoint 5) 725, a remote play controller 727, or a UA / US / FC controller 729.

[0144] In some embodiments, game engine 703 is configured to send raw video and audio data to transport bandwidth control system 705. Transport bandwidth control system 705 is configured to send RTP stream packets to, and receive RTCP packets from, UDP socket 1 707. UDP socket 1 707 is configured to send RTP stream packets to, and receive RTCP packets from, one or more of client device 1 741a, client device 1 741b, and client device 1 741c.

[0145] In some embodiments, Controller Handler 1 709, Controller Handler 2 711, Controller Handler 3 713, Controller Handler 4 715, Controller Input / Output Mapper 717, UDP Socket 2 (Connection Endpoint 2) 719, UDP Socket 3 (Connection Endpoint 3) 721, UDP Socket 4 (Connection Endpoint 4) 723, and UDP Socket 5 (Connection Endpoint 5) 725 are configured to function in a manner similar to that described above with respect to similarly labeled structures (e.g., 433-449) of system 400. That is, the controller handlers are configured to send and receive controller input and tactile data to and from one or more controllers (e.g., 731-737), and the UDP sockets are configured to send and receive controller input and tactile data to and from one or more remote client devices (e.g., 741a-741c).

[0146] In some embodiments, the UA / US / FC controller 729 is configured to exchange information with the UAAS 769 of the CGP / CS 763. For example, the UA / US / FC controller 729 is configured to send a user login along with a certificate 729a to the UAAS 769, which is configured to return a user authenticated response along with a user identification 769a. The UA / US / FC controller 729 is also configured to send a user search (or friend list) request along with a user identification 729b to the UAAS 769, which is configured to return a user list along with a username and user identification response 769b.

[0147] In some embodiments, the UAAS 769 is configured to access, modify, and / or store user account information in the data store 771.

[0148] In some embodiments, such as the embodiment of Figure 7, the remote client devices are (but are not necessarily) functionally identical. Thus, like reference numbers for remote client device 1 741a have the same description and functionality as like reference numbers for remote client device 2 741b and remote client device 3 741c. Duplicate descriptions are omitted for brevity.

[0149] In some embodiments, remote client device 1 741a includes at least one of a game remote play client app 743a, a UDP socket 745a, or a UDP socket 755a. For example, game remote play client app 743a includes at least one of a media transmission and rendering controller 747a, a remote play controller 749a, a UA / US / FC management controller 751a, or a controller for data transmission 753a. As illustrated in the embodiment of FIG. 7, media transmission and rendering controller 747a is configured to receive RTP-multiplexed encoded video and audio packets and transmit RTCP packets from UDP socket 745a. Media transmission and rendering controller 747a is configured to transmit rendered audiovisual frames to output device 757a (e.g., a video device and an audio device). Remote play controller 749a is configured to send an RTP streaming connection endpoint to UDP socket 745a and a connection controller connection endpoint to UDP socket 755a. The controller for data transmission 753a is configured to send controller input to the UDP socket 755a and game controller 759a and receive tactile data therefrom.

[0150] In some embodiments, the remote player management controller 765 and UAAS 769 of the CGP / CS 763 are configured to exchange information with one or more remote client devices (e.g., 741a-741c). Details herein will be understood to refer to the exchange of information between the remote player management controller 765 and UAAS 769 of the CGP / CS 763 and remote client device 1 741a, as well as if additional remote client devices were connected to the system 700.

[0151] In some embodiments, the UA / US / FC management controller 751a of the remote client device 1 741a is configured to exchange information with the UAAS 769. For example, the UA / US / FC management controller 751a is configured to send a user login along with a credential (S1) to the UAAS 769, which is configured to return a user authenticated response along with a user identity (S2).

[0152] In some embodiments, the remote play controller 749a of the remote client device 1 741a is configured to exchange information with the remote player management controller 765 of the CGP / CS 763. For example, the remote player management controller 765 is configured to send at least one of the following to the remote play controller 749a: a user invite (S3) with a user identification, a requester username and user identification, an RTP streaming connection endpoint, and a controller connection UDP 1 connection endpoint, a UDP 2 connection endpoint, a UDP 3 connection endpoint, a UDP 4 connection endpoint, and a remote session identifier (shown in the drawing as "remote session identifier"); A controller handover request (to the targeted user identity) (S6) with the controller number and UDP connection endpoint (or controller number), or A user list (S7) in the remote session response with user names (shown in the drawing as "user_names") and user identifications.

[0153] For example, the remote play controller 749a is configured to send at least one of the following to the remote player management controller 765: a Controller Connected Response (S4) with the controller number, UDP connection endpoint, remote session identifier, and user identification; A controller handover request (S5) accompanied by user identification and controller number; A list of users in the remote session request (S8), along with the remote session identifier; Controller connected mapping (UDP socket SRC connection endpoint), remote session identifier, and user identification request (S9), or Controller 1 enforces mapping (UDP socket SRC connection endpoint), remote session identifier, user identification request (S10).

[0154] In some embodiments, the remote play controller 727 is configured to exchange information with the remote player management controller 765 of the CGP / CS 763. For example, the remote play controller 727 is configured to send a remote session request to the remote player management controller 765 along with a user identification 727d, which is configured to return a remote session response along with a remote session identifier 765a. The remote play controller 727 is configured to send a user to invite to the remote player management controller 765 along with a user identification list with the requesting username and user identification, an RTP streaming connection endpoint, and controller connection UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and remote session identifier 727e. The remote play controller 727 is configured to send a controller handover request to the remote player management controller 765 along with the user identification and controller number (or controller number, UDP connection endpoint, and remote session identifier) 727f, which is configured to return a response 765b thereto. The remote player management controller 765 is configured to send the controller connected mapping (UDP socket SRC connection endpoint), remote session identifier, and user identification request 765c to the remote play controller 727. The remote play controller 727 is configured to send a user list remote session request along with the remote session identifier 727g to the remote player management controller 765, which is configured to return the user list along with the username and user identification 765d in a remote session response to the remote play controller 727. The remote player management controller 765 is configured to send a controller connected response to the remote play controller 727 along with the controller number, UDP connection endpoint, remote session identifier, and user identification 765e.

[0155] In some embodiments, the remote player management controller 765 is configured to access, modify, and / or store in the data storage device 767 the devices registered per remote play session and user identification.

[0156] In some embodiments, the controller input / output mapper 717 is configured to exchange information with the remote play controller 727. For example, the controller input / output mapper 717 is configured to send a local controller handover request (along with a controller input number) and a UDP connection endpoint to the remote play controller 727. The remote play controller 727 is configured to return to the controller input / output mapper 717 at least one of a controller connected response 727a with the controller number and the UDP connection endpoint, a controller number mapping request 727b for local or remote UDP port mapping, or a controller 1 forced mapping request 727c (with a UDP socket SRC connection endpoint).

[0157] FIG. 8 depicts a process 800 for a user to use a CD / CS application on a remote device to start a remote session, invite friends, and send connection information to the remote device, according to some embodiments of the present disclosure. The interface is defined, for example, in FIG. 7. Process 800 includes inviting a remote player to a game via a console or cloud service. The user logs into the system (e.g., 700), selects friends to invite, and sends the invitation. If the user searches for a name, the system checks the user identity and adds the searched user to the invite list. The system then manages the remote session, ensuring that only authenticated users can join. This ensures a secure and efficient gaming experience.

[0158] In some embodiments, process 800 includes step 804 (CD / CS) of logging the console / game owner into the console device (e.g., 701) or CGP / CS (e.g., 763) using the user login along with the credentials. The method includes step 808 of sending a User Authentication, User Lookup, and Friend Connection (UA / US / FC) Management user login along with the credentials from the CD / CS to a UAAS (e.g., 769) of the CGP / CS. Process 800 includes step 812 of sending a User Authenticated Response along with the user identification from the UAAS of the CGP / CS to the CD / CS and sending a UA / US / FC response to the console device or CD / CS. Process 800 includes step 816 of determining whether the user selected an option to invite a remote player from the console device, game service client, or user interface with the game paused. If the user selects the option to invite a remote player from the console device, the game service client, or the user interface with the game paused (816="yes"), process 800 proceeds to step 820. If the user does not select the option to invite a remote player from the console device, the game service client, or the user interface with the game paused (816="no"), process 800 ends (as shown), pauses, or repeats decision step 816 824 (not shown).

[0159] Process 800 includes step 820 of sending a request (e.g., a friends list) along with a user identification to a UAAS of a CGP / CS, for example, from a UA / US / FC controller 729. Process 800 includes step 828 of sending the user list along with a username and user identification response from the UAAS to a UA / US / FC controller (e.g., 729). Process 800 includes step 832 of determining whether the user selected friends to invite from the list. If the user selected friends to invite from the list (832="yes"), process 800 proceeds to step 836; if the user did not select friends to invite from the list (832="no"), process 800 proceeds to step 840.

[0160] Process 800 includes step 836 of adding the selected user identity to an invitation list for the user to invite in the CD / CS. The method includes step 840 of determining whether the user entered a name and selected search. If the user did not enter a name and selected search (840="no"), process 800 proceeds to step 844; if the user entered a name and selected search (840="yes"), process 800 proceeds to step 848.

[0161] Process 800 includes a step 844 for determining whether the user selected the send invite option. If the user selected the send invite option (844="yes"), process 800 proceeds to step 872; if the user did not select the send invite option (not shown), process 800 repeats decision step 844.

[0162] Process 800 includes step 848 of sending a search request from the UAAS with the user's name and user identification. Process 800 includes step 852 of determining whether the user identification is found in the UAAS. If the user identification is found in the UAAS (852="yes"), process 800 includes step 856 of sending from the UAAS a user list with the user name and a user identification response to the user authentication, along with the machine username and user identification, to the UA / US / FC. If the user identification is not found in the UAAS (852="no"), process 800 includes step 860 of sending from the UAAS an optional null response to the user list and user authentication, along with the user name, along with the machine username and user identification, to the UA / US / FC.

[0163] Process 800 includes a step 864 of determining whether the user added the searched user to the invitation list. If the user added the searched user to the invitation list (864="yes"), process 800 includes a step 868 of adding the selected user identity to the invitation list for the user to invite in the CD / CS and returning to decision step 840. If the user did not add the searched user to the invitation list (864="no"), process 800 returns to decision step 840.

[0164] Process 800 includes step 872 of sending a remote session request from the remote play controller of the CD / CS to the remote player management of the CGP / CS, along with the user identification and the remote session identifier. Process 800 includes step 876 of sending a remote session response from the remote player management of the CGP / CS or from the remote play controller of the CD / CS, along with the remote session identifier, to the remote play controller of the CD / CS. Process 800 includes starting process 600 of Figure 6. Process 800 includes step 884 of sending from the remote play controller of the CD / CS to the remote player management of the CGP / CS one or more users to invite, along with a user identification list with the requesting username and user identification, an RTP streaming connection endpoint, and controller connection UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and the remote session identifier. The process 800 includes a step 888 of sending a user invite request, along with the requesting username and user identification, the RTP streaming connection endpoint, and the controller connection UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and remote session identifier, from the remote player management of the CGP / CS to each invited user.

[0165] 9 depicts a system 900, in accordance with some embodiments of the present disclosure, that focuses on user invitations from a remote device by a console owner, for example, when the user is not at home. The overall structure of system 900, in some embodiments, is similar to that of system 700. Descriptions of reference numerals 901-971 in FIG. 9 correspond to reference numerals 701-771 in FIG. 7, respectively, and are omitted for brevity.

[0166] In some embodiments, the UA / US / FC management controller 951 is configured to send the user login along with the certificate 951a to the UAAS 969 of the CGP / CS 963. The UAAS 969 is configured to send a user authenticated response along with the user identification 969b to the UA / US / FC management controller 951.

[0167] In some embodiments, the remote player management controller 965 of the CGP / CS 963 is configured to initiate 965a a remote service request using the remote session identifier. In response, the remote player controller 927 is configured to initiate 927a a remote service response using the remote session identifier, the user identification, the RTP streaming connection endpoint, and the controller connection controller UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and the remote session identifier.

[0168] In some embodiments, the remote player management controller 965 is configured to access, modify, and / or store in data 967 the devices registered per remote play session and user identification.

[0169] In some embodiments, the remote play controller 949 is configured to send a user device request to the remote player management controller 965 along with the user identification 949a. The remote player management controller 965 is configured to send a user device response to the remote play controller 949 along with a list of user devices with device identifiers (shown in the drawings as "device_ID") 965b. The remote play controller 949 is configured to send a remote session request to the remote player management controller 965 along with the user identification and device identifiers 949b. The remote player management controller 965 is configured to send a remote service initiation response to the remote play controller 949 along with the remote session identifier, user identification, RTP streaming connection endpoint, and controller connection controller UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and remote session identifier 965c. The remote play controller 949 is configured to send a user search (or friend list) request to the UAAS 969 along with the user identification 949c. The UAAS 969 is configured to send the user list to the remote play controller 949 along with the username and user identification response 969a.

[0170] 10 depicts a process 1000 for a user to use the Remote Play application to start a remote session, invite friends, and send connection information to a remote device, according to some embodiments of the present disclosure. The interface is defined, for example, in FIG.

[0171] In some embodiments, process 1000 includes step 1003 of logging the console / game owner into the remote client application using the user login along with credentials entered into the UA / US / FC management of the remote application and sending the user login along with the credentials to the UAAS. Process 1000 includes step 1006 of sending a response from the UAAS to the UA / US / FC of the remote application, authenticating the user with the user identification. Process 1000 includes step 1009 of sending a user device request, along with the user identification, from the remote application's remote play controller to the CGP / CS's remote player management. Process 1000 includes step 1012 of sending a user device response, along with a list of user devices with device identifiers, from the CGP / CS's remote player management to the remote application's remote play controller. Process 1000 includes step 1015 of determining whether the user has selected the CGP / CS's remote player management. If the user selects remote player management for CD / CS (1015="Yes"), the method proceeds to step 1018. If the user does not select remote player management for CD / CS (not shown), the method returns to decision step 1015.

[0172] Process 1000 includes step 1018 of sending a remote session request from the remote application's remote play controller to the CGP / CS's remote player, along with the user identification and device identifier. Process 1000 includes step 1021 of saving, in the remote player management, the remote connection IP address (from the previous request) as the remote client device as the owner's device for the remote session identifier. Process 1000 includes step 1024 of sending, from the CGP / CS's remote player management, a remote service start request, along with the remote session identifier, to the CGP / CS's remote play controller. Process 1000 includes step 1027 of starting process 600 of FIG. 6.

[0173] Process 1000 includes step 1030 of sending a remote service initiation response from the remote play controller of CGP / CS to the remote player management of CGP / CS, along with the remote session identifier, user identification, RTP streaming connection endpoint, and controller connection controller UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and remote session identifier. Process 1000 includes step 1033 of sending a remote service initiation response from the remote player management of CGP / CS to the remote play controller of the remote application, along with the remote session identifier, user identification, RTP streaming connection endpoint, and controller connection controller UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and remote session identifier. Process 1000 includes step 1036 of connecting the remote play controller of the remote application to the console device using the RTP streaming UDP connection endpoint and the controller connection controller 1 UDP connection endpoint. Process 1000 includes step 1039 of decoding and rendering the game console display, sending controller input, and receiving tactile feedback on a UDP port connected to controller 1 in the remote application. Process 1000 includes step 1042 of conducting all interactions from the console using the remotely connected controller 1 through the remote display.

[0174] Process 1000 includes step 1045 of determining whether the user has selected a friend to invite from the list. In response to the user having selected a friend to invite from the list (1045="yes"), process 1000 proceeds to step 1048. In response to the user not having selected a friend to invite from the list (1045="no"), process 1000 proceeds to step 1051. Process 1000 includes step 1048 of adding the selected user identity to an invitation list for the user to invite in the CD / CS. Process 1000 includes step 1051 of determining whether the user has not entered a name and selected search. In response to the user not having entered a name and not selected search (1051="no"), the method proceeds to step 1054. In response to the user having entered a name and selected search (1051="yes"), the method proceeds to step 1063.

[0175] Process 1000 includes step 1057 of sending users to invite from the remote play controller of CGP / CS to the remote player management of CGP / CS, along with a user identification list with the requesting user name and user identification, the RTP streaming connection endpoint, and the controller connecting controller 2 connection endpoint, controller 3 connection endpoint, controller 4 connection endpoint, and remote session identifier. Process 1000 includes step 1060 of sending a user invite request from the remote player management of CGP / CS to each invited user, along with the requesting user identification, the user name and user identification, the RTP streaming connection endpoint, and the controller connecting controller 2 connection endpoint, controller 3 connection endpoint, controller 4 connection endpoint, and remote session identifier.

[0176] Process 1000 includes step 1063 of sending a search request from the UAAS, along with the user's name and user identification, to the UAAS. Process 1000 includes step 1066 of determining whether the user identification is found in the UAAS. In response to the user identification being found in the UAAS (1066="yes"), process 1000 includes step 1069 of sending a user list with the user name and a user identification response for user authentication from the UAAS, along with the machine user name and user identification. Process 1000 proceeds to step 1072. In response to the user identification not being found in the UAAS (1066="no"), process 1000 includes step 1078 of sending a user list with the user name and a null response for user authentication from the UAAS, along with the machine user name and user identification. The process 1000 proceeds to decision step 1045 .

[0177] Process 1000 includes a step 1072 for determining whether the user has added the searched user to an invitation list. In response to the user having added the searched user to the invitation list (1072="Yes"), process 1000 includes a step of adding, in the CD / CS, the selected user identity to an invitation list for the user to invite. In response to the user not adding the searched user to the invitation list (1072="No"), process 1000 proceeds to decision step 1045.

[0178] FIG. 11 depicts a process 1100 for a remote invitation request from an account owner logged in as a remote user, according to some embodiments of the present disclosure.

[0179] In some embodiments, process 1100 includes step 1105 of logging the user into the remote client application using the user login along with credentials entered into the UA / US / FC management of the remote application and sending the user login, along with the credentials, to the UAAS. Process 1100 includes step 1110 of logging the user into the CGP / CS through the remote client application. Process 1100 includes step 1115 of determining whether the user received a notification for a remote play invitation and accepted the invitation. In response to the user receiving a notification for a remote play invitation and accepting the invitation (1115="yes"), process 1100 proceeds to step 1120. In response to the user not receiving a notification for a remote play invitation and not accepting the invitation (not shown), determination step 1115 is repeated.

[0180] Process 1100 includes step 1120 of sending a user invitation acceptance response, along with the user identification and the remote session identifier, from the remote play controller of the remote client application to the remote play controller of the CD / CS. Process 1100 includes step 1125 of saving, in the remote play controller of the CD / CS, an IP address along with the user identification for the remote session identifier. Process 1100 includes step 1130 of receiving, in the remote play controller of the remote application, a session reconnection request <remote session identifier>, an RTP port, and a controller port from the remote play controller of the CD / CS. Process 1100 includes step 1135 of connecting the remote play controller of the remote application to an RTP streaming UDP connection endpoint of the console device. Process 1100 includes step 1140 of decoding and rendering a game console display in the remote application.

[0181] Process 1100 includes a connection success check for each controller port 1045. Process 1100 includes step 1150 of attempting a connection to a controller connection endpoint by a remote application's remote play application. Process 1100 includes step 1155 of determining whether the connection is successful. If the connection is successful (1155="yes"), process 1100 includes step 1160 of setting a success flag to true. If the connection is not successful (not shown), process 1100 includes step 1165 of terminating the connection success check.

[0182] Process 1100 includes step 1170 of determining whether a success flag is set to true or false. If the success flag is set to true (1170="true"), process 1100 includes step 1175 of sending controller input from the remote application and receiving tactile feedback on the UDP port connected to the controller remote UDP port. If the success flag is set to false (1170="false"), process 1100 includes step 1180 of displaying a user message (e.g., "All controllers are in use. Please wait until a player transfers a controller," or equivalent).

[0183] FIG. 12 depicts a process 1200 for initial controller connection for both local and remote users, according to some embodiments of the present disclosure.

[0184] In some embodiments, process 1200 includes step 1204 of determining whether the console is powered on locally. If the console is powered on locally (1204="Yes"), process 1200 includes step 1208 of determining whether the controller is powered on by a game controller. If the console is not powered on locally (not shown), determination step 1204 is repeated. If the controller is powered on by a game controller (1208="Yes"), process 1200 includes step 1212 of assigning the controller as controller 1. If the controller is not powered on by a game controller (not shown), determination step 1208 is repeated. Process 1200 includes step 1216 of restricting primary control to controller 1 at the console.

[0185] Process 1200 includes step 1220 of determining whether an additional controller is powered on. If the additional controller is not powered on (1220="No"), process 1200 proceeds to step 1232. If the additional controller is powered on (1220="Yes"), process 1200 includes step 1222 of determining whether all console controller slots are in use. If all console controller slots are not in use (1222="No"), process 1200 includes step 1228 of assigning a controller to the next available controller slot. If all console controller slots are in use (1222="Yes"), process 1200 includes step 1224 of setting a status (e.g., console does not accept controller connection). Process 1200 returns to decision step 1220.

[0186] Process 1200 includes step 1232, which determines whether a remote user has joined the controller UDP port. If a remote user has joined the controller UDP port (1232="yes"), process 1200 includes step 1236, which determines whether all console controller slots are in use. If a remote user has not joined the controller UDP port (not shown), decision step 1232 is repeated. If all console controller slots are in use (1236="yes"), process 1200 proceeds to step 1264.

[0187] If all console controller slots are not in use (1236="No"), process 1200 includes step 1240, which determines whether the remote user has joined the CD / CS as a service account owner. If the remote user has joined the CD / CS as a service account owner (1240="Yes"), process 1200 proceeds to step 1248. If the remote user has not joined the CD / CS as a service account owner (1240="No"), process 1200 includes step 1244, which maps the console UDP controller input to the next available controller slot. Process 1200 returns to decision step 1220.

[0188] Process 1200 includes a step 1248 of determining whether another remote or local game controller is connected. If another remote or local game controller is connected (1248="Yes"), process 1200 includes a step 1260 of unlocking options in the remote application to forcefully assign a console controller (including, for example, assigning controller 1), and process 1200 returns to decision step 1220.

[0189] If no other remote or local game controllers are connected (1248="No"), process 1200 includes step 1252 of sending a controller 1 forced mapping (e.g., UDP socket SRC connection endpoint) from the remote application to the CGP / CS. Process 1200 includes step 1256 of sending a controller number input mapping request from the game management application of the CGP / CS to the local or remote UDP port mapping controller I / O mapper, where the remote UDP port is the owner controller UDP connection and the mapping is to the controller 1 slot.

[0190] Process 1200 includes a step 1264 of determining whether the remote user has joined the CD / CS as a service account owner. If the remote user has joined the CD / CS as a service account owner (1264="Yes"), process 1200 includes a step 1268 of unlocking an option in the remote application to forcefully disconnect a connected local or remote user from a controller slot. Process 1200 proceeds to an unlock step 1260.

[0191] If the remote user is not joined to the CD / CS as a service account holder (1264="No"), process 1200 includes step 1272 of refusing the UDP controller port connection at the console. Process 1200 includes step 1276 of displaying a message (e.g., "Controller port is not available") to the user at the remote application and prompting the user (e.g., to "Request controller control" or "Wait until the player hands over the controller" or equivalent). Process 1200 returns to decision step 1220.

[0192] FIG. 13 depicts a process 1300 for controller delegation between local and remote users according to some embodiments of the present disclosure.

[0193] In some embodiments, process 1300 includes step 1302, determining whether all controller slots on the CGP / CS are filled. Process 1300 includes step 1304, determining whether a remote user is connected to a remote play game console session. Process 1300 includes step 1306, determining whether a remote user (user identification) connected to a controller slot is stored in the remote player management of the CGP / CS for a remote session identifier. Process 1300 includes step 1308, determining whether a single-player or multiplayer game is in progress. Process 1300 includes step 1310, determining whether a local player using a controller has paused the game and selected the "controller handover" option. If the local player using a controller has paused the game and not selected the "controller handover" option (1310="No"), process 1300 proceeds to step 1340.

[0194] If the local player using the controller pauses the game and selects the "controller handover" option (1310="Yes"), process 1300 includes step 1312, which determines whether a single-player or multiplayer game is in progress. Process 1300 includes step 1314, which sends a local controller handover request from the controller I / O mapper to the CD / CS, along with the controller input # and UDP connection endpoint. Process 1300 includes step 1316, which sends a user list from the remote play controller of the CD / CS to the remote player management of the CGP / CS in a remote session request, along with the controller number (shown in the figure as "controller #"), UDP connection endpoint, and remote session identifier. Process 1300 includes step 1318, which sends a user list remote session response from the remote player management of the CGP / CS to the remote play controller of the CD / CS, along with the user name and user identification. Process 1300 includes step 1320 of displaying a list of remote users at the console for user selection of a remote user to "transfer the controller." Process 1300 includes step 1322 of selecting a user (e.g., a controller number) to "transfer the controller" to by a user using a local controller. Process 1300 includes step 1324 of sending a "controller transfer" request from the remote play controller of the CD / CS to the remote player management of the CGP / CS, along with a user identification, a controller number, a remote session identifier, and a UDP connection endpoint. Process 1300 includes step 1326 of sending a "controller transfer" request from the remote player management of the PCGP / CS to the remote play controller of the remote player application, along with the targeted user identification, controller number, and UDP connection endpoint. Process 1300 includes step 1328 of connecting the remote player management of the remote play application to the controller UDP connection endpoint.Process 1300 includes step 1330 of sending a controller connected response from the remote play controller of the remote play application to the remote player management of CGP / CS along with the controller number, UDP connection endpoint, remote session identifier, and user identification. Process 1300 includes step 1332 of sending a controller connected response from the remote player management of the remote play application to the remote player management of CGP / CS along with the controller number, UDP connection endpoint, remote session identifier, and user identification. Process 1300 includes step 1334 of sending a controller connected response from the remote player management of CGP / CS to the remote play controller of CD / CS along with the controller number, UDP connection endpoint, remote session identifier, and user identification. Process 1300 includes step 1336 of sending a controller connected response from the remote play controller of CD / CS to the controller I / O mapper along with the controller number and UDP connection endpoint. The process 1300 includes a step 1338 of switching, by the controller I / O mapper, the local game controller input for the controller number to the incoming data on the remote controller UDP connection endpoint.

[0195] Process 1300 includes step 1340, which determines whether a remote player, using a controller, has paused the game and selected the "Transfer Controllers" option to another remote player. If the remote player using the controller has not paused the game and has not selected the "Transfer Controllers" option (1340="No"), process 1300 proceeds to step 1356.

[0196] If the remote player using the controller pauses the game and selects the “controller handover” option (1340="yes"), process 1300 includes step 1342 of sending a user list from the remote play controller of the remote play application to the remote player management of CGP / CS in a remote session request, along with the controller number, UDP connection endpoint, and remote session identifier. Process 1300 includes step 1344 of sending a user list remote session response from the remote player management of CGP / CS to the remote play controller of the remote play application, along with the user name and user identification. Process 1300 includes step 1346 of displaying a list of remote users from which the remote player application can select a remote user to “handover the controller.” Process 1300 includes a step of selecting a user to “hand over the controller” by the user using the remote player application. Process 1300 includes step 1350 of sending a "controller handover" request from the remote play controller of the remote play application to the remote player management of the CGP / CS, along with the user identification, controller number, remote session identifier, and UDP connection endpoint. Process 1300 includes step 1352 of sending a "controller handover" request from the remote player management of the CGP / CS to the remote play controller of the remote player application, along with the targeted user identification, controller number, and UDP connection endpoint. Process 1300 includes step 1354 of disconnecting the remote player application from the controller UDP connection endpoint for claiming user / device. Process 1300 proceeds to connection step 1328.

[0197] Process 1300 includes step 1356, which determines whether the remote player, using a controller, paused the game and selected the "Transfer Controller" option to a console game controller. If the remote player using a controller did not pause the game and select the "Transfer Controller" option to a console game controller (1356="No"), process 1300 proceeds to step 1368.

[0198] If the remote player using the controller pauses the game and selects the "controller handover" option to the console game controller (1356="yes"), process 1300 includes step 1358 of sending a "controller handover" request from the remote play controller of the remote play application to the remote player management of the CGP / CS, along with the user identification and the controller number. Process 1300 includes step 1360 of sending a "controller handover" request from the remote player management of the CGP / CS, along with the user identification and the controller number, to the remote play controller of the CD / CS. Process 1300 includes step 1362 of sending a local controller handover request from the remote play controller of the CD / CS, along with the controller input #, without a UDP connection IP connection endpoint, to the controller I / O mapper. Process 1300 includes step 1364 of switching the local game controller input for the controller number to the local game controller input connection in the controller I / O mapper. The process 1300 includes a step 1366 of disconnecting the remote play controller of the remote play application from the UDP socket for controller communication.

[0199] Process 1300 includes step 1368 of determining whether the local user has taken over the local controller and whether the controller has detected activity (e.g., an internal measurement sensor, a button press, or the like). If the local user has not taken over the local controller and the controller has not detected activity (1368="No"), process 1300 returns to decision step 1308. If the local user has taken over the local controller and the controller has detected activity (1368="Yes"), process 1300 includes step 1370 of switching the local game controller input for the controller number in the controller I / O mapper. Process 1300 includes step 1372 of disconnecting the remote user from the UDP socket assigned to that controller number in the controller I / O mapper. Process 1300 includes step 1374 of displaying a message to the remote user that the local console has controller input for that controller number.

[0200] FIG. 14 depicts a process 1400 for a remote owner to request control on a primary controller input (e.g., controller input 1) or force a disconnect when all controller slots are in use, according to some embodiments of the present disclosure.

[0201] In some embodiments, process 1400 includes step 1405 of determining whether the console owner is logged in to the remote application. Process 1400 includes step 1410 of determining whether the controller 1 slot on the game console or PC gaming system provider is occupied by a local user or a remote user. Process 1400 includes step 1415 of determining whether the owner has elected to forcibly grant control to controller 1. If the owner has not elected to forcibly grant control to controller 1 (1415="No"), process 1400 continues to step 1460.

[0202] If the owner chooses to forcibly grant control to controller 1 (1415="yes"), process 1400 includes step 1420 of sending a controller 1 forcible mapping (e.g., UDP socket SRC connection endpoint), remote session identifier, and user identification request from the remote application to the remote player management of the CGP / CS. Process 1400 includes step 1425 of sending a controller 1 forcible mapping (e.g., UDP socket SRC connection endpoint) request from the remote player management of the CGP / CS to a remote player controller of the remote player management of the CGP / CS. Process 1400 includes step 1430 of sending a controller 1 forcible mapping (e.g., UDP socket SRC connection endpoint) request from the remote player controller of the remote player management of the CGP / CS to the controller I / O mapper.

[0203] Process 1400 includes step 1435 of determining whether all UDP sockets are occupied. If all UDP sockets are not occupied (1435="no"), process 1400 includes step 1440 of remapping the remote (non-owner) user's connected controller UDP socket to an unused controller connection in the controller I / O mapper. Process 1400 includes step 1445 of mapping the owner's connected controller UDP socket connection to controller input 1 in the controller I / O mapper. If all UDP sockets are occupied (1435="yes"), process 1400 includes step 1450 of disconnecting the connected remote user from the mapped controller UDP socket in the controller I / O mapper and enabling a connection to the owner's mapped controller UDP socket. Process 1400 includes step 1455 of displaying a message to the disconnected remote user that the owner has requested control and that the user is not in spectator mode.

[0204] Process 1400 includes step 1460 of determining whether all UDP sockets are occupied. If all UDP sockets are not occupied (1460="No"), process 1400 includes step 1465 of assigning a free controller mapping to a UDP port to the owner, just like any other remote user. If all UDP sockets are occupied (1460="Yes"), process 1400 includes step 1470 of providing the owner with a list of remote users connected to controller inputs. Process 1400 includes step 1475 of selecting a user for removal of control by the owner. Process 1400 includes step 1480 of sending a controller number forced mapping (e.g., UDP socket SRC connection endpoint) request (for the remote user) to the controller I / O mapper from the remote player management of the CGP / CS remote play controller. Process 1400 includes step 1485 of sending a controller number forced mapping (e.g., UDP socket SRC connection endpoint) response (for the remote user) from the controller I / O mapper to the remote player management of the remote play controller of the CGP / CS. Process 1400 includes step 1490 of disconnecting the connected remote user from the controller UDP socket mapped to the controller number input (for the remote user) in the controller I / O mapper and enabling connection of the owner controller to the mapped controller UDP socket. Process 1400 proceeds to display step 1455.

[0205] FIG. 15 depicts a system 1500 focused on determining whether a user has sufficient bandwidth based on a standard value (e.g., a determined minimum value) that would cause the user to disconnect, according to some embodiments of the present disclosure. For example, Sony's PS Remote Play requires a minimum of 5 Mbps but recommends 15 Mbps. If a player's connection has poor network QoS, exemplified by a 5 Mbps connection, it will be removed from the peer-to-peer ultra-low latency RTP stream. In response, in some embodiments, system 1500 includes a feature that switches to a HyperText Transfer Protocol (HTTP) ABR stream (e.g., HTTP Live Streaming (HLS), Dynamic Adaptive Streaming over HTTP (known as MPEG DASH), Microsoft Smooth Streaming (SS), or Adobe HTTP Dynamic Streaming (HDS)) delivered from the cloud for those users. If system 1500 determines that the QoS has improved, the user's connection to the console or PC will be restored to a peer-to-peer ultra-low latency connection. However, when in ABR delivery mode, it is in spectator-only mode.

[0206] The overall structure of system 1500, in some embodiments, is similar to one or more portions of systems 400, 700, and 900. For example, the description of reference numerals 1501-1527 of FIG. 15, which in some embodiments correspond to, for example, reference numerals 401-431 of FIG. 4, are omitted for brevity. Note that in the embodiment of FIG. 4, the video encoder 409, the audio encoder 411, and the multiplexer 413 are shown and described. In contrast, it should be understood that in the embodiment of FIG. 15, the functions of the video encoder, audio encoder, and multiplexer are performed by the video and / or audio (V / A) encoder and multiplexer 1509. Other reference numerals 1511-1527 of FIG. 15, in some embodiments, correspond to, for example, reference numerals 415-431 of FIG. 4, respectively.

[0207] In some embodiments, the remote play controller 1529 of FIG. 15 includes one or more features of the remote play controller 727 of the embodiment of FIG. 7 or the remote play controller 927 of the embodiment of FIG. 9, which are omitted for brevity.

[0208] In some embodiments, reference numbers 1531-1559 in Figure 15 correspond to, for example, reference numbers 433-463 in Figure 4 and are omitted for brevity. The remote slots in Figure 15 are not numbered but, in some embodiments, are similar to remote slot 440. The controller I / O mapper is omitted in Figure 15 but, in some embodiments, is similar to, for example, controller I / O mapper 441.

[0209] In some embodiments, system 1500 includes client device 1 1561a and client device 2 1561b. In some embodiments, such as that of Figure 15, the remote client devices are the same (but not necessarily the same), i.e., client device 1 1561a is identical in structure to client device 2 1561b. Accordingly, details of client device 2 1561b are omitted for brevity.

[0210] In some embodiments, client device 1 1561a includes at least one of ABR player 1565a, remote play controller 1567a, UA / US / FC management controller 1569a, controller for data transmission 1571a, UDP socket 1573a, or UDP socket 1575a. For example, remote play controller 1567a is configured to send a disconnect force message to one of UDP sockets 1573a, 1575a. In some embodiments, ABR player 1565a is configured to send rendered A / V frames to output device 1577a (e.g., a video device and an audio device).

[0211] In some embodiments, system 1500 includes a console game provider or cloud service (CGP / CS; the cloud service may be a PC game publisher cloud service) 1583. CGP / CS 1583 includes, for example, a remote player management controller 1585 and a remote viewing session viewing ABR system 1587. In some embodiments, remote player management controller 1585 is configured to access, modify, and / or store remote play sessions in data storage 1586.

[0212] In some embodiments, the remote viewing session viewing ABR system 1587 includes at least one of an ABR delivery controller 1589, a UDP socket 1591, a transmission receiver 1593, a demultiplexer 1595, an ABR transcoder and segmenter 1597, an ABR A / V segment and manifest store 1598, or an ABR HTTP delivery system 1599.

[0213] In some embodiments, ABR delivery controller 1589 is configured to send an RTP connection (with a connection endpoint) to UDP socket 1591. UDP socket 1591 is configured to send RTP-multiplexed encoded video and audio packets to transmission receiver 1593, which is configured to return RTCP packets. Transmission receiver 1593 is configured to send the RTP-multiplexed encoded video and audio packets to demultiplexer 1595. Demultiplexer 1595 is configured to send the encoded video PES and encoded audio PES to ABR transcoder and segmenter 1597. ABR transcoder and segmenter 1597 is configured to send encoded Common Media Application Format (CMAF) video segments, encoded audio segments, and ABR live manifests to ABR A / V segment and manifest store 1598. ABR A / V segment and manifest store 1598 is configured to allow access by ABR HTTP delivery system 1599. The ABR HTTP delivery system 1599 is configured to access requested encoded CMAF video segments, requested encoded audio segments, and requested manifests / live manifest updates.

[0214] In some embodiments, the UDP socket 1517 is configured to exchange RTP-multiplexed encoded video and audio packets with, e.g., send 1517a, 1591a to, the UDP socket 1591 and receive RTCP packets (e.g., along with a remote cloud service IP address). In some embodiments, the UDP socket 1517 is configured to send / receive RTCP packets to / from a client device's UDP socket (e.g., along with a remote client device IP address). In some embodiments, the remote play controller 1592 is configured to send 1529a a force disconnect request to the remote player management controller 1585 along with the remote session identifier, IP address, and minimum bitrate requirement. In some embodiments, the remote play controller 1592 is configured to send 1529b an ABR session start to the remote player management controller 1585 along with the remote session identifier.

[0215] In some embodiments, the remote player management controller 1585 is configured to send 1585a an ABR session start, along with the RTP connection (connection endpoint) and the remote session identifier, to the ABR delivery controller 1589. The ABR delivery controller 1589 is configured to send 1589a an ABR live manifest URL to the remote player management controller 1585. The remote player management controller 1585 is configured to send 1585b an ABR session end, along with the remote session identifier, to the ABR delivery controller 1589.

[0216] In some embodiments, ABR HTTP delivery system 1599 is configured to send 1599b ABR segment downloads to ABR player 1565a. ABR player 1565a is configured to download and play ABR segments and receive live manifest updates (e.g., at 1565a1 and 1565a2). ABR player 1565a is configured to send 1565a3 the calculated bitrate to a remote play controller of the CS's remote player management.

[0217] In some embodiments, the remote player management controller 1585 is configured to send 1585c the ABR live manifest URL to the ABR player 1565a. The remote play controller 1567a and the remote player management controller 1585 are configured to exchange, e.g., send 1567a1 and receive 1585d, force disconnect requests over a list of UDP connection endpoints. The remote player management controller 1585 is configured to send 1585e a session reconnect request (including the remote session identifier), the RTP UDP port, and the controller UDP port to the remote play controller 1567a.

[0218] 16 depicts a process 1600 for determining whether sufficient bandwidth exists to support a remote client session, according to some embodiments of the present disclosure. In some embodiments, the process 1600 initiates an ABR session and terminates the ABR session based on a low-bandwidth client that is part of the remote gaming session. The process 1600 also includes disconnecting the low-bandwidth client from the RTP transmitter service, monitoring network QoS during the ABR session, and re-establishing the RTP transmitter service connection if QoS improves above a lower threshold.

[0219] In some embodiments, process 1600 includes step 1603 of determining whether the rate and video encoding property control determines that the target bit rate is less than the minimum value for the bit rate to encoding property mapping. If the rate and video encoding property control determines that the target bit rate is less than the minimum value for the bit rate to encoding property mapping (1603="Yes"), process 1600 includes step 1606 of sending a low bandwidth notification from the rate and video encoding property control to the RTCP reporting system. Process 1600 includes step 1609 of sending a client device disconnect request from the RTCP reporting system to the remote play controller along with the IP address of the worst-case client (e.g., the last one to report an RTCP packet based on packets sent).

[0220] Process 1600 includes step 1612 of sending a disconnection force request from the remote play controller of the console device to the remote play controller of the remote player management of the CGP / CS along with the remote session identifier, IP address, and minimum bit rate requirement request. Process 1600 includes step 1615 of sending a disconnection force request from the remote play controller of the remote player management of the CGP / CS to the remote play controller of the game remote play client application for the device IP address that matches the IP address stored for that client device on a list of UDP IP connection endpoints. Process 1600 includes step 1618 of forcing, at the remote play controller, a disconnection from the received connection endpoint.

[0221] In some embodiments, after sending step 1609, in a process parallel to steps 1612-1618, process 1600 includes step 1621 of sending an ABR session start (along with the remote session identifier) from the CD / CS to a remote play manager in a remote play controller in the CGP / CS. Process 1600 includes step 1624 of looking up an RTP connection (including connection endpoints) for the remote session identifier in the remote player manager in the CGP / CS.

[0222] Process 1600 includes step 1627 of determining whether the ABR delivery controller session has already activated the remote session identifier. In response to a determination that the ABR delivery controller session has already activated the remote session identifier (1627="yes"), process 1600 proceeds to step 1651. In response to a determination that the ABR delivery controller session has not already activated the remote session identifier (1627="no"), process 1600 proceeds to step 1630.

[0223] Process 1600 includes step 1630 of sending an ABR session initiation from the CGP / CS remote player management to the ABR delivery controller along with an RTP connection endpoint and a remote session identifier. Process 1600 includes step 1633 of initiating an ABR session instance in the ABR session controller using an RTP transmission receiver, a demultiplexer, an ABR transcoder, and a segmenter. Process 1600 includes step 1636 of connecting the transmission receiver of the ABR delivery controller to the RTP connection endpoint. Process 1600 includes step 1639 of receiving an RTP-multiplexed A / V packet stream in the RTP receiver. Process 1600 includes step 1642 of demultiplexing the multiplexed A / V packet stream into video and audio PES packet streams. Process 1600 includes step 1645 of sending the video and audio PES streams to the ABR encoder and segmenter. Process 1600 includes step 1648, in the ABR segmenter, generating a live manifest and writing segments to the ABR A / V segment and manifest storage. Process 1600 includes step 1651, in which a remote play controller of the remote player management of the CGP / CS sends an ABR live manifest URL notification to a remote play controller of a remote play client application whose IP address matches the low-bandwidth reporting IP address. Process 1600 includes step 1654, in the remote play controller of the remote play client application, starting an instance of an ABR video player and sending the live ABR manifest to the ABR player. Process 1600 includes step 1657, in the ABR video player of the remote play client application, downloading and playing ABR segments and receiving live manifest updates. Process 1600 includes step 1660, in the ABR video player of the remote play client application, sending the calculated bitrate to the remote play controller of the remote player management of the CS.

[0224] Process 1600 includes step 1663 of determining whether the reported bit rate is high enough to reconnect the peer-to-peer RTP UDP streaming session. If the reported bit rate is not high enough to reconnect the peer-to-peer RTP UDP streaming session (1663="No"), process 1600 returns to step 1657 of downloading, playing, and receiving. If the reported bit rate is high enough to reconnect the peer-to-peer RTP UDP streaming session (1663="Yes"), process 1600 includes step 1666 of sending a session reconnection request (including the remote session identifier), the RTP UDP port, and the controller UDP port from the remote play controller of the remote player management of the CGP / CS to the remote play controller of the remote play client application. Process 1600 includes step 1669 of initiating one or more steps of process 1200 of FIG. 12.

[0225] Process 1600 includes step 1672 of determining whether any low-bandwidth clients exist in the remote player management for the remote session ID. If any low-bandwidth clients exist in the remote player management for the remote session ID (1672="Yes"), process 1600 includes step 1675 of sending an ABR session termination request from the remote player management to the ABR delivery controller, along with the remote session identifier.

[0226] Additional features, systems, and methods are provided. It is understood that techniques for establishing local and remote sessions are fully disclosed and encompassed herein. In some embodiments, the system can detect nearby devices that advertise themselves as game controllers. This advertisement is initiated in response to an action taken by a user within a specific app, such as the Sony PlayStation app, or at the operating system level, for example, via a quick menu setting. The advertisement is provided via a communication protocol, such as Bluetooth Low Energy (BLE), or when the device is connected to the same local area network as the game console.

[0227] In response to detecting a nearby device and its associated user profile, the game console takes specific actions. These may include asking the console user whether they want to allow the detected device (and its associated user profile) to join the current game play, listing the device as a controller in the console system while it is connected, or sorting the list of remote players. For example, when a user is using the present invention for peer-to-peer connection or "controller handover" activity, nearby users are moved to the top of the list of remote users.

[0228] "Play Share" is a PlayStation feature that allows one player to share their screen with a second, remote player and "hand over" the controller to that player so that the other player can play instead. For example, the second player can share their screen and help the first player complete a level or defeat an enemy. This feature allows screens to be shared in real time when a second user is invited and the session is already active.

[0229] As an improvement to game sharing, a "controller takeover" feature is provided. This feature allows a second, remote player to take over the controller at a later time convenient for them to assist the first player. The first player assigns such a task to a group (e.g., a WhatsApp group) or a specific player or players via a direct invite or message to their social network, SMS, or messaging app. The invite is a secure link to the game console, specifically the shared game session. In addition, the first player has the option to sort through multiple players and assign them a priority order. This is useful if the first player "passes" or declines to assist the first player, in which case the second player on the list is notified or invited to take over the controller after a predetermined time. Such a time can be defined by the first player. In one embodiment, when the "controller takeover" feature is invoked, the game console enters "share mode," preventing the recipient of the invitation from controlling game console functionality on the game console (e.g., browsing apps, photos, and the like). In this mode, only the shared game is accessible.

[0230] "Controller takeover" shares the game's existing session ID or allows it to persist in the game console's database until another player can join. The game state of the existing session is saved or persisted, and the sharing player defines tasks for the other players. This may include, for example, eliminating specific targets in a war game or defeating a boss. In some embodiments, a second user can resume the game for the first user, but with level and / or task restrictions. The session can be accessible for a period defined by the players (e.g., via the console) or can expire when the first player attempts to resume their game.

[0231] In one embodiment, when a second player takes over the controller, the first player is notified so that they can view the second player on their device, such as their mobile device. Other features, such as voice chat, are also activated. The notification to the first player establishes a streaming session with the game console, either locally (when the user is on the same LAN as the game console) or over the Internet.

[0232] Other methods are also contemplated. For example, a seamless gaming experience is provided, where a user transitions between devices or games without interruption. This feature enhances the user experience by providing flexibility and convenience. In some embodiments, a user can start a game on one device, such as an iPhone®, and then continue on a different device, such as a PS5 console, without losing their progress. Similarly, the ability to accept an invite in the middle of another game and have the new session automatically load after quitting the current game is also provided.

[0233] FIG. 17 depicts a user experience (UX) 1700 after a user pauses a game with the option to invite remote players to join the game, according to some embodiments of the present disclosure. UX 1700 includes indicators and user-selectable buttons. For example, UX 1700 includes at least one of a player 1 pause indicator 1710, a resume button 1720, a settings button 1730, a skip step button 1740, an invite remote player button 1750 (highlighting in FIG. 17 indicates selection), or an end game button 1760. In response to selection of the invite remote player button 1750, at least one of the methods, processes, and steps detailed herein for inviting remote players is performed.

[0234] 18 depicts a UX 1800 including an option for a user to invite remote players to earn spots in a game, according to some embodiments of the present disclosure. In some embodiments, the UX 1800 includes a user-selectable option 1810 to invite remote players. In some embodiments, the option 1810 includes the text "Invite Remote Players" and a graphic of a controller and a cloud, along with an arrow, indicating a connection between the controller and the cloud.

[0235] FIG. 19 depicts a UX 1900 including an option for the console owner to select a remote player to view the main console as if the remote player were playing with the console owner, according to some embodiments of the present disclosure. In some embodiments, the console owner maintains control of the game controller 1, and any other remote or local player has the option to join a game initiated by the console owner. The console owner retains control of navigation and selection of a game to play. Once a game is initiated, local or remote users can join the game, as in FIG. 16 . In some embodiments, the UX 1900 includes an icon 1910 indicating that a remote user has successfully formed a peer-to-peer connection between the game console and at least one or more remote client devices. In some embodiments, the icon 1910 includes a graphic of a controller and a cloud, along with an arrow indicating a connection between the controller and the cloud. The icon 1910 may also include a status indicator, such as a green circle in the lower right corner of the controller graphic, indicating an active peer-to-peer connection.

[0236] FIG. 20 depicts a UX 2000 including options that allow a user to transfer a controller to a remote user or a local user to take over control of a game controller locally, according to some embodiments of the present disclosure. In some embodiments, the UX 2000 includes a user-selectable option 2010 to invite a remote player, accompanied by a graphic of a controller and a cloud, along with an arrow indicating a connection between the controller and the cloud. In some embodiments, the option 2010 includes a text field (e.g., "Transfer control to another player") 2020. See the related description of the "Controller Transfer" option provided herein.

[0237] Figure 21 depicts a UX2100 in a spectator mode according to some embodiments of the present disclosure. Figure 22 depicts a UX2200 in a game play mode according to some embodiments of the present disclosure.

[0238] In some embodiments, a process for remote play for a game running on a game console is provided. For example, the process includes forming a peer-to-peer connection between the game console and one or more remote client devices, and, in response to loss of the peer-to-peer connection between the game console and one or more remote client devices, causing the game to be played in a spectator mode (e.g., FIG. 21 ), highlighting an icon (e.g., 2120) corresponding to a player controlled at the one or more remote client devices with which the peer-to-peer connection was lost, and controlling the actions of the player with which the peer-to-peer connection was lost by artificial intelligence trained on the player's gameplay (see FIGS. 23 and 25 and related discussion). In response to restoration of the peer-to-peer connection between the game console and one or more remote client devices, the process includes causing the game to be played in a gameplay mode (e.g., FIG. 22 ).

[0239] For example, when a game is in spectator mode (e.g., FIG. 21), the game may display a message overlay 2110 including the sign "Spectator Mode," a graphic of the controller and cloud with an arrow indicating a potential connection between the controller and the cloud, an indicator of loss of connection (e.g., a red circle over part of the controller graphic, not shown), and a message (e.g., "You are currently in spectator mode. Remote gameplay will resume when network quality improves.").

[0240] Also, for example, when the game is in gameplay mode (e.g., FIG. 22), the game may display a message overlay 2210 including the indicator "Spectator Mode," a graphic of the controller and cloud with an arrow indicating the connection between the controller and the cloud, and a message (e.g., "Network quality has improved. Remote gameplay has resumed.") Also, for example, the controller graphic may include a status indicator (not shown), such as a green circle in the lower right corner of the controller graphic, indicating an active peer-to-peer connection.

[0241] In some embodiments associated with at least one of the UX1700-2200, a process is provided for prompting the addition of one or more remote client devices to a game console controlled by a local operator of the game console. The process includes, for example, inviting at least one of the one or more remote client devices to connect to the game console for display on a local display connected to the game console, and providing a selectable option. The process includes, in response to selection of the selectable option by at least one of the one or more remote client devices, forming a peer-to-peer connection between the game console and at least one of the one or more remote client devices.

[0242] In some embodiments, the process includes at least one of the following steps: in response to the local operator pausing the game, providing a selectable option for display; in response to launching a multiplayer game on the game console, providing a selectable option to win a spot in the multiplayer game for display; in response to the local operator of the game console providing an option to allow one or more remote client devices to view the main console of the game console as if the one or more remote client devices were themselves playing with the local operator; or in response to one or more remote client devices pausing the game, providing a selectable option for the remote client device to transfer controller to the local operator or another of the one or more remote client devices for display to the one or more remote client devices.

[0243] In some embodiments, reinforcement learning (RL) is used to train a model of a player for use as a proxy user, for example, after the player moves to spectator mode. Examples of RL used to train a model of a player for use as a proxy user and related topics are described, for example, in U.S. Patent Application Nos. 18 / 241,106 and 18 / 241,109, both of which were filed August 31, 2023, to Dasher et al. (each of which is incorporated by reference in its entirety).

[0244] For example, a proxy user is an emulated player character that may be controlled by a game system or another user and that mimics the behavior and preferences of the original player character. RL is a machine learning technique that enables an agent to learn its own actions and rewards in an environment without requiring explicit monitoring or labeled data. An RL architecture for training a proxy user may be provided. In some embodiments, the RL architecture comprises an agent, an environment, a policy, and a reward function. The agent is the proxy user, the environment is a video game, the policy is a function that maps the agent's state to actions, and the reward function is a function that evaluates the agent's performance and provides feedback. In some embodiments, the RL training process involves the agent interacting with the environment, observing the states and rewards, and updating the policy based on a learning algorithm.

[0245] The RL approach differs from traditional methods in which AI models govern non-player character (NPC) behavior based on predefined rules or simpler learning algorithms. RL-based proxy users are generated using historical game data and can emulate the playing styles, biases, and tendencies of real users.

[0246] Proxy users can be used within game sessions initiated through social media platforms, allowing for iterative training of NPC models as more players interact with them. This results in a more personalized and immersive experience for other players. The system uses rewards and return features to fine-tune proxy user profiles over thousands or millions of plays, resulting in models that better represent real players.

[0247] In some embodiments, features are recorded and extracted from a user's speech data to train a custom speech model.

[0248] FIG. 25 depicts an illustrative flowchart of a process 2500 for monitoring players using an AI model, according to some embodiments of the present disclosure. Process 2500 may be included, in whole or in part, within process 2300 shown in FIG. 23. Process 2500 may be implemented, in whole or in part, by a cloud server with AI, such as cloud server 763 of system 700 shown in FIG. 7. One or more actions of process 2500 may be incorporated within or combined with one or more actions of any other process or embodiment described herein. Process 2500 may be stored in a memory or storage device as one or more instructions or routines that may be executed by a corresponding device or system to implement process 2500.

[0249] In some embodiments, during gameplay, a cloud server actively monitors players, with a particular focus on players who share the game on social media platforms. The server uses machine learning algorithms and AI to learn about the player's abilities and playing style. This data is used to replicate the player's performance and behavior emulation in "ghost mode." In addition to learning the first user's skills and style, the system will also learn about the player's in-game preferences. This includes their favorite characters, weapons, strategies, and the like, providing a more realistic representation of the first user in ghost mode. Different algorithms, such as reinforcement learning, can also be used, where the AI is rewarded or penalized based on moves made in the game. In supervised learning, the AI algorithm learns from labeled training data and applies this knowledge to new data.

[0250] For example, different aspects of player behavior can be labeled and categorized (offensive, defensive, magic-loving, stealth-oriented, etc.), and supervised learning algorithms such as decision trees or neural networks can be trained on this data. Deep learning is a form of machine learning that uses artificial neural networks with multiple layers (hence the term "deep"), and can handle complex, high-dimensional data. Convolutional neural networks (CNNs) can be used for image-based data (such as recognizing preferred in-game locations or actions from screen images), while recurrent neural networks (RNNs) or long-short-term memory networks (LSTMs) can handle sequential data, such as a series of actions taken by a player. Pre-trained models are often trained on large, general datasets. Transfer learning is fine-tuned for specific tasks. For example, an AI model that has generally been trained on gameplay data from many players can be fine-tuned to mimic the style and preferences of a specific player.

[0251] Thus, at step 2510 of process 2500, a gameplay session is initiated. At step 2520, the system monitors players within the gameplay session. At step 2530, machine learning and AI analysis adjusts the models discussed above. At step 2540, the adjusted models are now able to replicate player behavior and their performance and provide a representative experience to other users joining their game session.

[0252] Predictive Model

[0253] Throughout this disclosure, in some embodiments, decisions, predictions, likelihoods, and the like are determined using one or more predictive models. For example, FIG. 23 depicts a predictive model. Prediction process 2300, in some embodiments, includes a predictive model 2350. Predictive model 2350 receives as input various forms of data about one, some, or all users, media content items, devices, and data described in this disclosure. Predictive model 2350 performs analysis based on at least one of hard rules, learning rules, hard models, learning models, usage data, load data, analysis thereof, metadata, profile information, combinations thereof, or the like. Predictive model 2350 outputs one or more predictions of future states of any of the devices described in this disclosure. Load increase events are determined by load balancing processes, such as least connections, least bandwidth, round robin, server response time, weighted versions thereof, resource-based processes, and address hashing. The predictive model 2350 is based on inputs including at least one of hard rules 2305, user-defined rules 2310, content provider-defined rules 2315, hard models 2320, learned models 2325, combinations thereof, or the like.

[0254] The predictive model 2350 receives as input the usage data 2330. The predictive model 2350, in some embodiments, is based on at least one of a usage pattern of a user or media device, a usage pattern of a requesting media device, a usage pattern of a media content item, a usage pattern of a communication system or network, a usage pattern of a profile, a usage pattern of a media device, a combination thereof, or the like.

[0255] The prediction model 2350 receives as input the load balancing data 2335. The prediction model 2350 is based on at least one of display device load data, requesting media device load data, media content item load data, communication system or network load data, profile load data, media device load data, combinations thereof, or the like.

[0256] The predictive model 2350 receives as input the metadata 2340. The predictive model 2350 is based on at least one of streaming service metadata, requesting media device metadata, media content item metadata, communication system or network metadata, profile metadata, media device metadata, combinations thereof, or the like. The metadata includes information of types represented in a media device manifest.

[0257] The predictive model 2350 is trained using the data. The training data, in some embodiments, is developed using one or more data processes, including, but not limited to, data selection, data sourcing, and data synthesis. The predictive model 2350, in some embodiments, is trained using one or more analytical processes, including, but not limited to, classification and regression trees (CART), discrete choice models, linear regression models, logistic regression, logit vs. probit, multinomial logistic regression, multivariate adaptive regression splines, probit regression, regression processes, survival or duration analysis, and time series models. The predictive model 2350, in some embodiments, is trained using one or more machine learning approaches, including, but not limited to, supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and dimensionality reduction. The predictive model 2350, in some embodiments, includes regression analysis (ANOVA), linear regression, logistic regression, ridge regression, and / or time series, including analysis of variance. The predictive model 2350, in some embodiments, includes classification analysis, including decision trees and / or neural networks. In FIG. 23, a depiction of a multi-layer neural network is provided as a non-limiting example of the predictive model 2350, where the neural network includes an input layer (left), three hidden layers (center), and an output layer (right), with 32 neurons and 192 edges, which is intended to be illustrative and not limiting. The predictive model 2350 is based on a data engineering and / or modeling process. The data engineering process includes exploration, cleaning, normalization, feature engineering, and scaling. The model process includes model selection, training, evaluation, and tuning. The predictive model 2350 is operationalized using a training, deployment, monitoring, and / or retraining process.

[0258] The predictive model 2340 is configured to output results to a device or devices that include means for implementing one, many, or all of the features referenced herein of the systems, methods, processes, inputs, and outputs of one or more of Figures 1-22, 24, and 25, in any suitable combination. The device is at least one of a server 2355, a tablet 2360, a media display device 2365, a network-connected computer 2370, a media device 2375, a computing device 2380, combinations thereof, or the like.

[0259] The prediction model 2350 is configured to output a current state 2381, and / or a future state 2383, and / or a decision, prediction, or likelihood 2385, and the like. The current state 2381, and / or the future state 2383, and / or the decision, prediction, or likelihood 2385, and the like, may be compared 2390 to a predetermined or determined standard value. In some embodiments, the standard value is met (2390 = OK) or rejected (2390 = not OK). If the standard value is met or rejected, the prediction process 2300 outputs at least one of the current state, future state, decision, prediction, likelihood to any device or module disclosed herein, a combination thereof, or the like.

[0260] communication systems

[0261] FIG. 24 depicts a block diagram of a system 2400 according to some embodiments. The system is shown to include a computing device 2402, a server 2404, and a communications network 2406. While a single instance of a component may be illustrated and described with respect to FIG. 24 , it should be understood that additional embodiments of the components may be employed. For example, the server 2404 may include or be incorporated within more than one server. Similarly, the communications network 2406 may include or be incorporated within more than one communications network. The server 2404 is shown as communicatively coupled to the computing device 2402 through the communications network 2406. Although not shown in FIG. 24 , the server 2404 may be communicatively coupled directly to the computing device 2402, for example, in a system that lacks or bypasses the communications network 2406.

[0262] Communications network 2406 may include one or more network systems, such as, but not limited to, the Internet, a LAN, Wi-Fi, wireless, or other network systems suitable for audio processing applications. System 2400 of FIG. 24 excludes server 2404; functionality that would otherwise be implemented by server 2404 is instead implemented by other components of the system depicted by FIG. 24, such as one or more components of communications network 2406. In yet other embodiments, server 2404 works in conjunction with one or more components of communications network 2406 to implement certain functionality described herein in a distributed or collaborative manner. Similarly, the system depicted by FIG. 24 excludes computing device 2402; functionality that would otherwise be implemented by computing device 2402 is instead implemented by other components of the system depicted by FIG. 24, such as one or more components of communications network 2406 or server 2404, or a combination thereof. In other embodiments, the computing device 2402 interfaces with one or more components of the communications network 2406 or the server 2404 to implement certain functionality described herein in a distributed or collaborative manner.

[0263] Computing device 2402 includes control circuitry 2408, a display 2410, and input / output (I / O) circuitry 2412. Control circuitry 2408 may be based on any suitable processing circuitry and may include control circuitry and memory circuitry, which may be located on a single integrated circuit or may be discrete components. As referred to herein, processing circuitry should be understood to mean circuitry based on at least one microprocessor, microcontroller, digital signal processor, programmable logic device, field programmable gate array (FPGA), application specific integrated circuit (ASIC), system on a chip (SoC), application specific standard component (ASSP), indium phosphide (InP)-based monolithic integration and silicon photonics, non-classical devices, organic semiconductors, compound semiconductors, "More Moore" devices, "More than Moore" devices, cloud-computing devices, combinations thereof, or the like, and may include multi-core processors (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores). In some embodiments, the processing circuitry may be distributed across multiple separate processors or processing units, for example, multiple processing units of the same type (e.g., two Intel Core i9 processors) or multiple different processors (e.g., an Intel Core i7 processor and an Intel Core i9 processor). Some control circuitry may be implemented in hardware, firmware, or software. Control circuitry 2408 in turn includes communications circuitry 2426, memory 2422, and processing circuitry 2418. Either of control circuitry 2408 and 2434 may be utilized to perform or implement any or all of the systems, methods, processes, inputs, and outputs, or any combination thereof (e.g., enabled by processing circuitry 2418 and 2436, respectively) of one or more of Figures 1-23 and 25.

[0264] In addition to control circuitry 2408 and 2434, computing device 2402 and server 2404 may each include storage (storage 2422 and storage 2438, respectively). Storage 2418 and 2438 may each be an electronic storage device. As referred to herein, the phrase “electronic storage device” or “storage device” should be understood to mean any device for storing electronic data, computer software, or firmware, such as random-access memory, read-only memory, cloud-based storage, hard drive, optical drive, digital video disc (DVD) recorder, compact disc (CD) recorder, BLU-RAY® Disc (BD) recorder, BLU-RAY 8D disc recorder, digital video recorder (DVR, sometimes called personal video recorder or PVR), solid-state device, quantum storage device, game console, game media, or any other suitable fixed or removable storage device, and / or any combination thereof. The storage devices 2422 and 2438 may each be used to store some types of content, metadata, and / or other types of data. Non-volatile memory may also be used (e.g., for launching boot-up routines and other instructions). Cloud-based storage may be used to supplement or in place of the storage devices 2418 and 2438. In some embodiments, user profiles and messages corresponding to chains of communication may be stored within one or more of the storage devices 2418 and 2438. The storage devices 2418 and 2438 may each be utilized to store commands, for example, when the processing circuits 2418 and 2436, respectively, are prompted through the control circuits 2408 and 2434, respectively. Either the processing circuit 2418 or 2436 may perform any of the system, method, process, input, and output steps, or any combination thereof, of one or more of Figures 1-23 and 25.

[0265] In some embodiments, control circuitry 2408 and / or 2434 executes instructions for an application stored in memory (e.g., storage 2422 and / or storage 2438). Specifically, control circuitry 2408 and / or 2434 may be instructed by the application to perform the functions discussed herein. In some embodiments, any action performed by control circuitry 2408 and / or 2434 may be based on instructions received from the application. For example, an application may be implemented as software and / or a set of one or more executable instructions that may be stored in storage 2422 and / or 2438 and executed by control circuitry 2408 and / or 2434. An application may be a client / server application, where only a client application resides on computing device 2402 and a server application resides on server 2404.

[0266] The application may be implemented using any suitable architecture. For example, it may be a stand-alone application implemented entirely on computing device 2402. In such an approach, instructions for the application are stored locally (e.g., in storage 2422) and data for use by the application is downloaded on a periodic basis (e.g., from an out-of-band feed, from an Internet resource, or using another suitable approach). Control circuitry 2408 may read the instructions for the application from storage 2422 and process the instructions to perform the functionality described herein. Based on the processed instructions, control circuitry 2408 may determine the type of action to perform in response to input received from I / O circuitry 2412 or from communications network 2406.

[0267] Computing device 2402 is configured to communicate with I / O devices via I / O circuitry 2412. The I / O devices include any suitable devices. In some embodiments, user input 2414 is received from the I / O devices. Wired and / or wireless connections between I / O circuitry 2412 and the I / O devices are provided in some embodiments.

[0268] In a client / server-based embodiment, control circuitry 2408 may include communications circuitry suitable for communicating with an application server (e.g., server 2404) or other networks or servers. Instructions for performing the functionality described herein may be stored on the application server. The communications circuitry may include a cable modem, an Ethernet card, or a wireless modem for communication with other devices or any other suitable communications network. Such communications may involve the Internet or any other suitable communications network or path (e.g., communications network 2406). In another example of a client / server-based application, control circuitry 2408 launches a web browser that interprets web pages provided by a remote server (e.g., server 2404). For example, the remote server may store instructions for the application in a storage device.

[0269] The remote server may use circuitry (e.g., control circuitry 2434) to process the stored instructions and / or generate a display. Computing device 2402 may receive a display generated by the remote server and display the contents of the display locally via display 2410. For example, display 2410 may be utilized to present text. In this manner, the processing of instructions is performed remotely (e.g., by server 2404), while the resulting display, such as a display window described anywhere herein, is provided locally on computing device 2404. Computing device 2402 may receive inputs from a user via input / output circuitry 2412 and transmit those inputs to the remote server for processing and to generate a corresponding display.

[0270] Alternatively, computing device 2402 may receive input from a user via input / output circuitry 2412 and process and display the received input locally via control circuitry 2408 and display 2410, respectively. For example, input / output circuitry 2412 may correspond to one or more keyboards and / or a set of speakers / microphones used to receive user input (e.g., input such as that displayed in a search bar or display of FIG. 24 on the computing device). Input / output circuitry 2412 may also correspond to a communications link between display 2410 and control circuitry 2408 so as to update display 2410 in response to input received via input / output circuitry 2412 (e.g., simultaneously updating the content shown on display 2410 based on the received input by generating corresponding output based on instructions stored in memory via a non-transitory computer-readable medium).

[0271] The server 2404 and the computing device 2402 may transmit and receive content and data, such as media content, over the communications network 2406. For example, the server 2404 may be a media content provider, and the computing device 2402 may be a smart television configured to download or stream media content, such as live news broadcasts, from the server 2404. The control circuitry 2434, 2408 may transmit and receive commands, requests, and other suitable data over the communications network 2406 using the communications circuitry 2432, 2426, respectively. Alternatively, the control circuitry 2434, 2408 may communicate with each other directly using the communications circuitry 2432, 2426, respectively, avoiding the communications network 2406.

[0272] It should be understood that computing device 2402 is not limited to the embodiments and methods shown and described herein. In non-limiting examples, computing device 2402 may be a television, a smart TV, a set-top box, an integrated receiver decoder (IRD) for controlling satellite television, a digital storage device, a digital media receiver (DMR), a digital media adapter (DMA), a streaming media device, a DVD player, a DVD recorder, a connected DVD, a local media server, a BLU-RAY® player, a BLU-RAY® recorder, a personal computer (PC), a laptop computer, a tablet computer, a Web TV box, a personal computer television (PC / TV), a PC media server, a PC media center, a handheld computer, a landline phone, a personal digital assistant (PDA), a mobile phone, a portable video player, a portable music player, a portable game console, a smartphone, or any other device, computing appliance, or wireless device, and / or combination thereof, capable of suitably displaying and manipulating media content.

[0273] The computing device 2402 receives user input 2414 at input / output circuitry 2412. For example, the computing device 2402 may receive user input such as a user swipe or a user touch. It should be understood that the computing device 2402 is not limited to the embodiments and methods shown and described herein.

[0274] User input 2414 may be received from a user-selected capture interface that is separate from device 2402, such as a remote control device, a trackpad, or any other suitable user movement-sensing, audio-sensing, or capture device, or as part of device 2402, such as the touchscreen of display 2410. Transmission of user input 2414 to computing device 2402 may be accomplished using a wired connection, such as an audio cable, a Universal Serial Bus (USB) cable, an Ethernet cable, and the like, attached to a corresponding input port on the local device, or may be accomplished using a wireless connection, such as Bluetooth, Wi-Fi, WiMAX, GSM, UTMS, CDMA, TDMA, 8G, 4G, 4G LTE, 5G, NearLink, ultra-wideband technology, or any other suitable wireless transmission protocol. The input / output circuitry 2412 may include a physical input port such as a 12.5 mm (0.4921 inch) audio jack, an RCA audio jack, a USB port, an Ethernet port, or any other suitable connection for receiving audio via a wired connection, or may include a wireless receiver configured to receive data via Bluetooth, Wi-Fi, WiMAX, GSM, UTMS, CDMA, TDMA, 3G, 4G, 4G LTE, 5G, NearLink, ultra-wideband technology, or other wireless transmission protocol.

[0275] Processing circuitry 2418 may receive user input 2414 from input / output circuitry 2412 using communication path 2416. Processing circuitry 2418 may convert or translate received user input 2414, which may be in the form of audio data, visual data, gestures, or movements, into a digital signal. In some embodiments, input / output circuitry 2412 performs the conversion to a digital signal. In some embodiments, processing circuitry 2418 (or processing circuitry 2436, as the case may be) performs the disclosed processes and methods.

[0276] Processing circuitry 2418 may provide the request to storage device 2422 over communications path 2420. Storage device 2422 may provide the requested information to processing circuitry 2418 over communications path 2446. Storage device 2422 may forward the request for information to communications network 2426, which may convert or encode the request for information into a format receivable by communications network 2406 before forwarding the request for information over communications path 2428. Communications network 2406 may forward the converted or encoded request for information to communications network 2432 over communications path 2430.

[0277] In communications circuitry 2432, the converted or encoded request for information received over communications path 2430 is converted or decoded for processing circuitry 2436, which will provide a response to the request for information based on information available through control circuitry 2434 or storage device 2438, or a combination thereof. The response to the request for information is then provided back to communications network 2406 in an encoded or converted format over communications path 2440, such that communications network 2406 forwards the encoded or converted response back to communications network 2426 over communications path 2442.

[0278] In communications circuitry 2426, the encoded or converted response to the request for information may be provided over communications path 2454 directly back to processing circuitry 2418, or may be provided over communications path 2444 to storage device 2422, which then provides the information to processing circuitry 2418 over communications path 2446. Processing circuitry 2418 may also provide a request for information over communications path 2452 directly to communications circuitry 2426, with storage device 2422 responding over communications path 2424 or 2446 to a request for information (provided over communications path 2420 or 2444) that storage device 2422 does not contain information related to the request from processing circuitry 2418.

[0279] Processing circuitry 2418 may process responses to requests received over communication paths 2446 or 2454 and may provide instructions to display 2410 for notifications to be provided to the user over communication path 2448. Display 2410 may incorporate a timer for providing the notification or may rely on input from the user over input / output circuitry 2412, which is automatically forwarded through processing circuitry 2418 over communication path 2448, to determine the length or format for providing the notification. Once display 2410 determines that the display is complete, the notification may be provided to processing circuitry 2418 over communication path 2450.

[0280] The communication paths provided in FIG. 24 between computing device 2402, server 2404, communication network 2406, and all depicted subcomponents are examples and may be modified by one skilled in the art to reduce processing time or improve processing performance steps within the processes disclosed herein.

[0281] Terminology

[0282] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure.

[0283] As used herein, the term "extended reality (XR)" includes at least one of augmented reality (AR), three-dimensional (3D) content, four-dimensional (4D) experiences, virtual reality (VR), mixed reality (MR), interactive experiences, control using next-generation user interfaces (next-generation UIs), combinations thereof, or the like.

[0284] As used herein, the terms "real-time," "simultaneous," "substantially on-demand," and the like, are understood to be near instantaneous, but may include delays due to practical limitations of the system. Such delays may be on the order of milliseconds, microseconds, or less, depending on the application and nature of the processing. Longer delays (e.g., greater than a few milliseconds) may occur due to communication or processing delays, particularly in remote and cloud computing environments.

[0285] As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It is further understood that, as used herein, the terms "comprises" and / or "comprising" specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0286] While at least some example embodiments are described as using multiple units or modules to perform a process or processes, it should be understood that a process or processes may also be performed by one or more units or modules. Additionally, it should be understood that the term controller / control unit may refer to a hardware device including a memory and a processor. The memory may be configured to store the unit or module, and the processor may be specifically configured to execute the unit or module and perform one or more processes described herein.

[0287] Unless specifically stated or clear from the context, as used herein, the term "about" is understood to mean within the normal tolerance in the art, for example, within 2 standard deviations of the mean. "About" can be understood as within 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, 0.5%, 0.1%, 0.05%, or 0.01% of the stated value. Unless otherwise clear from the context, all numerical values provided herein are modified by the term "about."

[0288] The use of the terms "first," "second," "third," etc. herein is provided to identify structures or operations without describing the order of the structures or operations, and to the extent that a structure or operation is used in an embodiment, the structure may be provided, or the operations may be performed, in an order other than that described, unless a specific order is clearly dictated by the context.

[0289] Methods, processes, and / or any instructions for implementing any of the embodiments discussed herein may be encoded on a computer-readable medium. A computer-readable medium includes any medium capable of storing data. The computer-readable medium may be transient, including but not limited to a propagating electrical or electromagnetic signal, or non-transitory (e.g., a non-transitory computer-readable medium accessible by an application from a storage device via control or processing circuitry), including volatile and non-volatile computer memory or storage devices such as, but not limited to, hard disks, floppy disks, USB drives, DVDs, CDs, media cards, register memory, processor cache, random access memory (RAM), ultra RAM, cloud-based storage, and the like.

[0290] The described interfaces, processes, and analyses may, in some embodiments, be implemented by an application. The application may be loaded directly onto each device in any of the described systems, or may be stored in a remote server or any memory and processing circuitry accessible to each device in the system. The generation of the interface and the analysis behind it may be implemented in the receiving device, the transmitting device, or some device or processor in between.

[0291] Any use of a phrase such as "in some embodiments" or the like referring to a feature is not intended to link that feature to another feature described using the same or a similar phrase. All embodiments disclosed herein may be combined or practiced separately, as appropriate. The absence of the phrase "in some embodiments" does not infer that the feature is required. The inclusion of the phrase "in some embodiments" does not infer that the feature is not applicable to other embodiments or even all embodiments.

[0292] The systems and processes discussed herein are intended to be illustrative, not limiting. Those skilled in the art will understand that actions of the processes discussed herein may be omitted, modified, duplicated, combined, rearranged, and / or substituted, and that any additional actions may be performed without departing from the scope of the present invention. More generally, the disclosure herein is meant to provide examples and not to be limiting. Only the following claims are meant to set boundaries regarding the content covered by this disclosure. Furthermore, it should be noted that features and limitations described in any of the embodiments may be applied to any other embodiment herein, and that flowcharts or examples associated with some embodiments may be combined with any other embodiment in a suitable manner, performed in a different order, or performed in parallel. In addition, the methods, processes, and systems described herein may be performed in real time. It should also be noted that the methods, processes, and / or systems described herein may be applied to or used in accordance with other methods, processes, and / or systems.

[0293] This description is to be construed as an example only, and not to otherwise limit the scope of the embodiments herein. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the embodiments herein.

[0294] 1. A method for playing a game, either locally or remotely, via a game console acting as a server, comprising: connecting the game console to one or more remote client devices; Launching the game title on the game console; Accessing the game title's Quality of Experience (QoE) remote play definition; and determining a bitrate / encoding property mapping based on a game title QoE remote play definition; determining a queue length, an audio and / or video bitrate, a multiplexer bitrate, a target video encoding bitrate, and a target video encoding property based on the bitrate / encoding property mapping; accessing a target video encoding bit rate and target video encoding properties in a Real-Time Transport Protocol (RTP) transmitter of the game console; encoding the video with the target video encoding properties according to a target video encoding bitrate; transmitting the encoded video to an RTP receiver and / or decoder of one or more remote client devices; A method comprising: 2. A method for local and / or remote play via a game console with local and / or remote participants, wherein the game console acts as a server, and the game console is configured to form a peer-to-peer connection with one or more remote client devices; accessing a game title Quality of Experience (QoE) remote play definition in a game engine of a game console that launches the game title; In the game console, determining a bitrate / encoding property mapping based on the game title QoE remote play definition; accessing a bitrate / encoding property mapping in a rate and video encoding property controller of the game console; determining a queue length in the rate and video encoding property controller based on a priority queue of Real-Time Transport Protocol (RTP) packets in the game console; determining, in a rate and video encoding property controller, audio and / or video bit rates from an audio and / or video encoder of the game console; determining, in a rate and video encoding property controller, a multiplexer bit rate from a multiplexer of the game console; determining, in a rate and video encoding property controller, a target video encoding bit rate and a target video encoding property based on a bit rate / encoding property mapping, a queue length from a priority queue of RTP packets, an audio and / or video bit rate, and a multiplexer bit rate; accessing a target video encoding bit rate and target video encoding properties in an RTP sender of the game console; encoding the video with the target video encoding properties according to a target video encoding bitrate; transmitting the encoded video via the RTP transmitter to one or more RTP receivers and / or decoders of one or more remote client devices; A method comprising: 3. The method of any one of the preceding items, wherein the game title QoE remote play definition includes frame rate and resolution for a range of bandwidth conditions. 4. In a video encoder, encoding the encoded video; In an audio encoder, encoding the encoded audio; accessing the encoded video and the encoded audio in a multiplexer; determining, in the multiplexer, multiplexed encoded video and audio packets based on the encoded video and encoded audio; accessing the multiplexed encoded video and audio packets at an RTP sender of the game console; determining, at an RTP sender, RTP multiplexed encoded video and audio packets; buffering a priority queue of RTP packets along with RTP multiplexed encoded video and audio packets; accessing the RTP-multiplexed encoded video and audio packets in a transmission scheduler of the game console; 10. The method of any one of the preceding items, comprising: 5. In network congestion control of a game console, receiving a Real-Time Control Protocol (RTCP) packet with a source Internet Protocol (SRC IP) address and an existing, new, or removed SRC IP connection from a User Datagram Protocol (UDP) socket connected to one or more remote client devices; determining, in a transmission scheduler of the game console, a synchronization source (SSRC) identifier, a transmission timestamp (TS(TX)), a Real-time Transport Protocol sequence number (RTP(SN)), and an RTP packet size (RTP(Size)); In network congestion control, accessing SSRC, TS(TX), RTP(SN), and RTP(size); In network congestion control, processing an RTCP response from an RTP receiver on one of the remote client devices; storing RTCP packets for each SRC in a packet response data store; In network congestion control, for each SRC, determining a congestion window (CWND) and a round trip time (RTT) based on received RTCP packets; In response to determining that the packet response data store is waiting for an RTCP response, storing the SRC IP address as a worst-case client device in network congestion control; Sending the CWND and RTT from the RTCP reporting system of the network congestion control to a transmission scheduler of the game console for the worst-case client device or the last client device to respond with an RTCP packet; 10. The method of any one of the preceding items, comprising: 6. Sending the RTP-multiplexed encoded video and audio packets from the transmission scheduler to a UDP socket connected to one or more remote client devices; transmitting the RTP-multiplexed encoded video and audio packets from the UDP socket to one or more remote client devices; receiving RTCP packets from one or more remote client devices, along with the SRC IP address, on a UDP socket; 10. The method of any one of the preceding items, comprising: 7. A method for local and remote gaming via a game console acting as a server and connecting with one or more remote client devices, comprising: In network congestion control of a game console, receiving Real-Time Control Protocol (RTCP) packets and connection updates; Processing the RTCP response; and storing RTCP packets for one or more remote client devices; In network congestion control, determining a congestion window and a round-trip time for one or more remote client devices based on received RTCP packets; Responsive to determining that the packet response data store is awaiting an RTCP response, storing IP addresses of the one or more remote client devices as worst-case client devices; sending the congestion window and round trip time to the transmission scheduler for the worst case client device or the last client device to respond with an RTCP packet; A method comprising: 8. A method for local and / or remote play via a game console with local and / or remote participants, wherein the game console acts as a server, and the game console is configured to form a peer-to-peer connection with one or more remote client devices; In network congestion control for a game console, receiving Real-Time Control Protocol (RTCP) packets with a source Internet Protocol (SRC IP) address and existing, new, or removed SRC IP connections from a User Datagram Protocol (UDP) socket connected to one or more remote client devices; determining, in a transmission scheduler of the game console, a synchronization source (SSRC) identifier, a transmission timestamp (TS(TX)), a Real-time Transport Protocol sequence number (RTP(SN)), and an RTP packet size (RTP(Size)); In network congestion control, accessing SSRC, TS(TX), RTP(SN), and RTP(size); In network congestion control, processing an RTCP response from an RTP receiver on one of the remote client devices; storing RTCP packets for each SRC in a packet response data store; In network congestion control, for each SRC, determining a congestion window (CWND) and a round trip time (RTT) based on received RTCP packets; In response to determining that the packet response data store is waiting for an RTCP response, storing the SRC IP address as a worst-case client device in network congestion control; Sending the CWND and RTT from the RTCP reporting system of the network congestion control to a transmission scheduler of the game console for the worst-case client device or the last client device to respond with an RTCP packet; A method comprising: 9. Launching the game title, accessing a game title Quality of Experience (QoE) remote play definition in a game engine of the game console; In the game console, determining a bitrate / encoding property mapping based on the game title QoE remote play definition; accessing a bitrate / encoding property mapping in a rate and video encoding property controller of the game console; determining a queue length in the rate and video encoding property controller based on a priority queue of Real-Time Transport Protocol (RTP) packets in the game console; determining, in a rate and video encoding property controller, audio and / or video bit rates from an audio and / or video encoder of the game console; determining, in a rate and video encoding property controller, a multiplexer bit rate from a multiplexer of the game console; determining, in a rate and video encoding property controller, a target video encoding bit rate and a target video encoding property based on a bit rate / encoding property mapping, a queue length from a priority queue of RTP packets, an audio and / or video bit rate, and a multiplexer bit rate; accessing a target video encoding bit rate and target video encoding properties in an RTP sender of the game console; encoding the video with the target video encoding properties according to a target video encoding bitrate; transmitting the encoded video via the RTP transmitter to one or more RTP receivers and / or decoders of one or more remote client devices; 10. The method of any one of the preceding items, comprising: 10. The method of any one of the preceding items, wherein the game title QoE remote play definition includes frame rate and resolution for a range of bandwidth conditions. 11. In a video encoder, encoding the encoded video; In an audio encoder, encoding the encoded audio; accessing the encoded video and the encoded audio in a multiplexer; determining, in the multiplexer, multiplexed encoded video and audio packets based on the encoded video and encoded audio; accessing the multiplexed encoded video and audio packets at an RTP sender of the game console; determining, at an RTP sender, RTP multiplexed encoded video and audio packets; buffering a priority queue of RTP packets along with RTP multiplexed encoded video and audio packets; accessing the RTP-multiplexed encoded video and audio packets in a transmission scheduler of the game console; 10. The method of any one of the preceding items, comprising: 12. Sending the RTP-multiplexed encoded video and audio packets from the transmission scheduler to a UDP socket connected to one or more remote client devices; transmitting the RTP-multiplexed encoded video and audio packets from the UDP socket to one or more remote client devices; receiving RTCP packets from one or more remote client devices, along with the SRC IP address, on a UDP socket; 10. The method of any one of the preceding items, comprising: 13. A method for connecting a single encoder and Real-time Transport Protocol (RTP) sender to multiple client devices via a User Datagram Protocol (UDP) delivery port, comprising: Invoking and rendering video at a video source; instantiating a video encoder in live low latency mode; encoding the video at a minimum bitrate set in an encoding profile controller; encoding the audio at a defined bit rate in an audio encoder; adjusting a video encoding bit rate at the video encoder based on a change in size of a priority queue of packets as a result of the transmission of RTP packets by a transmission scheduler; A method comprising: 14. In network congestion control, accessing a new source Internet Protocol (SRC IP) address connection with a client's IP address; In network congestion control, adding a new packet response data store for an SRC IP address; 10. The method of any one of the preceding items, comprising: 15. Accessing RTP multiplexed encoded video and audio packets in a packet priority queue; sending, in a transmission scheduler, a first RTP-multiplexed encoded video and audio packet from a priority queue of packets to a UDP delivery port for transmission; 10. The method of any one of the preceding items, comprising: 16. Receiving a Real-Time Control Protocol (RTCP) packet on a UDP socket; In network congestion control, processing an RTCP response from an RTP receiver on one of the remote client devices; In response to the last remote client device responding with an RTCP packet with the expected RTP sequence number, removing the RTCP packet from the packet response data store for the SRC IP address of the RTCP packet; In response to determining that the packet response data store is waiting for an RTCP response, storing the SRC IP address as a worst-case client device in network congestion control; 10. The method of any one of the preceding items, comprising: 17. Monitoring packet priority queue size in a rate and video encoding property controller; adjusting the encoder bit rate in a rate and video encoding properties controller; After all client devices have sent RTCP responses for receiving RTP packets, in an RTCP reporting system for network congestion control, determining a congestion window (CWND) and a round trip time (RTT) for a transmission scheduler; calculating a new video encoding rate in a rate and video encoding property controller; accessing encoding properties based on the multiplexer bit rate and the audio bit rate at a rate and video encoding property controller in response to determining that a bit rate adjustment increase or decrease is required based on the queue length; determining a new target video encoding bit rate and new target video encoding properties, including resolution and frame rate, for a video encoder; 10. The method of any one of the preceding items, comprising: 18. In the multiplexer, determining multiplexed audio and video PES streams for the RTP sender; accessing the multiplexed bitrate in a rate and video encoding property controller; 10. The method of any one of the preceding items, comprising: 19. A method for inviting remote players and controlling remote player connections via a game publisher console device or cloud service client, comprising: Logging in a profile associated with the console or game owner to a console device or cloud service (CD / CS) using the certificate; sending a user authenticated response, along with the user identification, from a user account and authentication system (UAAS) of a console game provider or cloud service (CGP / CS) to the CD / CS, and a user authentication, user lookup, or user connection (UA / US / UC) response to a UA / US / UC controller of the CD / CS; In response to a user selection of an option to invite a remote player from at least one of the console device, the game service client, or the paused game via a user interface, sending a participant list request from the UA / US / UC controller of the CD / CS to the UAAS of the CGP / CS, along with the user identification; sending, from the UAAS, the user list together with the username and user identification response to the UA / US / UC controller; A method comprising: 20. In response to a user selection of one or more participants to invite from the user list, adding, at the CD / CS, the selected user identities to an invitation list for the user to invite; In response to the user's name entry and the user's selection of search, sending a search request from the UA / US / UC controller to the UAAS, along with the user name and user identification; sending, from the UAAS, the user list, along with the user name and a null response, to the UAAS, the machine user name and user identification to the UA / US / UC controller; 10. The method of any one of the preceding items, comprising: 21. In response to identifying the user identification within the UAAS, sending, from the UAAS, a user list along with the user name and user identification response to the UAAS, the machine user name and user identification to the UA / US / UC controller; In response to the user adding the retrieved user to the invite list, adding, at the CD / CS, the selected user identification to an invite list for the user to invite; 10. The method of any one of the preceding items, comprising: 22. In response to the user selecting the send invitation selection option, sending a remote session request from the remote play controller of the CD / CS to the remote player management controller of the CGP / CS along with the user identification and the remote session identifier; sending a remote session response from the remote player management controller of the CGP / CS or the remote play controller of the CD / CS to the remote play controller of the CD / CS together with the remote session identifier; 10. The method of any one of the preceding items, comprising: 23. A method according to any one of the preceding items, comprising sending an invitation from the remote play controller of the CD / CS to one or more client devices of one or more users to the remote player management controller of the CGP / CS, together with the user identification list, the requesting user name and user identification, the RTP streaming connection endpoints, and the controller connection UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and the remote session identifier. 24. A method according to any one of the preceding items, comprising sending a user invite request from a remote player management controller of the CGP / CS to each invited user together with the requester user name and user identification, the RTP streaming connection endpoint, and the controller connection UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and a remote session identifier. 25. The method of any one of the preceding items, including sending user authentication, user search, and UA / US / UC management controller user login, together with a certificate, from the CD / CS to the UAAS of the CGP / CS. 26. A method comprising: Logging a user device into a remote client application; authenticating a user via an authentication application; Requesting a list of user devices from a gaming cloud service; prompting the user device to invite one or more participants to the game; sending the invitation to one or more invited participants; In response to acceptance by one or more Invited Participants, Sending a remote session request to a game cloud service; connecting the console device with one or more invited participants; A method comprising: 27. A method for inviting remote players and controlling remote player connections via a remote play application, comprising: Logging the user device into a remote client application and transmitting a certificate associated with the user device to a User Account and Authentication System (UAAS); At the UAAS, authenticating the user device and sending a user identification of the user device to the remote play application; requesting, at the remote play controller, user devices from a console game provider or cloud service (CGP / CS), which responds with a list of device identifiers; initiating, in the CGP / CS, a remote service and exchanging connection details with the remote play controller in response to a user selection of remote player management; prompting the user device to invite participants by adding the user identification to an invitation list; In response to the user selecting a name to retrieve, sending a request to the UAAS, which responds with a list of users; adding the user identification to the invitation list in response to a user selection to add the retrieved user to the invitation list; In response to a user selection to send an invitation, sending an invitation request from the remote play controller to the CGP / CS, which transmits the invitation request to each invited user; connecting a remote play controller to the console device and initiating interaction through a remote display; sending a remote session request from the remote play controller to the CGP / CS, which stores the remote connection IP address; sending, from the CGP / CS, a user invitation request to each invited user; A method comprising: 28. A method comprising: logging a remote user device into a remote client application; receiving an acceptance of the invitation from the remote user device; transmitting the user credentials to an authentication system of the game provider; connecting the remote user device to the controller port in response to an approval response from the remote user device; If the connection is unsuccessful, sending a message to the remote user device; If the connection is successful, connecting a remote user device to a streaming protocol of a console device; decoding and rendering the content for display at the remote user device; transmitting the controller input to a remote user device; receiving tactile feedback from a remote user device; A method comprising: 29. A method for a remote player to accept a remote play invitation, comprising: Logging a user device into a remote client application; Transmitting the user credentials to the User Account and Authentication System (UAAS) of the console game provider or cloud service (CGP / CS); in response to the remote user selecting acceptance of the remote play invitation, logging the remote user into the CGP / CS; Sending an invitation acceptance response from the remote play controller to the console device or cloud service (CD / CS), which stores the IP address of the remote user; In a remote play application, attempting to connect to a controller port; sending a message to the remote user's device in response to an unsuccessful connection to the controller port; In response to a successful connection to the controller port, sending controller input from a remote application to a console device; In a remote application, receiving tactile feedback from a console device; connecting, at the remote play controller, to a Real-time Transport Protocol (RTP) streaming User Datagram Protocol (UDP) connection endpoint of the console device; Decoding and rendering the content for display; and A method comprising: 30. A method for initial controller connection for local and / or remote users, comprising: Locally, power on the console and controllers, assigning the locally powered controller as controller 1; restricting primary control to controller 1 at the console; determining whether the additional controller is powered on; In response to the additional controller being powered on, in response to determining that all controller slots of the console are in use, setting the console to a state in which it will not accept the connection of additional controllers; In response to determining that all controller slots of the console are not in use, assigning an additional controller to the next available controller spot; In response to the additional controller not being powered on, determining whether a remote user is joined to a controller User Datagram Protocol (UDP) port; repeating the step of determining whether additional controllers have been powered on in response to determining that the remote user has not joined the controller UDP port; determining whether all controller slots are in use in response to determining that a remote user has joined the controller UDP port; A method comprising: 31. In response to determining that all controller slots are in use, determining whether the remote user is joined to the console device or cloud service (CD / CS) as a service account holder; in response to determining that the remote user has not joined the CD / CS as a service account holder, at the console, refusing the connection to the controller UDP port and displaying one or more messages to the remote user, prompting the remote user to request controller control or wait until a player relinquishes the controller; in response to determining that the remote user has joined the CD / CS as a service account holder, unlocking an option in the remote application to forcibly disconnect a connected local or remote user from a controller slot, and unlocking an option in the remote application to forcibly grant console control to controller 1; 10. The method of any one of the preceding items, comprising: 32. In response to determining that all controller slots are not in use, determining whether the remote user is joined to the console device or cloud service (CD / CS) as a service account holder; in response to determining that the remote user has not joined the CD / CS as a service account holder, mapping the console UDP controller input to a next available controller slot; In response to determining that the remote user has joined the CD / CS as a service account holder, determining whether other remote or local game controllers are connected; In response to determining that another remote or local game controller is connected, unlocking an option in the remote application to forcibly grant console control to controller 1; in response to determining that no other remote or local game controllers are connected, sending from the remote application a forced controller 1 mapping and from the game management application of the CD / CS a controller number input mapping request to a local or remote UDP port mapping controller input-output mapper, wherein the remote UDP port is assigned to the UDP connection of the controller associated with the owner, and the mapping is made to the slot of controller 1; 10. The method of any one of the preceding items, comprising: 33. A method for transferring a game controller for local and / or remote play via a game console, comprising: determining whether all controller slots of the console device are filled; verifying the connection of one or more remote users; determining whether the local player has selected a controller transfer option; exchanging user lists between the console device and the game provider's cloud service according to the type of game currently in progress; displaying a list of remote users for selection by the primary user; exchanging a series of requests and responses in response to a selection by the primary user and delegating control to the selected user; A method comprising: 34. A method for transferring a controller for local and / or remote play via a game console with local and / or remote participants, wherein the game console acts as a server, and the game console is configured to form a peer-to-peer connection with one or more remote client devices; Determining whether all controller slots are filled on the console device or cloud service (CD / CS); determining whether one or more remote users are connected to the remote play game console session; Verifying whether one or more remote users connected to the controller slot are stored for the remote player management controller of the console game provider or cloud service (CGP / CS) for the remote session identifier; determining whether the local player using the controller has paused the game and selected a "controller handover" option; Determining the type of game in progress; Based on the type of game in progress, sending the user list in a remote session request from the remote player controller of the CD / CS to the remote player management controller of the CGP / CS; receiving a user list remote session response from the CGP / CS remote player management controller; displaying, at the console, a list of remote users for user selection; receiving a user selection to "transfer control" using a local controller of the user device; sending a local controller handover request from the controller input-output mapper to the CD / CS; sending a "controller handover" request from the CD / CS remote play controller to the CGP / CS remote player management controller; sending a "controller handover" request from the remote player management controller of the PCGPCS to the remote play controller of the remote player application; connecting a remote player management controller of a remote playing application to a controller UDP connection endpoint; sending a controller connected response from the remote play controller of the remote play application to the remote player management controller of the CGP / CS; A method comprising: 35. A method for a console owner device to take over control of or disconnect a remote user device when all controller ports are full, comprising: Determining the login status of the console owner device; sending a forced controller mapping request; If all ports are occupied, providing the console owner device with a list of connected remote user devices and prompting the console owner device to select a connected remote user device to disconnect; notifying the remote user device to be disconnected; Disconnecting the remote user device; A method comprising: 36. A method for a console owner device to request main control for a user device or force a disconnect when a controller port is full, the method being for local and / or remote play via a game console with local and / or remote participants, the game console acting as a server, the game console configured to form a peer-to-peer connection with one or more remote client devices; determining whether the console owner device is logged into a remote application; determining whether a slot for Controller 1 on the console device or cloud service (CD / CS) is occupied by a local or remote user in response to the console owner selecting a forcible grant of control to Controller 1; sending a controller 1 forced mapping request from a remote application to a remote player management controller of a CD / CS; sending a controller 1 forced mapping request from a remote player management controller of a console game provider or cloud service (CGP / CS) to a remote player controller of the CGP / CS; sending a controller 1 forced mapping request from a remote play controller of a remote player management controller of the CGP / CS to a controller input-output mapper; In the controller input-output mapper, disconnecting the connected remote user from the mapped controller User Datagram Protocol (UDP) socket; connecting to the console owner device's mapped controller UDP socket; In response to determining that all UDP sockets are occupied, displaying a message to the remote user to be disconnected that the console owner device has requested control and that the user device is not in spectator mode; in response to determining that all UDP sockets are unoccupied, remapping, at the controller input-output mapper, the connected controller UDP sockets of the remote users to unused controller connections; in a controller input-output mapper, mapping a controller UDP socket connection to which a console owner device is connected to controller input 1; in response to determining that all UDP sockets are occupied, providing to the console owner device a list of remote users connected to the controller; receiving a console owner selection of the user device to remove control; sending a forced controller mapping request for the remote user from a remote player manager of a remote play controller of the CGP / CS to a controller input-output mapper; sending a forced controller mapping response for the remote user from the controller input-output mapper to a remote player manager of the remote play controller of the CGP / CS; disconnecting, in a controller input-output mapper, a connected remote user from a controller UDP socket mapped to a controller input for the remote user; connecting a controller of the console owner device to the mapped controller UDP socket; A method comprising: 37. A method for enabling a console owner device to request main control of or force disconnection of a remote user device, comprising: determining whether a console owner device is logged into a remote application and whether a slot for a controller on the console device or cloud service (CD / CS) is occupied; If the console owner device chooses to forcibly grant control, sending a forcible controller mapping request via the CD / CS, disconnecting the connected remote user device from the mapped controller User Datagram Protocol (UDP) socket and connecting it to the controller of the console owner device; If all UDP sockets are occupied, displaying a message on the remote user device to be disconnected and providing the console owner device with a list of connected remote user devices; prompting the console owner device to select a remote user device for removing control; exchanging, via the CD / CS, a forced controller mapping request and response for the remote user device, disconnecting and connecting the remote user device to the controller of the console owner device, and mapping the controller of the console owner device to the controller UDP socket; A method comprising: 38. A method for low-bandwidth adaptive bitrate streaming (ABR) for local and / or remote play via a game console with local and / or remote participants, wherein the game console acts as a server, and the game console is configured to form a peer-to-peer connection with one or more remote client devices; In response to determining that the target video encoding bit rate is less than a minimum value in a bit rate / encoding property mapping, sending a low bandwidth notification to a real-time control protocol (RTCP) reporting system; sending, from the RTCP reporting system, a client device disconnection request to a remote play controller of the game console; sending, from the remote play controller, a force disconnect request to the remote play client application in response to determining that an ABR delivery controller session has not yet been initiated for the remote session identifier; forcing a disconnection from the address and port received by the remote play controller; In response to determining that the reported bitrate is sufficiently high to reconnect to the peer-to-peer Real-time Transport Protocol (RTP) User Datagram Protocol (UDP) streaming session, sending a force disconnect request from the remote play controller to a remote player management controller of a console game provider or cloud service (CGP / CS) along with the remote session identifier, the IP address, and the minimum bitrate requirement request; sending a session reconnection request from the remote play controller to the remote play client application; sending, by the remote player management controller, an ABR session termination request to the ABR delivery controller of the CGP / CS in response to determining that no low-bandwidth clients are present; sending, by the remote player management controller, an ABR session start request to the ABR delivery controller in response to determining that a low-bandwidth client is present; Initiating an ABR session instance in an ABR session controller with an RTP transmit receiver, a demultiplexer, an ABR transcoder, and an ABR segmenter of the ABR delivery controller; establishing an RTP connection via an RTP transmitter-receiver; In an RTP transmission receiver, receiving an RTP-multiplexed audiovisual packet stream and demultiplexing the RTP-multiplexed audiovisual packet stream into video and audio packetized elementary stream (PES) packet streams; sending the video and audio PES streams to an ABR transcoder and an ABR segmenter; generating a live manifest in the ABR segmenter and writing the segments to an ABR audiovisual segment and manifest store; sending, from the remote play controller, an ABR live manifest URL notification to the remote play client application; at the remote play client application, starting an instance of an ABR video player of the remote play client application and sending the ABR live manifest to the ABR video player; downloading and playing ABR segments and receiving live manifest updates in an ABR video player; sending the calculated bit rate from the ABR video player to a remote play controller of a remote player management controller of the CGP / CS; A method comprising: 39. A method comprising: Providing local and remote play involving local and remote participants via a game console, the game console acting as a server; prompting a user to select an option to invite one or more remote players to the game session; forming a peer-to-peer connection between the game console and one or more remote client devices in response to a user selection of an option to invite one or more remote players to the game session; A method comprising: 40. The method of any one of the preceding items, wherein prompting a user for selection of an option to invite one or more remote players to the game session is provided after user selection of a pause command during the game session. 41. The method of any one of the preceding paragraphs, wherein prompting the user to select an option to invite one or more remote players to the game session is provided with a main game screen. 42. The method of any one of the preceding items, wherein prompting the user to select an option to invite one or more remote players to the game session is provided through a main console interface. 43. Prompting a user for selection of an option to transfer control to a remote user during a game session; in response to a user selection of an option to transfer control to a remote user during a game session; Determining whether all controller slots are filled on the console device or cloud service (CD / CS); determining whether one or more remote users are connected to the remote play game console session; Verifying whether one or more remote users connected to the controller slot are stored for the remote player management controller of the console game provider or cloud service (CGP / CS) for the remote session identifier; determining whether the local player using the controller has paused the game and selected a "controller handover" option; Determining the type of game in progress; Based on the type of game in progress, sending the user list in a remote session request from the remote player controller of the CD / CS to the remote player management controller of the CGP / CS; receiving a user list remote session response from the CGP / CS remote player management controller; displaying, at the console, a list of remote users for user selection; receiving a user selection to "transfer control" using a local controller of the user device; sending a local controller handover request from the controller input-output mapper to the CD / CS; sending a "controller handover" request from the CD / CS remote play controller to the CGP / CS remote player management controller; sending a "controller handover" request from the remote player management controller of the PCGPCS to the remote play controller of the remote player application; connecting a remote player management controller of a remote playing application to a controller UDP connection endpoint; sending a controller connected response from the remote play controller of the remote play application to the remote player management controller of the CGP / CS; 10. The method of any one of the preceding items, comprising: 44. The method of any one of the preceding items, comprising, in response to detecting a low bandwidth connection associated with the remote player, suspending the game play mode of the remote player and placing the remote player in a spectator mode. 45. The method of any one of the preceding items, including, in a spectator mode, modifying the appearance of displayed game elements associated with a remote player. 46. The method of any one of the preceding items, comprising, in response to detecting a low bandwidth connection associated with the remote player, replacing game play by the remote player with artificial intelligence trained on the playing style of the remote player. 47. The method of any one of the preceding items, comprising, in response to detecting a sufficient bandwidth connection associated with the remote player, terminating the spectator mode of the remote player and returning the remote player to a game play mode. 48. A method for prompting the addition of one or more remote client devices to a game console controlled by a local operator of the game console, comprising: providing a selectable option to invite at least one of one or more remote client devices to connect to the game console for display on a local display connected to the game console; forming a peer-to-peer connection between the game console and at least one of the one or more remote client devices in response to selection of a selectable option by at least one of the one or more remote client devices; A method comprising: 49. Providing selectable options for display in response to a local operator pausing a game; providing a selectable option to acquire a spot in the multiplayer game for display in response to launching the multiplayer game on the game console; Providing an option for a local operator of a game console to enable one or more remote client devices to view the main console of the game console as if the one or more remote client devices were themselves playing with the local operator; or providing a selectable option whereby, in response to one or more remote client devices pausing the game, the remote client device transfers the controller to a local operator or to another of the one or more remote client devices for display on the one or more remote client devices; 10. The method of any one of the preceding items, comprising at least one of: 50. A method for remote play for a game running on a game console, comprising: forming a peer-to-peer connection between the game console and one or more remote client devices; in response to a loss of a peer-to-peer connection between the game console and one or more remote client devices; playing the game in spectator mode; highlighting icons corresponding to players controlled at one or more remote client devices with which a peer-to-peer connection has been lost; controlling the actions of the player when the peer-to-peer connection is lost by an artificial intelligence trained on the player's gameplay; in response to restoration of a peer-to-peer connection between the game console and one or more remote client devices; playing a game in a gameplay mode; A method comprising: 51. A system for playing games, either locally or remotely, via a game console acting as a server, comprising: memory and communication ports; control circuitry communicatively coupled to the memory and the communication port, the control circuitry executing instructions; connecting the game console to one or more remote client devices; Launching the game title on the game console; Accessing the game title's Quality of Experience (QoE) remote play definition; and determining a bitrate / encoding property mapping based on a game title QoE remote play definition; determining a queue length, an audio and / or video bitrate, a multiplexer bitrate, a target video encoding bitrate, and a target video encoding property based on the bitrate / encoding property mapping; accessing a target video encoding bit rate and target video encoding properties in a Real-Time Transport Protocol (RTP) transmitter of the game console; encoding the video with the target video encoding properties according to a target video encoding bitrate; transmitting the encoded video to an RTP receiver and / or decoder of one or more remote client devices; and control circuitry configured to: A system comprising: 52. A system for local and / or remote play via a game console with local and / or remote participants, the game console acting as a server, the game console configured to form a peer-to-peer connection with one or more remote client devices; memory and communication ports; control circuitry communicatively coupled to the memory and the communication port, the control circuitry executing instructions; accessing a game title Quality of Experience (QoE) remote play definition in a game engine of a game console that launches the game title; In the game console, determining a bitrate / encoding property mapping based on the game title QoE remote play definition; accessing a bitrate / encoding property mapping in a rate and video encoding property controller of the game console; determining a queue length in the rate and video encoding property controller based on a priority queue of Real-Time Transport Protocol (RTP) packets in the game console; determining, in a rate and video encoding property controller, audio and / or video bit rates from an audio and / or video encoder of the game console; determining, in a rate and video encoding property controller, a multiplexer bit rate from a multiplexer of the game console; determining, in a rate and video encoding property controller, a target video encoding bit rate and a target video encoding property based on a bit rate / encoding property mapping, a queue length from a priority queue of RTP packets, an audio and / or video bit rate, and a multiplexer bit rate; accessing a target video encoding bit rate and target video encoding properties in an RTP sender of the game console; encoding the video with the target video encoding properties according to a target video encoding bitrate; transmitting the encoded video via the RTP transmitter to one or more RTP receivers and / or decoders of one or more remote client devices; and control circuitry configured to: A system comprising: 53. The system of any one of the preceding items, wherein the game title QoE remote play definition includes frame rate and resolution for a range of bandwidth conditions. 54. The control circuitry executes instructions; encoding the encoded video in a video encoder; In an audio encoder, encoding the encoded audio; accessing the encoded video and the encoded audio in a multiplexer; determining, in the multiplexer, multiplexed encoded video and audio packets based on the encoded video and encoded audio; accessing the multiplexed encoded video and audio packets at an RTP sender of the game console; determining, at an RTP sender, RTP multiplexed encoded video and audio packets; buffering a priority queue of RTP packets along with RTP multiplexed encoded video and audio packets; Accessing RTP multiplexed encoded video and audio packets in a transmission scheduler of a game console 10. The system of any one of the preceding items, configured to: 55. The control circuitry executes instructions; In network congestion control for a game console, receiving Real-Time Control Protocol (RTCP) packets with a source Internet Protocol (SRC IP) address and existing, new, or removed SRC IP connections from a User Datagram Protocol (UDP) socket connected to one or more remote client devices; determining, in a transmission scheduler of the game console, a synchronization source (SSRC) identifier, a transmission timestamp (TS(TX)), a Real-time Transport Protocol sequence number (RTP(SN)), and an RTP packet size (RTP(Size)); In network congestion control, accessing SSRC, TS(TX), RTP(SN), and RTP(size); In network congestion control, processing an RTCP response from an RTP receiver on one of the remote client devices; storing RTCP packets for each SRC in a packet response data store; In network congestion control, for each SRC, determining a congestion window (CWND) and a round trip time (RTT) based on received RTCP packets; In response to determining that the packet response data store is waiting for an RTCP response, storing the SRC IP address as a worst-case client device in network congestion control; The RTCP reporting system of the network congestion control sends the CWND and RTT to the transmission scheduler of the game console so that the worst-case client device or the last client device responds with an RTCP packet. 10. The system of any one of the preceding items, configured to: 56. The control circuitry executes instructions; sending the RTP-multiplexed encoded video and audio packets from the transmission scheduler to a UDP socket connected to one or more remote client devices; transmitting the RTP-multiplexed encoded video and audio packets from the UDP socket to one or more remote client devices; receiving RTCP packets from one or more remote client devices, along with the SRC IP address, on a UDP socket; 10. The system of any one of the preceding items, configured to: 57. A system for local and remote gaming via a game console acting as a server and connecting with one or more remote client devices, comprising: memory and communication ports; control circuitry communicatively coupled to the memory and the communication port, the control circuitry executing instructions; In network congestion control of a game console, receiving Real-Time Control Protocol (RTCP) packets and connection updates; Processing the RTCP response; and storing RTCP packets for one or more remote client devices; In network congestion control, determining a congestion window and a round-trip time for one or more remote client devices based on received RTCP packets; Responsive to determining that the packet response data store is awaiting an RTCP response, storing IP addresses of the one or more remote client devices as worst-case client devices; The worst-case client device or the last client device to respond with an RTCP packet sends the congestion window and round-trip time to the transmission scheduler. and control circuitry configured to: A system comprising: 58. A system for local and / or remote play via a game console with local and / or remote participants, the game console acting as a server, the game console configured to form a peer-to-peer connection with one or more remote client devices; memory and communication ports; control circuitry communicatively coupled to the memory and the communication port, the control circuitry executing instructions; In network congestion control for a game console, receiving Real-Time Control Protocol (RTCP) packets with a source Internet Protocol (SRC IP) address and existing, new, or removed SRC IP connections from a User Datagram Protocol (UDP) socket connected to one or more remote client devices; determining, in a transmission scheduler of the game console, a synchronization source (SSRC) identifier, a transmission timestamp (TS(TX)), a Real-time Transport Protocol sequence number (RTP(SN)), and an RTP packet size (RTP(Size)); In network congestion control, accessing SSRC, TS(TX), RTP(SN), and RTP(size); In network congestion control, processing an RTCP response from an RTP receiver on one of the remote client devices; storing RTCP packets for each SRC in a packet response data store; In network congestion control, for each SRC, determining a congestion window (CWND) and a round trip time (RTT) based on received RTCP packets; In response to determining that the packet response data store is waiting for an RTCP response, storing the SRC IP address as a worst-case client device in network congestion control; The RTCP reporting system of the network congestion control sends the CWND and RTT to the transmission scheduler of the game console so that the worst-case client device or the last client device responds with an RTCP packet. and control circuitry configured to: A system comprising: 59. The control circuitry executes instructions; accessing a game title Quality of Experience (QoE) remote play definition in a game engine of a game console that launches the game title; In the game console, determining a bitrate / encoding property mapping based on the game title QoE remote play definition; accessing a bitrate / encoding property mapping in a rate and video encoding property controller of the game console; determining a queue length in the rate and video encoding property controller based on a priority queue of Real-Time Transport Protocol (RTP) packets in the game console; determining, in a rate and video encoding property controller, audio and / or video bit rates from an audio and / or video encoder of the game console; determining, in a rate and video encoding property controller, a multiplexer bit rate from a multiplexer of the game console; determining, in a rate and video encoding property controller, a target video encoding bit rate and a target video encoding property based on a bit rate / encoding property mapping, a queue length from a priority queue of RTP packets, an audio and / or video bit rate, and a multiplexer bit rate; accessing a target video encoding bit rate and target video encoding properties in an RTP sender of the game console; encoding the video with the target video encoding properties according to a target video encoding bitrate; transmitting the encoded video via the RTP transmitter to one or more RTP receivers and / or decoders of one or more remote client devices; 10. The system of any one of the preceding items, configured to: 60. The system of any one of the preceding items, wherein the game title QoE remote play definition includes frame rate and resolution for a range of bandwidth conditions. 61. The control circuitry executes instructions; encoding the encoded video in a video encoder; In an audio encoder, encoding the encoded audio; accessing the encoded video and the encoded audio in a multiplexer; determining, in the multiplexer, multiplexed encoded video and audio packets based on the encoded video and encoded audio; accessing the multiplexed encoded video and audio packets at an RTP sender of the game console; determining, at an RTP sender, RTP multiplexed encoded video and audio packets; buffering a priority queue of RTP packets along with RTP multiplexed encoded video and audio packets; Accessing RTP multiplexed encoded video and audio packets in a transmission scheduler of a game console 10. The system of any one of the preceding items, configured to: 62. The control circuitry executes instructions; sending the RTP-multiplexed encoded video and audio packets from the transmission scheduler to a UDP socket connected to one or more remote client devices; transmitting the RTP-multiplexed encoded video and audio packets from the UDP socket to one or more remote client devices; receiving RTCP packets from one or more remote client devices, along with the SRC IP address, on a UDP socket; 10. The system of any one of the preceding items, configured to: 63. A system for connecting a single encoder and Real-time Transport Protocol (RTP) sender to multiple client devices via a User Datagram Protocol (UDP) delivery port, comprising: memory and communication ports; control circuitry communicatively coupled to the memory and the communication port, the control circuitry executing instructions; Invoking and rendering video at a video source; instantiating a video encoder in live low latency mode; encoding the video at a minimum bitrate set in an encoding profile controller; encoding the audio at a defined bit rate in an audio encoder; adjusting a video encoding bit rate based on a change in size of a priority queue of packets as a result of a transmission scheduler transmitting RTP packets in the video encoder; and control circuitry configured to: A system comprising: 64. The control circuitry executes instructions; In network congestion control, accessing a new source Internet Protocol (SRC IP) address connection with a client's IP address; In network congestion control, adding a new packet response data store for the SRC IP address; 10. The system of any one of the preceding items, configured to: 65. The control circuitry executes instructions; accessing the RTP multiplexed encoded video and audio packets in a packet priority queue; sending, in the transmission scheduler, the first RTP-multiplexed encoded video and audio packets from the packet priority queue to a UDP delivery port for transmission; 10. The system of any one of the preceding items, configured to: 66. The control circuitry executes instructions; receiving a Real-Time Control Protocol (RTCP) packet on a UDP socket; In network congestion control, processing an RTCP response from an RTP receiver on one of the remote client devices; In response to the last remote client device responding with an RTCP packet with the expected RTP sequence number, removing the RTCP packet from the packet response data store for the SRC IP address of the RTCP packet; and storing the SRC IP address as a worst-case client device in network congestion control in response to determining that the packet response data store is waiting for an RTCP response. 10. The system of any one of the preceding items, configured to: 67. The control circuitry executes instructions; monitoring a size of a priority queue of packets in a rate and video encoding property controller; adjusting the encoder bit rate in a rate and video encoding properties controller; After all client devices have sent RTCP responses for receiving RTP packets, in an RTCP reporting system for network congestion control, determining a congestion window (CWND) and a round trip time (RTT) for a transmission scheduler; calculating a new video encoding rate in a rate and video encoding property controller; accessing encoding properties based on the multiplexer bit rate and the audio bit rate at a rate and video encoding property controller in response to determining that a bit rate adjustment increase or decrease is required based on the queue length; determining a new target video encoding bit rate and new target video encoding properties, including resolution and frame rate, for the video encoder; 10. The system of any one of the preceding items, configured to: 68. The control circuitry executes instructions; determining, in the multiplexer, multiplexed audio and video PES streams for the RTP sender; Accessing multiplexed bit rates in the rate and video encoding property controller 10. The system of any one of the preceding items, configured to: 69. A system for inviting remote players and controlling remote player connections via a game publisher's console device or cloud service client, comprising: memory and communication ports; control circuitry communicatively coupled to the memory and the communication port, the control circuitry executing instructions; Logging in a profile associated with the console or game owner to a console device or cloud service (CD / CS) using the certificate; sending a user authenticated response, along with the user identification, from a user account and authentication system (UAAS) of a console game provider or cloud service (CGP / CS) to the CD / CS, and a user authentication, user lookup, or user connection (UA / US / UC) response to a UA / US / UC controller of the CD / CS; In response to a user selection of an option to invite a remote player from at least one of the console device, the game service client, or the paused game via a user interface, sending a participant list request from the UA / US / UC controller of the CD / CS to the UAAS of the CGP / CS, along with the user identification; The UAAS sends the user list to the UA / US / UC controller along with the user name and user identification response. and control circuitry configured to: A system comprising: 70. The control circuitry executes instructions; in response to a user selection of one or more participants to invite from the user list, adding, at the CD / CS, the selected user identities to an invitation list for the user to invite; In response to the user's name entry and the user's selection of search, sending a search request from the UA / US / UC controller to the UAAS, along with the user name and user identification; Sending the user list from the UAAS, along with the user name and an empty response, to the UAAS, the machine user name and user identification to the UA / US / UC controller; 10. The system of any one of the preceding items, configured to: 71. The control circuitry executes instructions; In response to identifying the user identification within the UAAS, transmitting, from the UAAS, a user list along with the user name and user identification response to the UAAS, the machine user name and user identification to the UA / US / UC controller; adding, at the CD / CS, the selected user identification to an invitation list for the user to invite in response to the user adding the user to the invite list of the retrieved user; 10. The system of any one of the preceding items, configured to: 72. The control circuitry executes instructions; In response to the user selecting the send invitation option, sending a remote session request from the remote play controller of the CD / CS to the remote player management controller of the CGP / CS along with the user identification and the remote session identifier; sending a remote session response from the remote player management controller of the CGP / CS or the remote play controller of the CD / CS to the remote play controller of the CD / CS together with the remote session identifier; 10. The system of any one of the preceding items, configured to: 73. The control circuitry executes instructions; 10. The system of claim 1, wherein the system is configured to send an invitation from the remote play controller of the CD / CS to one or more client devices of one or more users to the remote player management controller of the CGP / CS, along with the user identification list, the requesting user name and user identification, the RTP streaming connection endpoints, and the controller connection UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and the remote session identifier. 74. The control circuitry executes instructions; 10. The system of claim 1, wherein the system is configured to send a user invite request from the CGP / CS remote player management controller to each invited user, along with the requester username and user identification, the RTP streaming connection endpoint, and the controller connection UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and the remote session identifier. 75. The control circuitry executes instructions; 10. The system of claim 1, configured to send user authentication, user search, and UA / US / UC management controller user login, along with the certificate, from the CD / CS to the UAAS of the CGP / CS. 76. A system comprising: memory and communication ports; control circuitry communicatively coupled to the memory and the communication port, the control circuitry executing instructions; Logging a user device into a remote client application; authenticating the user device via an authentication application; Requesting a list of user devices from a gaming cloud service; prompting the user device to invite one or more participants to the game; sending the invitation to one or more invited participants; In response to acceptance by one or more Invited Participants, Sending a remote session request to a game cloud service; Connecting a console device to one or more invited participants and control circuitry configured to: A system comprising: 77. A system for inviting remote players and controlling remote player connections via a remote play application, comprising: memory and communication ports; control circuitry communicatively coupled to the memory and the communication port, the control circuitry executing instructions; Logging the user device into a remote client application and transmitting a certificate associated with the user device to a User Account and Authentication System (UAAS); At the UAAS, authenticating the user device and sending a user identification of the user device to the remote play application; requesting, at the remote play controller, user devices from a console game provider or cloud service (CGP / CS), which responds with a list of device identifiers; initiating, in the CGP / CS, a remote service and exchanging connection details with the remote play controller in response to a user selection of remote player management; prompting the user device to invite participants by adding the user identification to an invitation list; In response to the user selecting a name to retrieve, sending a request to the UAAS, which responds with a list of users; adding the user identification to the invitation list in response to a user selection to add the retrieved user to the invitation list; In response to a user selection to send an invitation, sending an invitation request from the remote play controller to the CGP / CS, which transmits the invitation request to each invited user; connecting a remote play controller to the console device and initiating interaction through a remote display; sending a remote session request from the remote play controller to the CGP / CS, which stores the remote connection IP address; The CGP / CS sends a user invitation request to each invited user. and control circuitry configured to: A system comprising: 78. A system comprising: memory an...

Claims

1. 1. A method for inviting remote players and controlling remote player connections via a game publisher console device or cloud service client, the method comprising: Logging a profile associated with the console or game owner into a console device or cloud service (CD / CS) using the certificate; Sending a user authenticated response, along with a user identification, from a user account and authentication system (UAAS) of a console game provider or cloud service (CGP / CS) to the CD / CS, and sending a user authentication, user lookup, or user connection (UA / US / UC) response to a UA / US / UC controller of the CD / CS; In response to a user selection, via a user interface, of an option to invite a remote player from at least one of the console device, the game service client, or the paused game, sending a participant list request from a UA / US / UC controller of the CD / CS to a UAAS of the CGP / CS, along with the user identification; sending a user list from the UAAS to the UA / US / UC controller together with a user name and a user identification response; A method comprising:

2. in response to a user selection of one or more participants to invite from the user list, adding, at the CD / CS, the selected user identities to an invitation list for the user to invite; In response to the user's name entry and the user's selection of search, sending a search request from the UA / US / UC controller to the UAAS along with the user's name and user identification; sending the user list, along with user names and a null response, to the UAAS, and sending machine user names and user identifications to the UA / US / UC controller; The method of claim 1 , comprising:

3. In response to identifying the user identity within the UAAS, transmitting from the UAAS the user list along with a user name and user identity response to the UAAS and transmitting the machine user name and the user identity to the UA / US / UC controller; adding, at the CD / CS, the selected user identity to an invitation list for the inviting user in response to the user adding the retrieved user to the invitation list; The method of claim 2 , comprising:

4. In response to a user selection of a send invitation selection option, sending a remote session request from a remote play controller of said CD / CS to a remote player management controller of said CGP / CS together with said user identification and a remote session identifier; sending a remote session response, together with the remote session identifier, from the remote player management controller of the CGP / CS or the remote play controller of the CD / CS to the remote play controller of the CD / CS; The method of claim 3, comprising:

5. 5. The method of claim 4, comprising sending an invitation from the remote play controller of the CGP / CS to one or more client devices of one or more users to a remote player management controller of the CGP / CS, along with the user identification list, requester username and user identification, RTP streaming connection endpoints, and controller connection UDP 1 connection endpoints, UDP 2 connection endpoints, UDP 3 connection endpoints, UDP 4 connection endpoints, and a remote session identifier.

6. 6. The method of claim 5, comprising sending a user invite request from the CGP / CS remote player management controller to each invited user, together with the requester username and user identification, RTP streaming connection endpoint, and controller connection UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and remote session identifier.

7. 7. The method of claim 6, comprising sending the user authentication, user search, and UA / US / UC management controller user login, along with a certificate, from the CD / CS to a UAAS of the CGP / CS.

8. 1. A method for inviting remote players and controlling remote player connections via a remote play application, the method comprising: Logging a user device into a remote client application and transmitting a credential associated with the user device to a User Account and Authentication System (UAAS); authenticating the user device at the UAAS and sending a user identification of the user device to the remote play application; requesting user devices from a console game provider or cloud service (CGP / CS) at the remote play controller, which responds with a list of device identifiers; initiating, in said CGP / CS, a remote service and exchanging connection details with said remote player controller in response to a user selection of remote player management; prompting the user device to invite participants by adding a user identification to an invitation list; In response to a user selection to retrieve a name, sending a request to the UAAS, which responds with a user list; adding the user identification to the invitation list in response to a user selection of adding the retrieved user to the invitation list; In response to a user selection to send an invitation, sending an invitation request from the remote play controller to the CGP / CS, which transmits the invitation request to each invited user; connecting the remote play controller to a console device and initiating interaction through a remote display; sending a remote session request from the remote play controller to the CGP / CS, which stores a remote connection IP address; sending a user invite request from the CGP / CS to each invited user; A method comprising:

9. 1. A system for inviting remote players and controlling remote player connections via a game publisher's console device or cloud service client, the system comprising: memory and communication ports; control circuitry communicatively coupled to the memory and the communication port, the control circuitry executing instructions; Logging a profile associated with the console or game owner into a console device or cloud service (CD / CS) using the certificate; Sending a user authenticated response, along with a user identification, from a user account and authentication system (UAAS) of a console game provider or cloud service (CGP / CS) to the CD / CS, and sending a user authentication, user lookup, or user connection (UA / US / UC) response to a UA / US / UC controller of the CD / CS; In response to a user selection of an option to invite a remote player from at least one of the console device, the game service client, or the paused game via a user interface, sending a participant list request from a UA / US / UC controller of the CD / CS to a UAAS of the CGP / CS, along with the user identification; sending a user list from the UAAS to the UA / US / UC controller together with a user name and a user identification response; control circuitry configured to A system comprising:

10. the control circuitry executes the instructions; in response to a user selection of one or more participants to invite from the user list, adding, at the CD / CS, the selected user identities to an invitation list for the user to invite; In response to the user's name entry and the user's selection of search, sending a search request from the UA / US / UC controller to the UAAS along with the user's name and user identification; sending the user list, along with user names and a null response, to the UAAS, and sending machine user names and user identifications to the UA / US / UC controller; The system of claim 9 configured to:

11. the control circuitry executes the instructions; In response to identifying the user identity within the UAAS, transmitting from the UAAS the user list along with a user name and user identity response to the UAAS and transmitting the machine user name and the user identity to the UA / US / UC controller; adding, at the CD / CS, the selected user identity to an invitation list for the inviting user in response to the user adding the retrieved user to the invitation list; The system of claim 10 configured to:

12. the control circuitry executes the instructions; In response to a user selection of a send invitation selection option, sending a remote session request from a remote play controller of said CD / CS to a remote player management controller of said CGP / CS together with said user identification and a remote session identifier; sending a remote session response, together with the remote session identifier, from the remote player management controller of the CGP / CS or the remote play controller of the CD / CS to the remote play controller of the CD / CS; The system of claim 11 configured to:

13. the control circuitry executes the instructions; 13. The system of claim 12, configured to send an invitation from the CD / CS remote play controller to the CGP / CS remote player management controller to one or more client devices of one or more users, along with the user identification list, requester username and user identification, RTP streaming connection endpoints, and controller connection UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and remote session identifier.

14. the control circuitry executes the instructions; 14. The system of claim 13, configured to send a user invite request from the CGP / CS remote player management controller to each invited user, along with the requester username and user identification, RTP streaming connection endpoint, and controller connection UDP 1 connection endpoint, UDP 2 connection endpoint, UDP 3 connection endpoint, UDP 4 connection endpoint, and remote session identifier.

15. the control circuitry executes instructions; The system of claim 14 , configured to send the user authentication, user search, and UA / US / UC management controller user login, along with a certificate, from the CD / CS to the UAAS of the CGP / CS.

16. 1. A system for inviting remote players and controlling remote player connections via a remote play application, the system comprising: memory and communication ports; control circuitry communicatively coupled to the memory and the communication port, the control circuitry executing instructions; Logging a user device into a remote client application and transmitting a credential associated with the user device to a User Account and Authentication System (UAAS); authenticating the user device at the UAAS and sending a user identification of the user device to the remote play application; requesting user devices from a console game provider or cloud service (CGP / CS) at the remote play controller, which responds with a list of device identifiers; initiating, in said CGP / CS, a remote service and exchanging connection details with said remote player controller in response to a user selection of remote player management; prompting the user device to invite participants by adding a user identification to an invitation list; In response to a user selection to retrieve a name, sending a request to the UAAS, which responds with a user list; adding the user identification to the invitation list in response to a user selection of adding the retrieved user to the invitation list; In response to a user selection to send an invitation, sending an invitation request from the remote play controller to the CGP / CS, which transmits the invitation request to each invited user; connecting the remote play controller to a console device and initiating interaction through a remote display; sending a remote session request from the remote play controller to the CGP / CS, which stores a remote connection IP address; sending a user invite request from the CGP / CS to each invited user; control circuitry configured to A system comprising:

Images