Method and system for rendering and encoding content for an online interactive gaming session

Abstract

The present application is directed to a method for managing processing power of a server system having one or more processing cores, wherein the one or more processing cores further include a plurality of processing slices. Upon receiving a request to initiate an online game session, the server system assigns each processing slice of the processing core to a subset of the online game session to be executed thereon. A first processing slice is assigned to a first subset of the online game session including a first game session and a second game session. At the first processing slice, a time-sharing processing schedule for the first subset of the online game session is determined. According to the time-sharing processing schedule, the first game session and the second game session share a duty cycle of the first processing slice and are dynamically executed in parallel according to the real-time data processing needs of the first game session and the second game session.

Description

Technical Field

[0001] The present application relates generally to computer technology, including but not limited to methods and systems for managing a server system to support online interactive sessions corresponding to one or more real-time user interactive applications. Background Art

[0002] Internet-connected electronic devices can support a variety of cloud-based media and entertainment applications. These applications include media streaming applications in which a server streams content to a user device, gaming applications in which a user interacts with a game executed on a server from a user device, and various social media and communication applications that allow a large number of users to interact with each other and cloud-hosted content and applications in parallel via their Internet-connected devices. Among cloud-based applications, cloud gaming presents some unique challenges due to the following reasons: widely varying hardware requirements for game titles; a variety of topologies in which cloud-based games can be participated (e.g., by one player, by multiple players in a single location, or by multiple players in multiple locations); the need to reliably and without delay transmit player input to a game server executing a game session and transmit game session output from the game server to the player's device / display; widely varying expectations of players' game speeds and responsiveness; and the desire to provide near real-time game content to viewers in some cases. Other challenges with cloud-based gaming involve providing a consistent gameplay experience for players regardless of where they are located (e.g., close to the server or far away from the server), how they connect to the gaming service (e.g., via a fast or slow Internet connection), and what type of device they use to play the game (e.g., a general-purpose personal device or a dedicated game controller) and view the gameplay output (e.g., a personal device or a media device connected to a media streaming device).

[0003] Specifically, there is a need for a cloud gaming system that supports multiple game sessions for multiple game titles, where games can be executed concurrently with acceptable latency and responsiveness, including for multiple users in the same or different locations playing the same game title, and a variety of input and output devices and network connections. In addition, there is a need for a cloud gaming system that processes user input and outputs high-definition images immediately after receiving player input (e.g., game input typed on a final-use gaming device / controller) in a gaming session, which reflects the results of the player input actions of all game players simultaneously and with acceptable latency. There is also a need for a gaming system that, in some cases, provides a high-definition video stream of gameplay activity to allow viewers to track the gameplay in real time on a corresponding display device. In this way, it would be beneficial to provide a cloud gaming system with an efficient game processing and output mechanism to expand the gaming experience in a wide range of gaming settings, from spontaneous gameplay of users gathered in the same location to online interactive gameplay of multiple users from multiple locations. Summary of the invention

[0004] The embodiments described in this specification are directed to providing a game application programming interface (API) and a cloud platform to achieve efficient, portable and low-latency hosting of game content. Some embodiments dynamically allocate cloud game hardware resources, and monitor and utilize the network bandwidth available to each end user to provide the best cloud game experience. Some embodiments provide multiple performance layers, including layers that support high-performance, real-time game sessions with high-definition media output and end-user streams. Some embodiments support different subscription models and / or are configured to provide one or more concurrent real-time gameplay and / or commentary media streams, which correspond to one or more actual game streams (e.g., video streams output to the client of users participating in online / cloud game sessions via mobile applications or browser-based programs) with almost no delay or no delay. In some embodiments, concurrent gameplay and / or commentary videos are provided to one or more users with almost no delay or no delay via a media streaming site such as YouTube.

[0005] In one aspect of the present application, a method for managing processing power is implemented on a server system, the server system including one or more processing cores (e.g., graphics processing unit "GPU" cores) and a memory for storing programs executed by the processing cores. In some embodiments, the one or more processing cores include (e.g., provide) a plurality of processing slices, each of which is configured to execute one or more online game sessions. The method includes: receiving a request to initiate a plurality of online game sessions, wherein each session corresponds to a respective real-time user interactive application executed on one or more remote client devices. The method also includes assigning each of the plurality of processing slices of the one or more processing cores to a subset of the plurality of online game sessions to be executed thereon (e.g., executed by a respective processing slice on the respective one or more processing cores), including assigning a first processing slice to a first subset of the online game sessions including a first game session and a second game session. The method also includes, at the first processing slice, determining a time-sharing processing schedule for the first subset of online game sessions, such that each of the first subset of game sessions is assigned a duty cycle of the first processing slice. The method further includes sharing a first duty cycle of a first processing slice by executing the first game session and the second game session in parallel according to real-time data processing requirements of the first game session and the second game session according to the time-sharing processing schedule. Optionally, the first processing slice includes only the first duty cycle and does not further include other duty cycles. Alternatively, it includes at least a second duty cycle to be executed after the first duty cycle.

[0006] On the other hand, a corresponding server system is provided, comprising: one or more processing cores, wherein the one or more processing cores include multiple processing slices; and a memory storing one or more programs for execution by the one or more processing cores, the one or more programs including the following instructions: receiving a request to initiate multiple online game sessions, each session corresponding to a corresponding real-time user interactive application executed on one or more remote client devices; assigning each of the multiple processing slices of the one or more processing cores to a subset of the multiple online game sessions to be executed thereon, including assigning a first processing slice to a first subset of the online game sessions including a first game session and a second game session; at the first processing slice, determining a time-sharing processing schedule for the first subset of the online game sessions, so that each of the first subset of the game sessions is assigned to a duty cycle of the first processing slice; and sharing the first duty cycle of the first processing slice by executing the first game session and the second game session in parallel according to the real-time data processing requirements of the first game session and the second game session according to the time-sharing processing schedule.

[0007] On the other hand, a non-transitory computer-readable medium is provided for storing one or more programs for execution by one or more processing cores of a server system, wherein the one or more processing cores include multiple processing slices, and the one or more programs include the following instructions: receiving a request to initiate multiple online game sessions, each session corresponding to a corresponding real-time user interactive application executed on one or more remote client devices; assigning each of the multiple processing slices of the one or more processing cores to a subset of the multiple online game sessions to be executed thereon, including assigning a first processing slice to a first subset of the online game sessions including a first game session and a second game session; at the first processing slice, determining a time-sharing processing schedule for the first subset of the online game sessions, so that each of the first subset of the game sessions is assigned to a duty cycle of the first processing slice; and sharing the first duty cycle of the first processing slice by executing the first game session and the second game session in parallel according to the real-time data processing requirements of the first game session and the second game session according to the time-sharing processing schedule.

[0008] On the other hand, a method for providing media streams in parallel is implemented at a server system including a plurality of encoder cores and a memory storing programs executed by the encoder cores. The method includes generating, by the encoder core, a plurality of media streams including a low-delay stream and a normal-delay stream. The low-delay stream corresponds to an online interactive session and has a faster response rate and a lower transmission delay than the normal-delay stream. The method also includes identifying a predefined frame rate for encoding a sequence of image frames of the low-delay stream. The method also includes, for each image frame in the first sequence of image frames of the low-delay stream, determining a first time interval corresponding to the predefined frame rate, encoding the image frame during the first time interval, and determining the remaining time left in the first time interval after encoding the image frame. The method further includes generating a subset of image frames of a second sequence of the normal-delay stream during the remaining time of the first time interval based on determining that the remaining time left in the first time interval meets the normal delay requirement associated with the normal-delay stream. The method also includes suspending the generation of any image frame of the second sequence of image frames during the remaining time of the first time interval based on determining that the remaining time left in the first time interval does not meet the normal delay requirement.

[0009] On the other hand, a server system is provided, comprising: an encoder; and a memory storing one or more programs for execution by the encoder, the one or more programs including instructions for: generating, by the encoder, a plurality of media streams including a low-delay stream and a normal-delay stream, wherein the low-delay stream corresponds to an online interactive session and has a faster response rate and a lower transmission delay than the normal-delay stream, comprising: identifying a predefined frame rate for encoding a sequence of image frames of the low-delay stream; for each image frame in a first sequence of image frames of the low-delay stream: determining a first time interval corresponding to the predefined frame rate; encoding the image frame during the first time interval; after encoding the image frame, determining a remaining time left in the first time interval; generating a subset of image frames of a second sequence of the normal-delay stream during the remaining time of the first time interval based on determining that the remaining time left in the first time interval meets the normal delay requirement associated with the normal-delay stream; and suspending the generation of any image frames of the second sequence of image frames during the remaining time of the first time interval based on determining that the remaining time left in the first time interval does not meet the normal delay requirement.

[0010] On the other hand, a non-transitory computer-readable medium storing one or more programs for execution by an encoder of a server system is provided, the one or more programs including instructions for: generating, by the encoder, a plurality of media streams including a low-delay stream and a normal-delay stream, wherein the low-delay stream corresponds to an online interactive session and has a faster response rate and a lower transmission delay than the normal-delay stream, including: identifying a predefined frame rate for encoding a sequence of image frames of the low-delay stream; for each image frame in a first sequence of image frames of the low-delay stream: determining a first time interval corresponding to the predefined frame rate; encoding the image frame during the first time interval; after encoding the image frame, determining a remaining time left in the first time interval; generating a subset of image frames of a second sequence of the normal-delay stream during the remaining time of the first time interval based on determining that the remaining time left in the first time interval meets a normal delay requirement associated with the normal-delay stream; and suspending the generation of any image frames in the second sequence of image frames during the remaining time of the first time interval based on determining that the remaining time left in the first time interval does not meet the normal delay requirement.

[0011] In another aspect, a method of encoding a media stream is implemented on a server system, the server system including a GPU, a plurality of encoder cores, a data buffer, and a memory storing a program for execution by the GPU and the encoder cores. The method includes: identifying, by the plurality of encoder cores, frame data of an image frame associated with an online game session and stored in the data buffer; and assigning the plurality of encoder cores to simultaneously and in parallel process a plurality of image tiles of the frame data defined according to a rendering specification. As an example, the rendering specification may define (optionally, how) to segment the image frame. The plurality of image tiles include a first image tile assigned to a first encoder core, and the first image tile includes a first sequence of blocks. The method also includes, at the first encoder core, obtaining, from the data buffer, a first block of a first sequence of blocks of the first image tile assigned to the first encoder core, and obtaining boundary information of the first block. The method further includes encoding a first block of the first sequence of blocks based on the boundary information while or before obtaining a second block following the first block in the first sequence of blocks from the data buffer. The method also includes transmitting the encoded first block to a client device of a user associated with the online game session.

[0012] On the other hand, a server system is provided, comprising: a GPU, multiple encoder cores, a data buffer, and a memory storing one or more programs for execution by the GPU and the encoder cores, the one or more programs including instructions for: identifying, by the multiple encoder cores, frame data of an image frame associated with an online game session and stored in the data buffer; allocating the multiple encoder cores to simultaneously and in parallel process multiple image tiles of the frame data defined according to a rendering specification, wherein the multiple image tiles include a first image tile assigned to a first encoder core, and the first image tile includes a first sequence of blocks, including: at the first encoder core: obtaining from the data buffer a first block in a first sequence of blocks of the first image tile assigned to the first encoder core; obtaining boundary information of the first block; encoding the first block in the first sequence of blocks based on the boundary information while or before obtaining from the data buffer a second block following the first block in the first sequence of blocks; and transmitting the encoded first block to a client device of a user associated with the online game session.

[0013] On the other hand, a non-transitory computer-readable medium storing one or more programs for execution by a server system is provided, wherein the server system also includes a GPU, multiple encoder cores and a data buffer, and the one or more programs include instructions for: identifying, by the encoder core, frame data of an image frame associated with an online game session and stored in a data buffer; allocating multiple encoder cores to simultaneously and in parallel process multiple image tiles of the frame data defined according to a rendering specification, wherein the multiple image tiles include a first image tile assigned to a first encoder core, and the first image tile includes a first sequence of blocks, including: at the first encoder core: obtaining from the data buffer a first block in a first sequence of blocks of the first image tile assigned to the first encoder core; obtaining boundary information of the first block; encoding the first block in the first sequence of blocks based on the boundary information while or before obtaining from the data buffer a second block following the first block in the first sequence of blocks; and transmitting the encoded first block to a client device of a user associated with the online game session.

[0014] In another aspect, a method for rendering an online interactive game session is implemented at a server system, the server system including multiple processing cores and a memory for storing programs executed by the processing cores. The method includes receiving a user command from a first user associated with the online game session. The method further includes determining the type of command and the expected response delay associated with the type of command, the round-trip communication delay between the client device of the first user who is participating in the online game session and the server, the processing delay associated with executing the user command on the server, and the actual transmission delay based on the round-trip communication delay and the processing delay. The method also includes executing the command in the online game session at least by generating one or more intermediate frames and a first frame reflecting the initial result of the command. The one or more intermediate frames when transmitted at a predetermined frame rate occupy a transmission time corresponding to the actual transmission delay. The method also includes transmitting one or more intermediate frames followed by a first frame at a predefined frame rate, so that the first frame is received at the client device of the first user at a time corresponding to the expected response delay.

[0015] In yet another aspect, a method of allocating resources is implemented at a server system including a plurality of virtual machines, each virtual machine having a corresponding resource profile. The method includes receiving a request to establish a real-time interactive session from a computing device, and receiving the request through a network connection with the computing device. The method also includes: determining a device capability of the computing device; determining a connection capability of the network connection; and determining one or more target quality parameters for the real-time interactive session based on the device capability and the connection capability. The method also includes associating a first virtual machine of the plurality of virtual machines with the real-time interactive session based on the determined one or more target quality parameters, and establishing the real-time interactive session based on the association of the first virtual machine with the real-time interactive session. The method also includes providing resources for processing input and generating output within the real-time interactive session to the real-time interactive session based on the resource profile of the first virtual machine.

[0016] According to some aspects of the present application, a server system includes a memory storing instructions for causing the server system to execute any of the above methods.

[0017] In addition, according to some aspects of the present application, the instructions stored in the memory of the server system include instructions for causing the server system to execute any of the above methods.

[0018] Other embodiments and advantages may be apparent to those skilled in the art from the description in this specification and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] For a better understanding of the various described embodiments, reference should be made to the following description of the embodiments in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout the figures.

[0020] Figure 1A is an example online interactive gaming environment 100 according to some implementations. Figure 1B is an example gaming platform 150 for managing third-party content according to some implementations, and Figure 1C According to some embodiments Figure 1B 1. An example flow chart of a method 180 for executing a gaming session on a gaming platform 150 is shown in FIG.

[0021] Figure 2 is a block diagram illustrating an example client device 200 (eg, client device 102 or 104 ) of gaming environment 100 in accordance with some implementations.

[0022] Figure 3 is a block diagram illustrating an example media device in accordance with some implementations.

[0023] Figure 4is a block diagram illustrating an example server according to some implementations.

[0024] Figure 5A is a block diagram of an example server system including at least a GPU, according to some implementations. Figure 5B is a block diagram of a GPU kernel partitioned into multiple processing slices according to some implementations. Figure 5C-5E is an example process in which two or more game sessions are assigned to process a slice in accordance with some embodiments.

[0025] Figure 6 is an example process for dynamically sharing processing slices by two game sessions that generate media content based on Open Graphics Library (OpenGL) according to some implementations.

[0026] Figure 7 is a flow chart illustrating an example method for managing processing capabilities of a server system according to some implementations.

[0027] Fig. 8A is a block diagram of an example server system including at least an encoder according to some implementations. Figure 8B is a timing diagram of time intervals according to some embodiments. Figure 8C and 8D is an example image of two different media streams displayed concurrently at two separate client devices associated with a gaming scene, according to some implementations.

[0028] Fig. 9 is a flow chart illustrating an example method for concurrently providing media streams according to some implementations.

[0029] Fig. 10A is a block diagram of an encoder core assigned to process multiple image tiles of an image frame in a media stream according to some implementations. Fig. 10B is an example process for encoding blocks of image tiles according to some implementations.

[0030] Figures 11A-11C is an example image tile of an image defined according to a rendering specification for frame data of an image frame associated with an online interactive session according to some implementations.

[0031] Figures 12A-12C is an example raster scan of an image associated with an online interactive session in accordance with some implementations.

[0032] Fig.13 is a flow chart illustrating an example method for encoding a media stream according to some implementations.

[0033] Like reference numerals designate corresponding parts throughout the drawings. DETAILED DESCRIPTION

[0034] The embodiments described in this specification are directed to providing a cloud platform and API to enable efficient, portable, low-latency hosting of cloud gaming content including third-party gaming content. Some embodiments dynamically allocate cloud gaming hardware resources (e.g., CPU, GPU, memory, input / output, and video stream encoders), and monitor and utilize the network bandwidth available to a single end user to simultaneously provide the best online gaming experience to the community of gamers. Some embodiments provide multiple performance layers, including layers that support high-performance, real-time gaming sessions with high-definition media streams for end users. Some embodiments support different subscription models and / or are configured to provide one or more concurrent real-time gameplay and / or commentary media streams with little or no delay corresponding to one or more actual game streams (e.g., video streams output to the client devices of users participating in online / cloud gaming sessions via mobile applications or browser-based programs). In some embodiments, real-time gameplay and / or commentary media streams are provided to one or more users with little or no delay via a media streaming website such as YouTube.

[0035] In some embodiments of the cloud gaming environment, a server system provides hardware resources for a real-time, interactive gaming session for processing player input and generating output streams for display to one or more players, and optionally, to a gaming audience. In response to a request to establish a real-time, interactive gaming session, the server system determines the device capabilities (e.g., hardware and / or software capabilities) of the requesting client device (i.e., the player's controller device), the connection capabilities (e.g., bandwidth, latency, and / or error rate) of the network connection, and one or more target quality parameters (e.g., resolution of the output video stream, game response latency, etc.) of the gaming session, and accordingly associates one of its virtual machines with the real-time, interactive session to establish the session.

[0036] In some embodiments, the processing and encoding capabilities of game data (e.g., generating output video streams for players and / or spectators) are managed for one or more processing cores (e.g., GPU cores and encoder cores) in a server system that hosts a real-time, online, and interactive gaming environment. For example, in some embodiments, one or more processing cores operate with multiple processing slices (e.g., each executed on a processing core within a time period, particularly within a predetermined time period (e.g., 16.67 ms)). The server system can assign each of the multiple processing slices to a subset of multiple online game sessions to be executed thereon. For one of the processing slices, the server system determines a time-sharing processing schedule so that the corresponding subsets of the game sessions share the duty cycle of the processing slice and execute in parallel according to their corresponding real-time data processing needs. In addition, in order to accelerate image encoding within a time interval, the encoder of the server system does not need to wait until the GPU has made all the data of the image frame available. Instead, in some embodiments, once the information required for encoding a portion of the image frame is provided by the GPU, the portion is encoded, regardless of whether the GPU is available to make other portions of the image frame that are not related to the encoded portion.

[0037] In addition, the server system can dynamically generate multiple frames in response to user commands received from users participating in an online game session. Based on the type of user command, the server system determines the expected response delay, the actual communication and processing delay, and the actual transmission delay. The user command is then executed in the online game session by generating a frame set that reflects the effect of the command. When the frame set is transmitted at a predetermined frame rate, it takes a transmission time corresponding to the actual transmission delay and can be received at the user's client device within a time corresponding to the expected response delay.

[0038] Figure 1A An example online interactive gaming environment 100 is illustrated in accordance with some implementations. Figure 1B An exemplary gaming platform 150 for managing third-party content according to some embodiments is shown, and Figure 1C According to some embodiments Figure 1B150 . The online interactive gaming environment 100 includes one or more client devices (e.g., client devices 102 and 104). Each client device 102 executes one or more end-user gaming applications (e.g., a game controller application that allows a user to interact with a cloud gaming session executed on a remote server system). A gaming session can be run on a specific gaming application (sometimes referred to as a game title) to allow a user of a client device 102 to play an online interactive game hosted by a server system 114. In some embodiments, a client device 102 (e.g., a host client) is configured to invite one or more other client devices 102 to join a gaming scene of a specific gaming application. The gaming sessions of these client devices 102 are synchronized to display the same gaming scene, optionally with different perspectives corresponding to their respective users.

[0039] Instead, the server system 114 hosts an online interactive game platform to support client devices 102 to participate in one or more game applications, including a specific game application. Specifically, the server system 114 includes multiple user accounts associated with the client device 102, and authenticates the user of the client device in association with each of the one or more game applications. The server system 114 renders and refreshes the scene of the online interactive game on the client device 102, and the client device 102 joins the corresponding game session associated with the scene. In some embodiments, the server system 114 can evaluate the capabilities of the client device 102 and / or the quality of the communication connection between the server system 114 and each client device 102, and adaptively generate a synchronous data stream for the game session associated with the client device 102. By these means, the server system 114 is configured to facilitate synchronous game sessions of online interactive games on two or more client devices 102 at the same time and with relatively low latency.

[0040] In some embodiments, the server system 114 includes a game server 122 and a media streaming server 124. The game server 122 is configured to simultaneously provide two or more media streams for an online interactive game session running on a first client device 102A. The two or more media streams include a low-latency stream and a normal-latency stream, which are provided to the first client device 102A and the commentator client device 104 via one or more communication networks 112, respectively. Optionally, the normal-latency stream is provided for educational purposes. When a user on the first client device 102 participates in a game session on the first client device 102A, the game session is recorded and broadcast to one or more viewers via the normal-latency stream, i.e., the viewers can comment on the game session on the commentator client. The low-latency stream corresponds to the gameplay of the online interactive game session and has a faster response rate and lower transmission delay than the normal-latency stream corresponding to the associated commentary session. For example, the low-latency stream has a predefined frame rate of 60 frames per second (fps) and provides at least one frame during each 16.67 ms time interval to the first client device 102A, and the normal-latency stream has a predefined frame rate of 30 fps and provides at least one frame during each 33.33 ms time interval to the commentator client device 104. In some implementations, the normal-latency stream has a lower resolution than the low-latency stream.

[0041] In some embodiments, the client device 102 or 104 has a display screen integrated therein for displaying media content. In some embodiments, the client device 102 or 104 is coupled to a media device 106 and an output device 108. Specifically, the client device 102 or 104 can be directly (e.g., via a Bluetooth or other wireless communication link), via a local network (e.g., a Wi-Fi network), or via one or more communication networks 112 to communicate with the media device 106. In some embodiments, the client device (102 or 104) and the media device 106 are local to each other (e.g., in the same room, in the same house, etc.). The media device 106 is further coupled to one or more output devices 108 (e.g., a television, a display monitor, a sound system, a speaker, etc.) that can output visual and / or audio content. The media device 106 is configured to output content to the output device 108. In some embodiments, the media device 106 is a delivery device (e.g., CHROMECAST by Google Inc.) or a device that otherwise includes delivery functionality.

[0042] Each client device 102 or 104 is capable of communicating data and sharing information with each other, a central server or cloud computing system (e.g., server system 114), and / or other devices connected by the network (e.g., another client device 102 or 104, a media device 106, and an output device 108). Data communications may be performed using any of a variety of custom or standard wireless protocols (e.g., IEEE802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART, MiWi, etc.) and / or any of a variety of custom or standard wired protocols (e.g., Ethernet, HomePlug, etc.), or any other suitable communication protocol including communication protocols that have not been developed as of the date of filing of this document. In some embodiments, the online interactive gaming environment 100 includes conventional network equipment (e.g., a router) via which a collection of client devices 102 and 104 and their corresponding media and output devices (if any) are communicatively coupled to each other on a local area network, and the local area network is communicatively coupled to other portions of a communication network 112 (e.g., a wide area network and the Internet). In some embodiments, each of the client devices 102 and 104 optionally communicates with one or more other client devices, a corresponding media device 106, or a corresponding output device 108 using one or more radio communication networks (e.g., ZigBee, Z-Wave, Insteon, Bluetooth, Wi-Fi, and / or other radio communication networks).

[0043] In some embodiments, the client devices 102 are remote from each other, i.e., they are not located in the same room or even building. Figure 2 228 in the game application 102) to start the game to execute on each client device 102. In some embodiments, for each client device 102, the game application independently establishes an online game session 116 with the server system 114. The online game sessions 116 of two or more client devices 102 (e.g., 102A and 102B) are related to each other (e.g., because they are all playing games in the same game domain of the game application), and therefore, share the game scene in the game application. The related online game sessions 116 are synchronized with each other, and each online game session 116 optionally shows the same game scene with a unique player perspective corresponding to the corresponding client device 102. Therefore, the user of each client device 102 can play the game on the corresponding client device and affect the output of the online game sessions 116 on other client devices 102.

[0044] Optionally, in some other embodiments, after the game application of the first client device 102A establishes the online game session 116, one or more second client devices 102B are invited to join the online game session 116 through an invitation message, and for example, a message with a link (e.g., a URL address) to join the online game session 116 is sent to each of the second client devices 102B. Appropriate controller configurations are provided to each second client device 102B invited to join the online game session 116. In this application, when the second client 102B joins the online game session 116, the server system 114 creates a separate game session for each individual second client device 102B. Each individual game session 116 of the corresponding second client device 102B is synchronized with the game session 116 of the first client device 102A and shares the same scene with the game session 116, but may have a unique player perspective corresponding to the corresponding second client device 102B. After each second client device 102B has received the appropriate controller configuration and joined the online game session 116 (more precisely, started its associated online game session 116), the user can play the game on the corresponding second client device 102B and affect the output of the online game session 116 running on the other client devices 102.

[0045] The client device 102 is a device that includes and can run one or more different user applications including a game application. In some embodiments, the client device 102 is a smart phone, a tablet device, a laptop computer, a desktop computer, or a multimedia device, each of which is configured to run a game application (also referred to as a controller application) for implementing an online game session. In some embodiments, the client device 102 includes a dedicated game controller that executes a game application (also referred to as an end-user game control application) that allows a user to interact with the online game session. In some embodiments, the client device 102 includes one or more user applications configured to operate in conjunction with the media device 106. In some embodiments, the application includes a media device application for pairing the client device 102 with the media device 106 and configuring the media device 106. The application also includes one or more applications that can deliver associated content to the media device 106. In some implementations, the application delivers data and / or content to the media device 106 by sending the data / content directly to the media device 106 (e.g., via a local network) and / or by directing the media device 106 to a remote location (e.g., a URL or other link to a location at a server system) from which the media device 106 can stream or otherwise receive the data / content. The media device 106 receives the data / content from the application and / or remote location and outputs visual and / or audio content corresponding to the received data / content to the output device 108. Thus, an online gaming session 116 is established between the gaming application running on the client device 102, the remote server system 114, and the media device 106.

[0046] In some embodiments, as part of the process of linking related online game sessions 116, the server system 114 evaluates the capabilities of each corresponding client device 102 and / or the quality of the communication connection between the server system 114 and the client device 102. In some embodiments, the server system 114 measures the network latency between the client device 102 and the server system 114. If the measured latency is above a threshold and a lower latency connection is available, the server system 114 can suggest that the client device 102 change to a lower latency connection, or invite the user of the client device 102 to change the client device 102 to a lower latency connection. For example, if the client device 102 is on a cellular wireless connection 118 and a local network is available, the server system 114 can suggest that the client device 102 should connect via the available local network. In some embodiments, the latency threshold requirements vary between games. For example, some games (e.g., action games) are best experienced on lower latency connections, while other games (e.g., online board games or card games) are less demanding on latency. In view of these different requirements associated with different types of games, the server system 114 can make connection recommendations.

[0047] As part of the client device 102 starting or joining a game session 116, the server system 114 communicates with the client device 102 to set up a controller on the client device 102. In some embodiments, this includes the server system 114 evaluating whether the client device 102 has the resources and communication capabilities required for the controller. Depending on the available resources at the client device 102, the quality of the connection, and the game requirements, the controller can be implemented differently at the client device 102. In some embodiments, a web-based controller interface can be used to play the game. For example, the controller interface for the game can be embedded in a web page, and the web page is rendered in a web browser on the client device 102. Alternatively, in some embodiments, the standardized controller is implemented in a predefined application that is not specific to the game or is directly associated with the game (e.g., a delivery device application, such as Chromecast or Google Cast by Google Inc., or other media device applications), or in the operating system of the client device 102. For example, the device operating system or predefined application on the client device 102 can have a controller submodule. The controller submodule includes one or more standardized controller configurations, templates, etc. Each standardized controller configuration configures the controller submodule to utilize input devices and / or sensors on the client device 102 in some manner to implement a virtual controller. The standardized controller configuration used may vary depending on the type of game and / or client device.

[0048] In addition, in some embodiments, the game has a specific controller configuration that can be implemented on the controller submodule. Such a configuration can be stored at the server system 114 and transmitted to the client device 102 as a part of the process of the client device 102 joining or starting an online game session 116. In some embodiments, the specific controller configuration can be a mixture of a custom controller or a standard controller and a custom controller completely. In addition, in some embodiments, the game requires a specific application associated with the game. For example, the game may require a controller application specifically associated with the game. In some embodiments, the client device 102 can be instructed to download a specific application or a predefined application as a part of starting or joining a session 116. For example, if the client device 102 does not have a predefined application (and controller submodule) or a specific application associated with the game, and such an application is required for playing games, the server system 114 indicates that the client device 102 prompts its user to download and request permission to continue to the user.

[0049] The server system 114 stores at least one or more game applications (e.g., Figure 2 The server system 114 stores user information 126 associated with a user account for each of the online gaming applications 228 of the client device 102. Examples of user information 126 include, but are not limited to, user account information (e.g., identification and password), membership type, preferences, and activity history. In some embodiments, the server system 114 stores session data 128 associated with online gaming sessions participated in on the client device 102. Examples of session data 128 for each online gaming session 116 include, but are not limited to, frame rate, rendering specifications, normal latency requirements, GPU allocation information, encoder allocation information, identification of the related session, and latest status information. The session data 128 is accessed more frequently than the user information 126. In some embodiments, the memory storing the user information 126 and session data 128 is composed of flash memory and random access memory (RAM), respectively.

[0050] In some embodiments, the creation of a real-time interactive gaming environment 100 requires hardware resources that are roughly proportional to the pixel resolution of the game scene to be rendered. The physical processors of the server system 122 can be virtualized to enable different resource allocations according to the target resolution. Virtualization technology is based on time division multiplexing (TDM), spatial multiplexing, or a combination thereof. In some embodiments, TDM provides workload access to relatively large resources in a short amount of time, and spatial multiplexing provides continuous access to relatively small resources. In some embodiments, hardware virtualization is limited to one of TDM and spatial multiplexing, which may not be optimal. Spatial allocation of workloads throughout the frame may not fully utilize physical resources, and TDM may destroy many workloads and does not provide continuous access to resources throughout the real-time deadline for rendering frames. A game scene that must maintain 60fps has a real-time deadline of 16.67ms to complete the frame, and segmentation by using TDM access may result in rendering being impossible despite sufficient resources.

[0051] According to some embodiments of the present application, processing power is therefore managed for a server system 114 (particularly a game server 122) that includes one or more processing cores (e.g., a GPU core of a GPU 140) and hosts a real-time, online, and interactive gaming environment 100. In some embodiments, the one or more processing cores run with a plurality of processing slices (e.g., each lasting 16.67 milliseconds). Upon receiving a request to initiate a plurality of online gaming sessions 116, the server system assigns each of the plurality of processing slices of the one or more processing cores to a subset of the plurality of online gaming sessions 116 to be executed thereon, for example, assigning a first processing slice to a first subset of online gaming sessions, the online gaming sessions including a first gaming session and a second gaming session. Each session 116 corresponds to a corresponding real-time user interactive application executed on one or more remote client devices 102. At the first processing slice, the server system 114 determines a time-sharing processing schedule for the first subset of online gaming sessions 116, such that each of the first subset of gaming sessions 116 is assigned a duty cycle of the first processing slice. According to the time-sharing processing schedule, the first and second game sessions 116 share a first duty cycle of the first processing slice, thereby executing in parallel according to the real-time data processing requirements of the first and second game sessions 116 .

[0052] Online interactive gaming sessions 116 require encoding of high quality real-time video streams with high resolution and low latency. In some implementations, an application specific integrated circuit (ASIC) may be designed for a very specific use case. Alternatively, in some implementations, a general purpose ASIC is configured to support a first set of relatively low resolution game contexts and a second set of relatively high resolution game contexts while meeting the latency requirements of both sets of game contexts.

[0053] In one aspect of the present application, the server system 114 (particularly the game server 122) is configured to provide two or more media streams simultaneously. The encoder 142 of the server system 114 is configured to generate a plurality of media streams including a low-delay stream 130 and a normal delay stream 132. The low-delay stream 130 corresponds to the online interactive session 116 and has a faster response rate and a lower transmission delay than the normal delay stream 132. Specifically, the encoder 142 first identifies a predefined frame rate (e.g., 60pfs) for encoding a sequence of image frames of the low-delay stream, and determines a first time interval (e.g., 16.67ms) corresponding to the predefined frame rate. For each image frame in the first sequence of image frames of the low-delay system 130, the encoder 142 encodes the image frame during the first time interval, and determines the remaining time left in the first time interval after encoding the image frame. When the remaining time left in the first time interval satisfies the normal delay requirement associated with the normal delay stream 132, the encoder 142 of the server system 114 generates a subset of the image frames of the second sequence of the normal delay stream 132 during the remaining time of the first time interval. Alternatively, when the remaining time left in the first time interval does not meet the normal delay requirement, the encoder 142 suspends the generation of any image frames of the second sequence of image frames during the remaining time of the first time interval. In this way, the encoding priority is given to the low-delay stream 130, while the remaining time left in the first time interval is not wasted, but is used to encode the normal delay stream 132.

[0054] In addition, in order to speed up the encoding of the image within the first time interval, the encoder 142 of the server system 114 does not need to wait until the GPU 140 makes all the data of the image frame available. Instead, in some embodiments, as soon as the information required for encoding a portion of the image frame is provided by the GPU 140, the portion is encoded, regardless of whether other portions of the image frame that are not relevant for encoding the encoded portion are available through the GPU 140. More specifically, for the server system 114 including the GPU, the encoder 142, and the data buffer 144, the encoder 142 identifies the frame data of the image frame associated with the online game session 116 and stored in the data buffer 144, and allocates multiple encoder cores of the encoder 142 to process multiple image tiles of the frame data simultaneously and in parallel. The multiple image tiles include a first image tile assigned to a first encoder core, and the first image tile includes a first sequence of blocks. The encoder 142 obtains a first block in the first sequence of blocks of the first image tile assigned to the first encoder core and boundary information of the first block from the data buffer 144. The encoder encodes the first block in the first sequence of blocks based on the boundary information while or before obtaining the second block following the first block in the first sequence of blocks from the data buffer 144. The encoder 142 then transmits the encoded first block to the client device 102 of the user associated with the online game session 116. By these means, the server system 114 better utilizes the encoding capacity of the encoder 142 by reducing the idle time for waiting for the GPU 140 to provide a complete set of raw data for each image frame, and thus can encode image frames of two or more media streams within a time interval associated with a predefined frame rate of high-definition game content (e.g., 16.67 ms).

[0055] In addition, the server system 114 can dynamically generate multiple frames in response to user commands from users playing the online game session 116. Based on the type of user command (e.g., jump, shoot gun), the server system determines the expected response delay, the actual communication and processing delays, and the actual transmission delay. The number of frames is calculated based on the actual transmission delay and the predefined frame rate. As a result, the command is executed in the online game session by generating a frame set that reflects the effect of the command. The frame set when transmitted at the predetermined frame rate takes a transmission time corresponding to the actual transmission delay and can be received at the user's client device in a time corresponding to the expected response delay.

[0056] refer to Figure 1B, the server system 114 provides a gaming API and a cloud platform 150 to enable efficient, portable, low-latency hosting of third-party gaming content used in online gaming sessions 116. In some implementations, the gaming API and cloud platform 150 are enabled by the server system 114, which further includes one or more of the following: a front-end server 134, a media streaming server 124, a gaming server 122, and one or more third-party content servers 136. The front-end server 134 manages user accounts associated with the client devices 102 and 104, for example, subscriptions to memberships in one or more online interactive games through the user accounts. After a client device 102 logs into its corresponding user account and joins its online game session 116, the game server 122 sets up the game session 116 and manages each specific game session 116 for the corresponding client device 102 by obtaining game content from the content server 136, sending the game content to the game application executing on the client device 102, identifying user requests or actions, rendering gameplay output for the client device 102 in response to the user request or action, and storing game state data during the corresponding game session 116. The game server 122 includes one or more processing units (e.g., CPU 138, GPU 140, and encoder 142), memory 146, and data buffer 144, which temporarily stores multimedia content generated by the GPU 140 and provides the multimedia content to the encoder 142 for further encoding (e.g., normalization or compression). The data buffer 144 is optionally integrated in the memory 146 or independent of the memory 146.

[0057] In some embodiments, the game server 122 dynamically allocates cloud game hardware resources (e.g., GPU 140 and encoder 142), and monitors and utilizes the network bandwidth available to individual end users to provide the best cloud game experience. In some embodiments, the game server 122 provides multiple performance layers, including a layer that supports high-performance, real-time game sessions with high-definition video / media streaming. In some embodiments, the game server 122 supports different subscription models and / or is configured to provide one or more concurrent real-time gameplay and / or commentary media streams, which correspond to one or more actual game streams (e.g., video streams output to the client devices of users participating in online / cloud game sessions via mobile applications or browser-based programs) with almost no delay or no delay. Specifically, the game server 122 is configured to generate concurrent media streams and commentary videos for gameplay, and provide commentary videos for concurrent gameplay to the media streaming server 104. Such commentary videos are provided to one or more users with almost no delay or no delay via a media streaming website, such as YouTube. The media streaming site is optionally managed by the media streaming server 124.

[0058] Some embodiments enable the hosting of public events in conjunction with gaming competitions. For example, in conjunction with a multiplayer gaming event or competition based on a hosted game, a cloud gaming site hosted by a gaming server 122 may broadcast or stream the following to a particular reviewer client device 104, optionally via a media streaming server 123: (a) one or more concurrent auxiliary or supplemental media streams, including an associated commentary track / stream, (b) a gaming stream from the perspective of different competitors, with a highlight stream showing particularly compelling gaming action based on cloud server analysis and / or scoring of multiple gaming sessions associated with the gaming event, (c) one or more gaming viewpoint streams reflecting gameplay sessions 116 of one or more active players, and / or (d) a tutorial track from one or more active players and / or reviewers, which may include real-time picture-in-picture (PIP) video of active players sent to the cloud gaming server system 114 and their corresponding gameplay responses.

[0059] According to some embodiments, examples of third-party content that can be effectively hosted by content server 136 include, but are not limited to, sports games, racing games, role-playing games (RPGs), and first-person shooter (FPS) games. Different instances of these games may have widely varying cloud hardware requirements and networks based on different associated latency requirements and expectations, output video resolutions and game server computing workloads and video encoding / streaming resources, and network bandwidth (e.g., to ensure an optimal user gaming experience - in some cases consistent with different subscription performance tiers).

[0060] In some embodiments, the front-end server 134 provides an account management API and / or software module that monitors gameplay activity and related requests of subscribers (e.g., requests by end users to invite other players to participate in a game session, upgrade their game tools, and / or game performance), and transmits or makes available the associated information to the third-party content server 136 via the API to enable content providers to track the settings of their subscribers and / or followers (including but not limited to billing information, in-game credits, subscription levels, etc.). In some embodiments, content providers of hosted content can provide one or more different subscription models for hosted content via the same hosting platform 150. In some embodiments, users (e.g., subscribers of a gaming service) are granted unlimited access and gameplay to all games offered by the content provider on the hosting platform 150. In some embodiments, users (e.g., subscribers of a gaming service) are granted unlimited access and gameplay to one or more specific game franchises (e.g., a specific football or first-person shooter franchise) offered by the content provider on the hosting platform 150. In some embodiments, subscriptions are for limited participation by the user - participation may be limited based on gameplay time, hardware resource levels committed by the end user, or end user device type / location. In some embodiments, the account API and modules configure and monitor gameplay sessions and enable content providers to track the gaming activity of individual subscribers based on their most recent subscription information, even during active gameplay.

[0061] The server system 114 enables cloud functionality that allows users to move around, for example, suspend a first game stream of a first game session executing on a first client device 102, and restart the first game stream on a second game session of a second client device 102 to continue the first game session. The server system 114 also supports multiple players at scale and provides a richer, more persistent cloud-based world. The server system 114 uses a cloud-based system to store session data 128 related to different game sessions 116 of the same user or different game sessions 116 of different users.

[0062] The server system 114 renders the game content on a plurality of client devices 102 and 104, including but not limited to mobile phones, tablet computers, desktop computers, and televisions. Optionally, the game content is dynamically adjusted to conform to the specifications of these client devices 102 and 104. In some embodiments, the client devices 102 and 104 have limited or no storage capabilities because the game API platform 150 provides instant access and requires little or no user device storage (e.g., a user can start playing in 5 seconds and save 250GB of console hard drive space).

[0063] In addition to the game content, the server system 114 also streams additional content to the client devices 102 and 104, such as new league rosters, statistics, and preview access to early titles, which are optionally updated regularly (e.g., updated over time, daily or hourly). In some embodiments, the additional content includes search results from an Internet search or a database search.

[0064] In some embodiments, the server system 114 supports a real-time online community associated with a game application. Users (e.g., subscribers of the service) participate in live events, tournaments, or activities on the corresponding game API platform all day long. Examples of live events, tournaments, or activities include watching live game sessions played by other users, publishing your achievements to a public domain (e.g., YouTube), and getting live tips and instructional videos. For example, in response to user actions, the game server 122 provides two or more real-time streams 130 and 132. While the first game stream 130 is maintained on the first game session 116 of the first client device 102A of the game player, the server system 114 also broadcasts a second real-time comment stream 132 (e.g., YouTube stream) to one or more other client devices 104 (e.g., subscribers). The second real-time comment stream 132 allows users to share game experience with listeners. Optionally, the second real-time stream is a reproduction of the screen of the first client device 102A of the player. The server system 114 can obtain an audio stream in which the player explains the first game session 116, or a video stream of the player who plays and explains the first game session 116. The audio stream is optionally played for the audience while the second real-time commentary stream 132 is played for the audience. The video stream is optionally played in an embedded window in the second real-time commentary stream 132. Figures 8A-8C To explain in more detail.

[0065] Some embodiments provide true mobile gaming - users are able to take their most anticipated games with them to any location or client device. For example, a user can start an online game session 116 on a mobile device 102A during his or her commute, and then seamlessly resume the game session 116 at his or her destination on a laptop 102B. Moreover, in some embodiments, based on the different client device resources available to the user when the game session 116 switches between different devices 102, the server system 114 (specifically, the game server 122) can dynamically deploy different sets of hard resources (e.g., GPU 140 and encoder 142) to optimize the user's gaming experience based on different end-user current device resources (e.g., client hardware capabilities and network bandwidth).

[0066] In server system 114, front-end server 134 and game server 122 can have corresponding user account system.In example, the user account system for front-end server 134 is used to manage the subscription to specific game content and service, and the user account system (for example, YouTube or Google account) for game server 122 is used to manage game experience (for example, rendering game content to meet specific game standard) and many other purposes.In some embodiments, these two user account systems share clients and use data (for example, social, friend, existence, authentication, account information, billing information).And, content front-end server 134 provides the service layer on the top of the technical layer enabled by game server 122.In some embodiments, game content server 136 manages the additional user account system for accessing their content.Alternatively, the additional user account system for game content is integrated with the user account system of the front-end server 134 for managing user subscription.

[0067] like Figure 1B As shown in , the game API platform 150 is created / hosted by the game server 122 and enables the game session 116 together with the front-end server 134 and the content server 136. The front-end server 134 is configured to provide services to the users of the game session 116 and manage the user's account. Optionally, the user must subscribe to the game service via the front-end server 134. The content server 136 provides the game content related to the game session 116. Figure 1C, the game server 122 executes (152) the game application based on the corresponding game content to enable the game session 116. In some embodiments, the front-end server 134 and the game server 122 are owned and managed by different entities. Specifically, the online game application is executed to display a game user interface on the first client device 102 or an output device 108 (e.g., a television) coupled thereto. A user selection for subscribing to a game service is received at the user interface. In order to subscribe to the game service, the server system 114 receives (154) user information, thereby allowing the user of the first client device 102 to register for one of the different types of services (e.g., a free trial service or a regular subscription service). On the user interface, the user of the client device 102 can browse (156) a series of rich hubs (also referred to as content franchises) for each content domain. The user interface is configured to display game content, additional content (e.g., news, statistics), and live events (e.g., games broadcast from another user, online chat messages).

[0068] In some embodiments, a user selection of a game (e.g., a live soccer game) is received (158) at a user interface. In response to the selection, a gaming application is loaded for the first client device 102 within a predetermined delay time, and the gaming environment is made available to the user. Alternatively, in some embodiments, in response to the user action, the server system 114 provides (160) a real-time stream to the user of the first client device 102. For example, the server system 114 provides a first gaming stream in a first gaming session of the first client device, and simultaneously broadcasts a second real-time YouTube stream to one or more other client devices 104. The second real-time stream allows the user to share the gaming experience with an audience. Optionally, the second real-time stream includes an audio feed or a video feed provided by the user of the first client device 102 while the user is playing a game based on the first gaming stream.

[0069] In some implementations, the user of the first client device 102 may need to use a different client device (e.g., due to moving to a different location). In response to the user action to pause the first game session, the server system suspends (162) the first game stream of the first game session executing on the first client device 102 and records the latest state information about the first game session 116. The user of the first client device 102 may restart the first game stream on the second game session of the second client device to continue the first game session. The latest state information is transmitted to the second game session to allow users of the first and second client devices to have a seamless gaming experience.

[0070] Figure 21 is a block diagram illustrating an example client device 200 (e.g., client device 102 or 104) of a gaming environment 100 according to some embodiments. Examples of client devices 200 include, but are not limited to, mobile phones, tablet computers, laptop computers, desktop computers, and wearable personal devices. Client device 200 includes one or more processing units (CPUs) 202, one or more network interfaces 204, memory 206, and one or more communication buses 208 for interconnecting these components (sometimes referred to as chipsets). Client device 200 includes one or more input devices 210 that facilitate user input, such as a keyboard, a mouse, a voice command input unit or microphone, a touch screen display, a touch-sensitive input pad, a gesture capture camera, or other input buttons or controls. In addition, some client devices 200 use microphones and voice recognition or cameras and gesture recognition to supplement or replace keyboards. In some embodiments, client device 200 includes one or more cameras, scanners, or photo sensor units for capturing images of, for example, a graphic series code printed on an electronic device. Client device 200 also includes one or more output devices 212 capable of presenting a user interface and displaying content, including one or more speakers and / or one or more visual displays. Optionally, the client device 200 includes a location detection device 214, such as a GPS (global positioning satellite) or other geo-location receiver, for determining the location of the client device 200. The client device 200 may also include a proximity detection device 215, such as an IR sensor, for determining the proximity of the media device 106 and / or the player client 104. The client device 200 may also include one or more sensors 213 (e.g., accelerometers, gyroscopes, etc.) for sensing the motion, orientation, and other parameters of the client device 200, which may be used as input.

[0071] Memory 206 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally, includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Memory 206 optionally includes one or more storage devices remotely located from one or more processing units 202. Memory 206, or alternatively, non-volatile memory within memory 206, includes non-transitory computer-readable storage media. In some embodiments, memory 206 or non-transitory computer-readable storage media of memory 206 stores the following programs, modules, and data structures, or a subset or superset thereof:

[0072] = Operating system 216, including procedures for handling various basic system services and for performing hardware-related tasks;

[0073] A network communication module 218 for connecting the client device 200 to other devices (e.g., server system 114, media device 106, and other client devices 102 or 104) via one or more network interfaces 204 (wired or wireless) and a local network and one or more networks 112 (such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.);

[0074] A user interface module 220 for enabling presentation of information at the client device 200 via one or more output devices 212 (e.g., a display, speakers, etc.) (e.g., a graphical user interface for presenting applications, widgets, websites and their pages, and / or games, audio and / or video content, text, etc.);

[0075] An input processing module 222 for detecting one or more user inputs or interactions from one of the one or more input devices 210 and interpreting the detected inputs or interactions;

[0076] A web browser module 224 for navigating, requesting (e.g., via HTTP), and displaying websites and their web pages, including a web interface for joining a session 116;

[0077] ● a media device application 226 for interacting with the media device 106 , including logging into a user account associated with the media device 106 , controlling the media device 106 if associated with the user account, and editing and commenting on settings and data associated with the media device 106 ;

[0078] A gaming application 228 for providing gaming on the client device 200, including facilitating corresponding gameplay and facilitating invitations of other players;

[0079] ● A game controller module 230 for providing a game play input interface to the game application 228;

[0080] ● Data download module 231, for downloading data (e.g., game controller configuration 456 ( Figure 4 ), game applications 228 and other applications, updates to modules and applications and data in memory 206); and

[0081] Client data 232, which stores at least data associated with the game application 228 and other applications / modules, including:

[0082] o Client device settings 234, for storing information associated with the client device 200 itself, including common device settings (e.g., service tier, device model, storage capacity, processing power, communication capabilities, etc.);

[0083] o Media device settings 236 for storing information associated with a user account of the media device application 226, including one or more of account access information and device settings (e.g., service tier, device model, storage capacity, processing capability, communication capabilities, etc.);

[0084] o Game application settings 238 for storing information associated with a user account of the game application 228, including one or more of account access information, in-game user preferences, game play history data, and other player information;

[0085] o Game controller configuration 240, for storing configurations of the game controller module 230 for the game application 228 (e.g., from Figure 4 information associated with the game controller configuration 456 received by the user; and

[0086] o Location / proximity data 242 , including information associated with the presence, proximity, or location of any of the client devices 102 and 104 and the media device 106 .

[0087] In some implementations, the game controller module 230 is part (e.g., a submodule) of the media device application 226 or another application in the memory 206. In some implementations, the game controller module 230 is part of the operating system 216. In some implementations, the game controller module 230 is a different module or application.

[0088] In some implementations of the client device 102, the media device application 226 (and corresponding media device settings 236) and the game application 228 (and corresponding game application settings 238) are optional. Depending on the particular game that the client device 102 is invited to join, the media device application 226 and the game application 228 may not be required to play. If any of these applications are required in order to play the game (e.g., the game uses the game controller module 230 in the media device application 226), and the application is not in the memory 206, the client device 102 can be prompted to download the application.

[0089] Each of the elements identified above can be stored in one or more aforementioned storage devices, and corresponds to the instruction set for performing the above functions. The modules or programs (that is, instruction sets) identified above do not need to be implemented as independent software programs, processes, modules or data structures, and therefore the various subsets of these modules can be combined or otherwise rearranged in various implementations. In some embodiments, memory 206 is optionally stored in the subset of the modules and data structures identified above. In addition, memory 206 optionally stores additional modules and data structures not described above.

[0090] Figure 3 1 is a block diagram illustrating an example media device 106 of a gaming environment 100 according to some embodiments. The media device 106 typically includes one or more processing units (CPUs) 302, one or more network interfaces 304, memory 306, and one or more communication buses 308 for interconnecting these components (sometimes referred to as chipsets). Optionally, the media device 106 includes a proximity / position detection unit 310, such as an IR sensor, for determining the proximity of a client device 102 or 104.

[0091] Memory 306 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Memory 306 optionally includes one or more storage devices remotely located from one or more processing units 302. Memory 306 or alternatively non-volatile memory within memory 306 includes non-transitory computer-readable storage media. In some embodiments, memory 306 or non-transitory computer-readable storage media of memory 306 stores the following programs, modules, and data structures, or subsets or supersets thereof:

[0092] = Operating system 316, including procedures for handling various basic system services and for performing hardware-related tasks;

[0093] A network communication module 318 for connecting the media device 106 to other computers or systems (e.g., server system 114 and client device 102) via one or more network interfaces 304 (wired or wireless) and one or more networks 112 (such as the Internet, other wide area networks, local area networks, metropolitan area networks, cable television systems, satellite television systems, IPTV systems, etc.);

[0094] A content decoding module 320 for decoding content signals received from one or more content sources (e.g., server system 114 output from gaming session 116) and outputting the content in the decoded signals to an output device 108 coupled to media device 106;

[0095] A proximity / position determination module 322 for determining the proximity of the client devices 102 and / or 104 based on proximity-related information detected by the proximity detection unit 310 or provided by the server system 114 ;

[0096] ●Media display module 324, used to control media display; and

[0097] ●Media device data 326, for storing at least the following data:

[0098] o Media device settings 328 for storing information associated with a user account for a media device application, including one or more of account access information and device settings information (e.g., service tier, device model, storage capabilities, processing capabilities, communication capabilities, etc.); and

[0099] o Location / proximity data 330 , including information associated with the presence, proximity, or location of any of the client devices 102 and / or 104 and the media device 106 .

[0100] Each element identified above can be stored in one or more aforementioned memory devices, and corresponds to the instruction set for performing the above functions. The modules or programs (i.e., instruction sets) identified above do not need to be implemented as separate software programs, processes, modules or data structures, and therefore the various subsets of these modules can be combined or otherwise rearranged in various implementations. In some embodiments, memory 306 optionally stores the subsets of the modules and data structures identified above. In addition, memory 306 optionally stores additional modules and data structures not described above.

[0101] Figure 4 1 is a block diagram of an example server in a server system 114 of a gaming environment 100 according to some embodiments. The server system 114 typically includes one or more processing units (e.g., CPU 402, GPU 140, and encoder 142), one or more network interfaces 404, memory 406, and one or more communication buses 408 for interconnecting these components (sometimes referred to as chipsets). The server system 114 may optionally include one or more input devices 410 to facilitate user input, such as a keyboard, mouse, voice command input unit or microphone, touch screen display, touch-sensitive input pad, gesture capture camera, or other input buttons or controls. In addition, the server system 114 may use a microphone and voice recognition or a camera and gesture recognition to supplement or replace the keyboard. In some embodiments, the server system 114 may optionally include one or more cameras, scanners, or photo sensor units for capturing images of, for example, a graphic series code printed on an electronic device. The server system 114 may also include one or more output devices 412 capable of presenting a user interface and displaying content, including one or more speakers and / or one or more visual displays.

[0102] Memory 406 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Memory 406 optionally includes one or more storage devices remotely located from one or more processing units. Memory 406, or alternatively non-volatile memory within memory 406, includes non-transitory computer-readable storage media. In some embodiments, memory 406 or non-transitory computer-readable storage media of memory 406 stores the following programs, modules, and data structures, or subsets or supersets thereof:

[0103] = Operating system 416, including procedures for handling various basic system services and for performing hardware-related tasks;

[0104] ● A network communication module 418 for connecting the server system 114 to other devices (e.g., the server system 114, the client devices 102 and / or 104, and the various servers in the media device 106) via one or more network interfaces 404 (wired or wired) and one or more networks 112 (such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.);

[0105] A user interface module 420 for enabling presentation of information at the client device 102 and / or 104 (e.g., a graphical user interface for presenting applications, widgets, websites and their web pages and / or games, audio and / or video content, text, etc.);

[0106] ● Media device module 422 (optional), executed to provide server-side functionality for device provisioning, device control, and user account management associated with media devices 106;

[0107] A proximity / position determination module 424 for determining proximity of the client device 102 and / or 104 to the media device 106 based on location information of any of the client device ( 102 and / or 104 ) and the media device 106 ;

[0108] A game server module 426 for providing server-side functionality associated with a game (e.g., a game application 228), including, but not limited to, setting up a game session, storing session state data and other game-related data, processing gameplay input from client devices 102 and 104, and rendering game output in response to the gameplay input;

[0109] ● a media stream server module 438 for hosting a media streaming site, receiving concurrent auxiliary or supplemental media streams associated with an online gaming session, and providing the concurrent media streams to a client device 104 for concurrent display with an online gaming session being executed by a gaming application 228 on the same client device 104 or a different client device 102;

[0110] A front-end server module 440 for managing user accounts associated with the client devices 102 and 104, for example, subscriptions to memberships for one or more online interactive games through the user accounts, enabling services to subscribers for forwarding subscriber requests to the game server module 426, and monitoring gameplay activity and related requests of subscribers;

[0111] 0 media content server module 442, for providing access to game content hosted by one or more third-party content providers;

[0112] ● A device / network assessment module 444 for assessing the device and network capabilities of the client device 102 and / or 104, including but not limited to assessing the network bandwidth of the connection to the client device 102 / 104 and assessing whether the client device 102 / 104 has the modules or applications required to play the game;

[0113] A data transfer module 446 for providing data (e.g., game controller configuration 456, software updates, etc.) to the client device 102 / 104; and

[0114] ●Server system data 448, including:

[0115] o Client device settings 450, for storing information associated with client devices 102 and 104, including common device settings (e.g., service tier, device model, storage capacity, processing power, communication capabilities, etc.);

[0116] o Media device settings 452 (optional), for storing information associated with a user account of the media device application 422, including one or more of account access information and device setting information (e.g., service tier, device model, storage capacity, processing capability, communication capability, etc.);

[0117] o Location / proximity data 454 , including information related to the presence, proximity, or location of any of the client devices 102 and / or 104 and the media device 106 ;

[0118] o Game controller configurations 456, for storing controller configurations for various games; and

[0119] o User information 126, which is used to store and manage one or more game applications hosted on the server system 114 (e.g., Figure 2 Information associated with a user account for each of the gaming applications 228 in the game, including, for example, user account information (e.g., identification and password), membership type, preferences, and activity history; and

[0120] ○ Game session data 128, used to store data associated with the game session (e.g., game status data, other game-related data), including, for example, data 128-1 for a first game session and data 128-2 for a second game session, wherein the session data 128 for each game session includes but is not limited to frame rate, rendering specifications, normal latency requirements, GPU allocation information, encoder allocation information, identification of the relevant session, and the latest status information associated with the corresponding game session.

[0121] In some implementations, the game server module 426 includes the following programs, modules, or subsets or supersets thereof:

[0122] GPU time sharing module 428 for allocating each processing slice of GPU 140 to a corresponding subset of online game sessions and dynamically sharing the duty cycle of the processing slice among the two or more game sessions according to the real-time data processing requirements of the first and second game sessions (i.e., executing the two or more game sessions simultaneously and in parallel during the duty cycle period);

[0123] A multi-stream generation module 430 for encoding both the low-delay stream and the normal-delay stream in a time interval corresponding to a predefined frame rate;

[0124] A parallel encoding module 432 for allocating encoder cores of the encoder 142 to process image tiles of an image and enabling the encoder cores to encode individual blocks in the corresponding image tiles as soon as data of the corresponding blocks and their corresponding boundary information are available;

[0125] A dynamic frame generation module 434 for dynamically generating a variable number of frames during an online gaming session in response to a user command based on a predefined frame rate and corresponding communication and processing delays associated with the user command; and

[0126] ●A resource allocation module 436, which is used to determine target quality parameters of an online game session based on corresponding devices and connection capabilities, associate resources of the server system 114 (e.g., the virtual machines therein) with the online game session based on the target quality parameters, and manage the online game session according to a resource profile of the associated resources.

[0127] In some implementations, memory 406 also includes a data buffer 144 configured to couple encoder 142 to GPU 140. Specifically, data buffer 144 temporarily stores gameplay multimedia content generated by GPU 140 in association with one or more output media streams, such that encoder 142 can retrieve the gameplay multimedia content from data buffer 144 and encode the retrieved content into one or more media streams, for example, for normalization, speed, or compression.

[0128] Each of the elements identified above can be stored in one or more aforementioned memory devices, and corresponds to the instruction set for performing the above functions. The modules or programs (that is, instruction sets) identified above do not need to be implemented as independent software programs, processes, modules or data structures, and therefore the various subsets of these modules can be combined or otherwise rearranged in various implementations. In some embodiments, memory 406 optionally stores the modules identified above and the subset of data structures. In addition, memory 406 optionally stores additional modules and data structures not described above.

[0129] GPU Virtualization

[0130] Figure 5A is a block diagram of an example server system 114 including at least a GPU 140 according to some implementations. Figure 4, the server system 114 includes one or more processing units (e.g., CPU 402, GPU 140, and encoder 142), memory 406, and data buffer 144, which may be optionally part of or separate from the memory 406. The CPU 402, GPU 140, and encoder 142 function together to provide server-side functions associated with a game (e.g., game application 228), including, but not limited to, setting up a game session, storing game state data and other game-related data, processing gameplay input from a client device 102 or 104, and rendering gameplay output associated with the gameplay input. In some embodiments, the CPU 402 is configured to interact with the client device to provide a game controller configuration 456, facilitate setting up the game application 228, setting up a game session or a spectator commentary session, and / or receiving gameplay input. The GPU 140 is configured to obtain gameplay input and generate gameplay multimedia content, for example, by creating lighting effects and transforming objects each time a scene is regenerated. The encoder 142 is configured to encode (e.g., normalize or compress) the gameplay multimedia content provided by the GPU into one or more output media streams, each having a predetermined media format. Optionally, the encoder 142 is part of the GPU 140. Optionally, the encoder 142 is different from the GPU 140, and the data buffer 144 is coupled between the encoder 142 and the GPU 140. Specifically, the data buffer 144 temporarily stores the gameplay multimedia content generated by the GPU 140 in association with the one or more output media streams, so that the encoder 142 can retrieve the gameplay multimedia content from the data buffer 144 and encode the retrieved content into one or more media streams, for example, for normalization, speed or compression.

[0131] In some embodiments, the GPU 140 includes a plurality of GPU cores 500. In some embodiments, the GPU 140 interacts with one of an Open Graphics Library (OpenGL) platform, a Vulkan API platform, and a DX API platform. For example, the OpenGL platform includes a cross-language, cross-platform API for rendering 2D and 3D vector graphics and implementing hardware-accelerated rendering via the GPU 140. The OpenGL platform is implemented via at least a subset of a pipeline including the following operations: vertex processing (e.g., transforming vertices into screen space), primitive processing (e.g., organizing vertices into primitives that are clipped and culled), rasterization (e.g., rasterizing primitives into pixel fragments), fragment processing (e.g., shading fragments to calculate the color at each pixel), and pixel operations (e.g., blending fragments into a frame buffer at their pixel locations). In some embodiments, the OpenGL platform allows GPU 140 to provide gameplay multimedia content to encoder 142 that includes multiple pixels, and each pixel is associated with color information stored in data buffer 144, optionally in three components (e.g., RGB (green, red, blue)) or four components (e.g., CMYK (cyan, magenta, yellow, black)).

[0132] Figure 5B is a block diagram of one or more GPU kernels 500 that are partitioned into a plurality of processing time slices 502 according to some embodiments. The one or more GPU kernels 500 are Figure 5A . In some embodiments, the one or more GPU cores 500 include a single GPU core 500 that is configured to independently undertake one or more image rendering tasks (e.g., rendering a media stream for one or more online game sessions). In some embodiments, the one or more GPU cores 500 include two or more GPU cores 500 that are configured to collaboratively undertake one or more image rendering tasks. The one or more GPU cores 500 are divided into a plurality of processing slices 502. In some embodiments, each of the plurality of processing slices 502 lasts for a target frame rate corresponding to at least one of the plurality of online game sessions (e.g., a first online game session). For example, the target frame rate is 60fps, and each processing slice 502 lasts for 16.667ms. The first media stream is generated by the one or more GPU cores 500 for the first online game session, and includes image frames output during each of the corresponding processing slices 502 of the one or more GPU cores 500.

[0133] Specifically, in some embodiments, the server system 114 receives a request to initiate multiple online game sessions, and each session corresponds to a corresponding real-time user interactive application executed on one or more remote client devices. Each of the multiple processing slices 502 of the one or more GPU cores 500 is assigned to a subset of the multiple online game sessions to be executed thereon. For example, the first processing slice 502A is assigned to a first subset of online game sessions including the first game session GS1 and the second game session GS2. In some embodiments, each processing slice 502 is assigned to at least one identical game session (e.g., the first game session GS1), and at least one image is generated during each processing slice 502 for at least one identical game session. In some embodiments, the subsets of online game sessions assigned to two consecutive processing slices 502 are different from each other, that is, at least one online game session assigned to one of the two consecutive processing slices 502 is not assigned to the other of the consecutive processing slices 502. Further, in some embodiments, the online game session not assigned to one of the two consecutive processing slices 502 is assigned to the other of the consecutive processing slices 502.

[0134] In addition, at the first processing slice 502A, the server system 114 (e.g., the game server module 426) determines a time-sharing processing schedule for the first subset of online game sessions, so that each of the first subset of online game sessions is assigned a duty cycle of the first processing slice 502A. According to the time-sharing processing schedule, the server system 114 enables sharing of the first duty cycle of the first processing slice 502A by executing the first game session GS1 and the second game session GS2 in parallel according to the real-time data processing needs of the first and second game sessions GS1 and GS2. For example, the first processing slice 502A lasts for 16.667ms, and the first duty cycle is 50% of the duration of the first processing slice, for example, 8.333ms. In the example, during the first duty cycle, the first game session GS1 involves complex user actions (e.g., a fast sprint of a game character), while the second game session GS2 involves simple user actions (e.g., a slow walk of a game character). The first duty cycle dynamically reserves more resources (processing time or GPU cores) for the first game session GS1. Specifically, in some embodiments, during a first duty cycle of the first processing slice 502A, one or more GPU cores 502 execute the first game session GS1 and the second game session GS2 in parallel by allocating a corresponding subset of the one or more GPU cores 500 to each of the first and second game sessions GS1 and GS2 according to corresponding real-time data processing needs (for example, allocating a greater number of GPU cores 500 to the first game session GS1 involving more complex user actions than to the second game session involving simple user actions).

[0135] Optionally, in some embodiments, the first game session GS1 and the second game session GS2 alternate during a first duty cycle. For example, the first duty cycle is 50% of the duration of the first processing slice, for example, 8.333ms. One or more GPU cores execute each of the first game session and the second game session. Each of the first game session and the second game session executes for a short period of time (e.g., 1ms) before switching to the other of the first game session and the second game session.

[0136] In some embodiments, in order to assign each processing slice 502 to its corresponding subset of game sessions, the server system 114 identifies the game delay criteria of the corresponding slice 502 and provides a response that meets the game delay criteria to each of the subset of online game sessions corresponding to the corresponding processing slice 502. That is, at the first processing slice 502A, the duty cycle of the first processing slice 502A of each of the first subset of game sessions corresponds to the corresponding processing time that meets the game delay criteria. Specifically, in one example, the first processing slice 502A lasts for 16.667ms, and the first duty cycle is 50% of the duration of the first processing slice, for example, 8.333ms. The game delay criteria define a processing delay of at most 8.333ms. Each of the first subset of game sessions (e.g., the first and second game sessions GS1 and GS2) has a processing delay of less than 8.333ms.

[0137] Figure 5C-5E 5 are example processes 504, 506, and 508 for assigning two or more game sessions to the first processing slice 502A according to some implementations. Figure 5C In some embodiments, in addition to the first game session GS1 and the second game session GS2, the first processing slice 502A is also allocated to the third game session GS3. The first subset of the plurality of online game sessions allocated to the first processing slice 502A also includes the third game session GS3. According to the time-sharing processing schedule, the one or more GPU cores 500 facilitate dynamic sharing of the first duty cycle of the first processing slice 502A among the first, second, and third game sessions GS1-GS3. According to the real-time data processing needs of the third game session, the third game session GS3 is executed in parallel with the first and second game sessions GS1 and GS2. In addition, referring to Figure 5D In some embodiments, the first subset of online game sessions also includes a third game session GS3. Instead of executing the third game session in the first duty cycle, the one or more GPU cores 500 execute the third game session in a second duty cycle that follows but is different from the first duty cycle of the first processing slice 502A, and the third game session does not share the second duty cycle with any other game session.

[0138] refer to Figure 5E In some embodiments, the first subset of online game sessions assigned to the first processing slice 502A also includes a third game session GS3 and a fourth game session GS4. According to the time-sharing processing schedule, the third game session GS3 and the fourth game session GS4 dynamically share a second duty cycle that follows but is different from the first duty cycle of the first processing slice 502A. One or more GPU cores 500 execute the third and fourth game sessions GS3 and GS4 in parallel according to the real-time data processing needs of the third and fourth game sessions GS3 and GS4. For example, the first processing slice 502A lasts for 16.667ms and includes a first duty cycle and a subsequent second duty cycle. During the first duty cycle, two image frames correspond to the first and second game sessions GS1 and G2, respectively, and are dynamically processed according to the real-time data processing needs of the first and second game sessions GS1 and GS2. During the second duty cycle, two image frames correspond to the third and fourth game sessions GS3 and GS4, respectively, and are dynamically processed according to the real-time data processing needs of the third and fourth game sessions GS3 and GS4.

[0139] Figure 6 The present invention is an example process 600 of dynamically sharing a processing slice (e.g., a first processing slice 502A) by two game sessions (e.g., a first and a second game session GS1 and GS2) generating media content based on an OpenGL platform according to some embodiments. Each of the first game session GS1 and the second game session GS2 is executed based on the open graphics library and includes graphics pipeline processing. One or more GPU cores 500 execute the first game session GS1 and the second game session GS2 in parallel. Specifically, during a first duty cycle of the first processing slice 502A, while the first game session GS1 is on standby at the first pipeline stage, the one or more GPU cores 500 implement the second pipeline stage of the second online game session GS2. While the second online game session GS2 is on standby at the third pipeline stage, the one or more GPU cores 500 implement the fourth pipeline stage of the first online game session GS1. In some cases, the first and second online game sessions GS1 and GS2 are on standby at the first and third pipeline stages, respectively, because they are waiting for data required in the first and third pipeline stages to be prepared.

[0140] As described above, the OpenGL platform is implemented by at least a subset of the pipeline including the following operations: vertex processing (e.g., transforming vertices to screen space), primitive processing (e.g., organizing vertices into primitives that are clipped and culled), rasterization (e.g., rasterizing primitives into pixel fragments), fragment processing (e.g., shading fragments to calculate colors at each pixel), and pixel operations (e.g., blending fragments into a frame buffer at their pixel locations). In a specific example, the OpenGL platform implements nine consecutive pipeline stages to render an image frame for each online game session. The consecutive nine pipeline stages include a subset or all of the following:

[0141] ● Vertex specification 602, used to prepare vertex array data;

[0142] ●Vertex shader 604, used to act on each vertex;

[0143] = Subdivision surfaces 606 for optionally subdividing patches of vertex data into smaller primitives;

[0144] A geometry shader 608 for optionally converting smaller primitives into output primitives;

[0145] Vertex post-processing 610, which is used to adjust the output primitives of the final stage and transport them to different locations, including but not limited to transform feedback, primitive clipping, perspective segmentation, and viewport transformation;

[0146] = a primitive component 612 for dividing the output primitive into a sequence of individual basic primitives;

[0147] Rasterization 614 for decomposing each individual primitive into discrete elements (also called fragments) based on the primitive's sample coverage;

[0148] A fragment shader 616 for processing each fragment generated by rasterization into a set of colors and a single depth value; and

[0149] ●Pre-sampling operations 618 for processing color sets and individual depth values ​​to process gameplay multimedia content that can be processed by the encoder 142, for example, for normalization, speed and / or compression, including but not limited to pixel ownership testing, scissor testing, multi-sample operations, template testing, depth testing, occluded query updates, blending, sRGB conversion, dithering, logical operations, and write mask and logical operations.

[0150] The one or more GPU cores 500 execute the corresponding nine consecutive pipeline stages of the first and second game sessions GS1 and GS2 in parallel during the first duty cycle of the first processing slice 502A. Figure 6In this example shown in , one or more GPU cores 500 first implement stages 602 and 604 of a first game session GS1. While the first game session GS1 is paused in the middle of a vertex shader 604 (e.g., waiting for data to be ready), the one or more GPU cores 500 obtain vertex specifications 602 of a second game session GS2. The second game session GS2 is then paused at its vertex shader stage 604 (e.g., waiting for data to be ready), and the one or more GPU cores 500 continue to implement stages 604-610 of the first game session GS1. After completing the vertex post-processing stage 610 of the first game session GS1 and before starting the primitive assembly stage 612 of the first game session GS1, the one or more GPU cores 500 execute stages 604-610 of the second game session GS2. After completing the vertex post-processing stage 610 and before starting the primitive assembly stage 612 of the second game session GS2, the one or more GPU cores 500 continue to execute stages 612-618 of the first game session GS1. When the first game session GS1 is completed, the one or more GPU cores 500 continue to execute stages 612-618 of the second game session GS2.

[0151] exist Figure 6 In some embodiments not shown in FIG. 1 , one or more GPU cores 500 execute each of the nine consecutive pipeline stages of the first and second game sessions GS1 and GS2 in an alternating manner. Said another way, the one or more GPUs sequentially implement the vertex specification stage 602 of the first game session GS1, the vertex specification stage 602 of the second game session GS1, the vertex shader stage 604 of the first game session GS1, the vertex shader stage 604 of the second game session GS1, and so on until they complete the first and second game sessions.

[0152] In some embodiments, the first game session GS1 and the second game session GS2 are related to each other and are configured to share a portion of the application data for executing their respective real-time user interactive applications. For example, the first and second game sessions correspond to two players in a scene of the first real-time user interactive game application. The second game session GS2 can be initialized in response to an invitation from the first game session GS1. During the first duty cycle of the processing slice 502A, one or more GPU cores 500 can use the shared portion of the application data to implement both the first game session GS1 and the second game session GS2.

[0153] In some embodiments, the first game session and the second game session are independent of each other and are configured to execute two different real-time user interactive applications. For example, the first game session GS1 implements a Candy Crash game, while the second game session GS2 implements a Car Race game.

[0154] In some embodiments, the GPU 140 includes at least two or more control registers including a first control register and a second control register, and the first and second registers are respectively configured to control the first and second game sessions GS1 and GS2 in parallel when the first game session GS1 and the second game session GS2 dynamically share the first duty cycle of the first processing slice 502A. In addition, each of the first session GS1 and the second session GS2 is associated with a corresponding cache for temporarily storing the corresponding game session data during the processing. For example, referring to Figure 6 When the first game session GS1 is suspended during the vertex shader stage 604 and after the vertex post-processing stage 610, the game session data of the first game session GS1 is stored in its corresponding buffer, and the game session data is retrieved for further processing during the vertex shader stage 604 and after the first game session GS1 is reactivated at the primitive assembly stage 612.

[0155] In some implementations, vertex data includes context bits that are configured to be tagged based on corresponding game sessions (e.g., "job A" for a first game session and "job B" for a second game session). Data fragments are tagged in association with "job A" and "job B" throughout pipeline process 600. Optionally, hardware resources (e.g., control registers and caches) are also tagged.

[0156] Note that in some embodiments, the first and second game sessions GS1 and GS2 are implemented on a Vulkan API platform, and in some embodiments, the first and second game sessions GS1 and GS2 are implemented on a DX API platform. In some embodiments, the first game session GS1 and the second game session GS2 are implemented on two different platforms of a plurality of platforms (e.g., OpenGL, Vulkan, and DX). The consecutive pipeline stages of each of the first and second game sessions GS1 and GS2 correspond to the respective platforms on which the respective game sessions are implemented. The consecutive pipeline stages of the first game session GS1 and the second game session GS2 may be executed in parallel during the first duty cycle of the first processing slice 502A even when implemented on two different platforms.

[0157] Figure 7is a flowchart illustrating an example method 700 for managing the processing power of server system 114 (e.g., the processing power of GPU 140) according to some embodiments. Method 700 is optionally governed by instructions stored in a computer memory or non-transitory computer-readable storage medium (e.g., in GPU time-sharing module 422 of game server module 426) and executed by one or more processors of server system 114 (e.g., game server 122). The computer-readable storage medium may include a magnetic or optical disk storage device, a solid-state storage device such as flash memory, or other one or more non-volatile storage devices. The instructions stored on the computer-readable storage medium may include one or more of the following: source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Some operations in method 700 may be combined and / or the order of some operations may be changed.

[0158] Method 700 is performed by server system 114, which includes one or more processing cores (e.g., GPU 140, further including one or more GPU cores 500) and a memory storing programs executed by the processing cores. The one or more processing cores include multiple processing slices 502. In some embodiments, each of the multiple processing slices 502 lasts for a duration (e.g., 16ms) corresponding to a target frame rate (e.g., 60fps) for multiple online game sessions.

[0159] The server system 114 receives (702) a request to initiate a plurality of online game sessions, and each session corresponds to a corresponding real-time user interactive application executed on one or more remote client devices (e.g., client devices 102 or 104). The server system 114 allocates (704) each of a plurality of processing slices of one or more processing cores to a subset of the plurality of online game sessions to be executed thereon. Specifically, a first processing slice is allocated (706) to a first subset of online game sessions including a first game session GS1 and a second game session GS2. In some embodiments, the first game session GS1 and the second game session GS2 are related to each other and are configured to share a portion of application data for executing their respective real-time user interactive applications. In addition, in some embodiments, the first game session GS1 and the second game session GS2 correspond to two players in a scene of the first real-time user interactive game application. Optionally, the first game session GS1 and the second game session GS2 are independent of each other and are configured to execute two different real-time user interactive applications.

[0160] At the first processing slice, a time-sharing processing schedule for the first subset of online game sessions is determined (708) such that each of the first subset of game sessions is assigned a duty cycle of the first processing slice. In one example, the first processing slice lasts 16.667 ms, and the first duty cycle is 50% of the duration of the first processing slice, i.e., 8.333 ms. In some embodiments, the first duty cycle is continuous within the first processing slice 502A. Optionally, in some embodiments, the first duty cycle is discontinuous within the first processing slice 502A, and, for example, includes a plurality of first short slices (e.g., 1 ms long) interleaved with and separated by a plurality of second short slices (e.g., 1 ms long).

[0161] In some embodiments, for each of a plurality of processing slices, a game latency criterion is identified to provide a response delay that satisfies the game latency criterion to each of a subset of online game sessions corresponding to the corresponding processing slice. In addition, in some embodiments, at a first processing slice, a duty cycle of a first processing slice of each of a first subset of game sessions corresponds to a corresponding processing time that satisfies the game latency criterion. For example, the first processing slice 502A lasts 16.667ms, and the game latency criterion may require that the response delay must be less than 8ms. The corresponding processing time must be less than the response delay, i.e., 8ms, and therefore, the duty cycle of the first processing slice must be less than 48%. In some cases, a subset of one or more processing cores needs to be reallocated to the corresponding game session to meet the game latency criterion.

[0162] In some embodiments, the allocation of the duty cycle of a processing slice and a subset of one or more processing cores to a particular game session is implemented in a static manner. GPU 140 estimates the respective data processing requirements of each of the plurality of game sessions and allocates the duty cycle of a processing slice and a subset of one or more processing cores to the particular game session accordingly. Alternatively, in some embodiments, GPU 140 estimates the data processing requirements of a particular game session in real time (e.g., based on the complexity of a scene or user actions), and accordingly, dynamically allocates the duty cycle of a processing slice and a subset of one or more processing cores to the particular game session.

[0163] According to the time-sharing processing schedule, the first game session and the second game session share (710) the first duty cycle of the first processing slice 502A by executing the first and second game sessions GS1 and GS2 in parallel according to the real-time data processing needs of the first and second game sessions GS1 and GS2. In some embodiments, executing the first game session and the second game session in parallel also includes allocating a corresponding subset of one or more processing cores to each of the first game session GS1 and the second game session GS2 according to the corresponding data processing needs. In some embodiments, the one or more processing cores include at least two or more control registers, the at least two or more control registers also include a first control register and a second control register, and the first and second registers are configured to control the first and second game sessions GS1 and GS2 in parallel when the first game session GS1 and the second game session GS2 dynamically share the first duty cycle of the first processing slice 502A.

[0164] In some embodiments, each of the first game session GS1 and the second game session GS2 is executed based on an OpenGL platform and includes graphics pipeline processing. During a first duty cycle of the first processing slice 502A, the first online game session is on standby at a first pipeline stage (e.g., at a vertex shader stage 604), and a second pipeline stage (e.g., vertex specification 602) of the second online game session is implemented by one or more processing cores. In addition, while the second online game session is on standby at a third pipeline stage (e.g., vertex shader 604), a fourth pipeline stage (e.g., one of stages 606-610) of the first online game session GS1 is implemented by one or more processing cores. Reference above Figure 6 Discusses more details on dynamically executing first and second game sessions on OpenGL platforms.

[0165] In some embodiments, the first subset of online game sessions further includes a third game session GS3. According to the time-sharing processing schedule, the third game session GS3 also dynamically shares (712) the first duty cycle of the first processing slice with the first and second game sessions GS1 and GS2. According to the real-time data processing needs of the third game session, the third game session GS3 is executed in parallel with the first and second game sessions GS1 and GS2. In some embodiments, the first subset of online game sessions also includes the third game session GS3, and the third game session is executed in a second duty cycle following the first duty cycle of the first processing slice (714). The third game session GS3 does not share the second duty cycle with any other game session. In some embodiments, the first subset of online game sessions also includes a third game session GS3 and a fourth game session GS4. According to the time-sharing processing schedule, the third and fourth game sessions dynamically share (716) the second duty cycle of the first processing slice 502A by executing the third and fourth game sessions in parallel according to the real-time data processing needs of the third and fourth game sessions. The second duty cycle is different from the first duty cycle. Reference above Figure 5C-5E Discuss further details of executing one or more additional game sessions during the first processing slice.

[0166] In the example, GPU 140 processes four or more online game sessions. Two of the four or more online game sessions are processed by a dedicated GPU core without sharing the dedicated GPU core with any other online game session, for example, the first online game session uses the first GPU core and the second online game session uses the second GPU core and the third GPU core. According to the method 700 of managing processing power explained above with reference to Figures 5-7, the remaining two or more online game sessions share the fourth GPU core. Specifically, at least two of the remaining two or more online game sessions share the duty cycle of each processing slice of the fourth GPU core and execute in parallel according to their corresponding data processing needs.

[0167] It should be understood that the Figure 7 The specific order of operations in is exemplary only and is not intended to indicate that the described order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize various methods to manage the processing power of the server system as described herein.

[0168] Concurrent rendering of multiple media streams

[0169] Fig. 8A is a block diagram of an example server system 114 including at least an encoder 142 according to some implementations. Figure 8B is a time graph 820 of time intervals according to some embodiments. Figure 8C and 8D840 and 860 are example images of two different media streams displayed simultaneously at two separate client devices 102 in association with a game scene, respectively, according to some embodiments. Figure 4 , the server system 114 includes one or more processing units (e.g., CPU 402, GPU 140, and encoder 142), memory 406, and data buffer 144, which may be optionally part of or separate from the memory 406. The CPU 402, GPU 140, and encoder 142 function together to provide server-side functions associated with a game (e.g., game application 228), including, but not limited to, setting up a game session, storing game state data and other game-related data, processing gameplay input, and rendering gameplay output on the client device 102 and / or 104. In some embodiments, the CPU 402 is configured to interact with the client device to provide a game controller configuration 456, facilitate setting up the game application 228, setting up a game session or a spectator commentary session, and / or receiving gameplay input. The GPU 140 is configured to obtain game input and generate gameplay multimedia content, for example, by creating lighting effects and transforming objects each time a scene is regenerated.

[0170] The encoder 142 is configured to encode (e.g., normalize or compress) the gameplay multimedia content provided by the GPU 140 into one or more output media streams each having a predetermined media format. Media encoding can be performed in the encoder 142 based on any of various custom or standard codecs (e.g., H.265 / MPEG-H HEVC codec, H.264 / MPEG-4 AVC codec, H.263 / MPEG4 Part 2 codec, H.262 / MPEG-2 codec, Microsoft codec, and Google On2 codec (e.g., VP6, VP6-E, VP6-S, VP7, VP8, and VP9)) and / or any other suitable codec including codecs that have not been developed as of the archival data of this document. Optionally, the encoder 142 is part of the GPU 140. Optionally, the encoder 142 is different from the GPU 140, and the data buffer 144 is coupled between the encoder 142 and the GPU 140. Specifically, data buffer 144 temporarily stores the gameplay multimedia content generated by GPU 140 in association with one or more output media streams. Encoder 142 can retrieve the gameplay multimedia content from data buffer 144 and encode the retrieved content into one or more media streams for the purpose of standardizing the gameplay multimedia content, increasing the speed of the gameplay multimedia content, or compressing the gameplay multimedia content. In some embodiments, encoder 142 includes multiple encoder cores 800.

[0171] Specifically, the encoder 142 is configured to generate multiple media streams including a low-latency stream 130 and a normal-latency stream 132. The low-latency stream 130 corresponds to the online interactive game session 116 and has a faster response rate and lower transmission delay than the normal-latency stream 132. In some embodiments, the normal-latency stream 132 also corresponds to the online interactive game session 116. In some embodiments, the normal-latency stream 132 is independent of the low-latency stream 130 and can be a media stream that is not related to any gaming application. The server system 114 identifies a predefined frame rate (e.g., 60fps) in a sequence of image frames for encoding the low-latency stream 130. A first time interval (sometimes referred to as a processing slice) is determined to correspond to the predefined frame rate of each image frame in the first sequence of image frames of the low-latency stream 130. For example, for a predefined frame rate of 60fps, the first time interval is 16.667ms. Reference Figure 8B The online game session 116 includes a plurality of first time intervals, each of which is determined according to a predefined frame rate. During each first time interval, an image frame in a first sequence of image frames of the low-latency stream 130 is encoded so that the low-latency stream 130 can be output at the predefined frame rate.

[0172] After encoding the image frames of the low-delay stream 130, the server system 114 further determines whether there is remaining time left in the corresponding first time interval. In some cases, it is determined that the remaining time left in the first time interval meets the normal delay requirement associated with the normal delay stream 132, and a subset of the image frames of the second sequence of the normal delay stream are encoded during the remaining time of the first time interval. Conversely, in some cases, it is determined that the remaining time left in the first time interval does not meet the normal delay requirement, and the server system 114 suspends the generation of any image frames in the second sequence of image frames during the remaining time of the first time interval. In this way, encoding priority is given to the low-delay stream 130, and the normal delay stream 132 is generated according to the capabilities of the encoder 142 without guaranteeing a predefined frame rate.

[0173] In some embodiments, the low-latency stream 130 includes a first image frame, and a subset of the image frames of the second sequence of the normal-latency stream 132 corresponding to the first image frame includes a portion of the image frames in the normal-latency stream 132 as allowed by the normal-latency requirement. After encoding the first image frame, the server system 114 determines a first remaining time left in the first time interval. It is determined that the first remaining time left in the first time interval meets the normal-latency requirement associated with the normal-latency stream. The encoder 142 generates a portion of the image frame in the normal-latency stream during the remaining time of the first time interval. For example, the portion of the image frame includes half of the image frame in the normal-latency stream. According to the normal-latency requirement, the first remaining time must be sufficient to encode at least a quarter of the image frames of the normal-latency stream, which takes about 2 ms to encode. According to the normal-latency requirement, the first remaining time must be 2 ms or longer. In this example, the first remaining time is about 5 ms, so half of the image frames in the normal-latency stream will be encoded according to the normal-latency requirement.

[0174] In some embodiments, the low-delay stream 130 includes a second image frame, and a subset of the image frames of the second sequence of the normal delay stream 132 corresponding to the second image frame includes two or more image frames in the normal delay stream 132. After encoding the second image frame, the server system 114 determines the first remaining time left in the first time interval. It is determined that the first remaining time left in the first time interval meets the normal delay requirement associated with the normal delay stream. Thus, the encoder 142 generates two or more image frames in the normal delay stream during the remaining time of the first time interval. For example, the normal delay requirement may require that the first remaining time must be sufficient to encode an integer number of image frames of the normal delay stream. It takes about 2ms to encode each image frame of the normal delay stream. When the remaining time of the first time interval is 5ms, two image frames of the normal delay stream 132 are encoded in the remaining time of the first time interval.

[0175] In some embodiments, after the encoder 142 generates a subset of image frames of the second sequence of the normally delayed stream during the remaining time of the first time interval, the server system 114 determines a second remaining time remaining in the first time interval. It is determined that the second remaining time remaining in the first time interval satisfies a third requirement associated with a third media stream. The encoder 142 then generates a subset of the third media stream during the second remaining time of the first time interval. In some embodiments, the third media stream includes a player demonstration video stream that captures a game player who is participating in the online game session. The player demonstration video stream optionally provides real-time commentary created by the game player while the player is participating in the online interactive session 116.

[0176] refer to Figure 8CIn some embodiments, the low-latency stream 130 and the normal-latency stream 132 are associated with the same online game session 116. The low-latency stream 130 has a first resolution (e.g., 4K) and the normal-latency stream 132 has a second resolution (e.g., 1080p) lower than the first resolution. The low-latency stream 130 is sent to a first client device of a game player for enabling the game player to perform a plurality of interactive game actions (e.g., shooting, running, racing) during the online interactive game session 116. The normal-latency stream 132 is sent to one or more second client devices to allow one or more viewers to comment on the online interactive session 116 in real time. In some embodiments, the low-latency stream 130 is provided to a front-end server 134, which is configured to distribute the stream 130 to game players who subscribe to the game service via the front-end server 134. The normal-latency stream 132 is provided to a media streaming server 124, which is configured to broadcast the normal-latency stream 132 to viewers who have subscribed to the commenting service via the media streaming server 124. Optionally, the front-end server 134 and the media streaming server 124 are managed by an entity using a single gaming application. Optionally, the front-end server 134 and the media streaming server 124 are managed by two different entities using two different terminal applications, such as a gaming application (e.g., a FIFA soccer application) and a media streaming application (e.g., YouTube). Optionally, the front-end server 134 and the media streaming server 124 may also be managed by a single entity using two different terminal applications.

[0177] In some embodiments, each of the second client devices does not only display a normal delayed stream 132 that replicates the content displayed on the game player's first client device at a lower resolution, higher transmission delay, or slower response rate. Instead, additional information about the game player is also presented to the audience simultaneously with the content displayed on the game player's first client device. For example, when the game player is participating in an online game session 116, a camera is used to capture the game player's image data (i.e., a player demonstration video stream). The player demonstration video stream may include real-time comments created by the game player when the game player is participating in the online interactive session 116. The server system 114 obtains a player demonstration video stream that captures the game player. Optionally, the player demonstration video stream is merged into the normal delayed stream 132 sent to the second client device of the audience. Optionally, the player demonstration video stream is sent separately to the corresponding client device, but is played in the normal delayed stream 132. The player demonstration video stream is played to one or more viewers simultaneously with the normal delayed stream of the online interactive session 116. In some cases, the player demonstration video stream is played in the embedded window 862 of each second client device while the normal delayed stream 130 is displayed on the corresponding second client device.

[0178] Alternatively, in some embodiments, a player demonstration audio stream of a game player is recorded while the game player is participating in an online game session. Optionally, the player demonstration audio stream is merged into a normal delayed stream 132 sent to a second client device of the viewer. Optionally, the player demonstration audio stream is sent to each second device separately and played along with the normal delayed stream 132. The player demonstration audio stream is played to one or more viewers simultaneously with the normal delayed stream 132 of the online interactive session.

[0179] In some embodiments, a predefined frame rate for encoding the first sequence of image frames of the low-latency stream 130 is determined based on a first threshold delay. Further, in some embodiments, the normal-latency stream 130 has a second delay time that is less than a second threshold delay, and the second threshold delay is greater than the first threshold delay. In particular, for example, the normal-latency stream 132 provided to viewers is not as sensitive to delay as the low-latency stream 130 provided to game players. In some embodiments, both the low-latency stream 130 and the normal-latency stream 132 correspond to the same online interactive session 116. The low-latency stream 130 and the normal-latency stream 132 have a first response rate and a second response rate, respectively, for rendering results of player actions by a game player associated with the online interactive session 116. The first response rate of the low-latency stream 130 is greater than the second response rate of the normal-latency stream 132 because game players are more sensitive to delays for rendering results of player actions than viewers.

[0180] Fig. 9 is a flow chart illustrating an example method 900 for providing media streams simultaneously according to some embodiments. The method 900 is optionally implemented by a computer stored in a computer memory or a non-transitory computer readable storage medium (e.g., Figure 4 The computer-readable storage medium may include a magnetic or optical disk storage device, a solid-state storage device such as a flash memory, or one or more other non-volatile storage devices. The instructions stored on the computer-readable storage medium may include one or more of the following: source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Some operations in method 900 may be combined and / or the order of some operations may be changed.

[0181] The method 900 is performed by a server system 114, which includes one or more processing cores (e.g., an encoder 142, further including one or more encoder cores 800) and a memory storing programs executed by the processing cores. The server system 114 generates (902) a plurality of media streams including a low-delay stream 130 and a normal-delay stream 132 by the encoder 142. The low-delay stream 130 corresponds to an online interactive session and has a faster response rate and lower transmission delay than the normal-delay stream 132. Specifically, the server system 114 identifies (904) a predefined frame rate for encoding a sequence of image frames of the low-delay stream. For each image frame in the first sequence of image frames of the low-delay stream, the server system 114 determines (906) a first time interval corresponding to the predefined frame rate. The encoder 142 encodes the image frame during the first time interval (908).

[0182] After encoding the image frame, the server system determines (910) a remaining time left in the first time interval. The server system 114 (in some embodiments, the encoder 142 itself) determines whether the remaining time left in the first time interval satisfies a normal delay requirement associated with the normal delay stream. In some cases, based on determining that the remaining time left in the first time interval satisfies the normal delay requirement, the encoder 142 generates (912) a subset of the image frames of the second sequence of the normal delay stream during the remaining time of the first time interval. Alternatively, in some cases, based on determining that the remaining time left in the first time interval does not satisfy the normal delay requirement, the encoder 142 suspends (914) generating any image frames of the second sequence of image frames during the remaining time of the first time interval.

[0183] More details of the method 900 for providing media streams simultaneously are discussed above with reference to FIG8. Fig. 9 The specific order of operations in is exemplary only and is not intended to indicate that the described order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize various methods to manage the processing power of the server system as described herein.

[0184] In this application, a method 900 for providing concurrent media streams grants encoding priority to a low-latency stream that requires a higher response rate and lower transmission delay than one or more additional streams (e.g., a normal delay stream). During each of a series of time intervals, the image of the low-latency stream is encoded first, and the remaining time left in the time interval is not wasted, but is used to encode at least a portion of one or more additional media streams. In some embodiments, the server system 114 configures the encoder 142 to encode the low-latency stream in an efficient manner so that more remaining time can be left in each time interval to encode one or more additional media streams. For example, once the information required for encoding a portion of an image frame is provided by the GPU 140, the encoder 142 of the server system 114 can accelerate the encoding of the image frame of the low-latency stream during the corresponding time interval by encoding the portion, regardless of whether other portions of the image frame that are not related to the encoded portion are available by the GPU 140. In this way, the encoder 142 of the server system 114 does not need to wait until the GPU 140 makes the data of the entire image frame available, thereby accelerating the encoding of the image frames of the low-latency stream and saving more time in the corresponding time interval to encode one or more additional streams different from the low-latency stream.

[0185] Multi-core parallel encoding

[0186] Fig. 10A is a block diagram of an encoder core 800 assigned to process a plurality of image tiles 1000 of an image frame in a media stream, according to some embodiments. Fig. 10B is an example process 1040 for encoding a block in an image tile according to some embodiments. Figure 4 and Fig. 8A , the server system 114 includes an encoder 142 coupled to the GPU 140 and configured to encode (e.g., standardize or compress) the multimedia content provided by the GPU into one or more output media streams each having a predetermined media format. The multimedia content provided by the GPU 140 includes a media stream consisting of a series of image frames. Typically, each image frame needs to be prepared by the GPU 140 and stored in its entirety in a data buffer 144, so that the encoder 142 can retrieve the entire image frame from the data buffer 144 for further encoding. According to some embodiments of the present application, the encoder 142 is configured to encode a portion of the image frame when the GPU 140 provides the information required for encoding the portion, regardless of whether other portions of the image frame that are not required for encoding the portion can be made available or unavailable by the GPU 140.

[0187] The encoder 142 includes a plurality of encoder cores 800. The encoder core 800 of the server system 114 identifies frame data of an image frame associated with an online game session and stored in a data buffer 144. The frame data has a first resolution. Then, the plurality of encoder cores 800 are allocated to simultaneously and in parallel process a plurality of image tiles 1000 of the frame data, and the plurality of image tiles 1000 are defined according to a rendering specification of the image frame. The plurality of image tiles 1000 include a first image tile 1000A assigned to a first encoder core 800A, and the first image tile 1000A includes a first sequence of blocks. The encoder core 800 (e.g., the first encoder core 800A) begins encoding the image frame based on a determination of the availability of a portion of the frame data, rather than waiting to begin encoding the image frame until the frame data of the entire image frame is available.

[0188] Specifically, in some embodiments, the first encoder core 800A obtains a first block 1002 of a first sequence of blocks of a first image tile 1000A assigned to the first encoder core 800A from the data buffer 144. The first encoder core 800A also obtains boundary information of the first block 1002. While or before obtaining a second block 1004 following the first block in the first sequence of blocks from the data buffer 144, the first encoder core 1000A encodes the first block 1002 of the first sequence of blocks based on the boundary information. The encoded first block 1002 is transmitted to a client device of a user associated with an online game session. Optionally, the second block 1004 is directly adjacent to the first block in the image frame. Optionally, the second block 1004 is not directly adjacent to the first block in the image frame.

[0189] In some embodiments, the encoded first block 1002 is temporarily stored in a frame buffer 1006 of the client device before being displayed on the client device. Here, the second block 1004 is not involved in the operation of encoding the first block 1002, and therefore, the first block 1002 can be encoded as long as the first block and the corresponding boundary information are available and regardless of whether the second block 1004 is available in the data buffer 144 (i.e., the second block 1004 may be already available in the data buffer 144 or not yet provided by the GPU 140). In some cases, although the first block is obtained from the data buffer 144, the second block has not yet been stored or prepared in the data buffer 144 and is loaded into the data buffer 144 by the GPU. In addition, in some cases, when the first block is encoded in the first encoder kernel 800A, the second block is already available in the data buffer 144, and the third block is loaded into the data buffer 144 by the GPU 140. The third block is then encoded after the first block 1002 and the second block 1004 are encoded.

[0190] In some embodiments, the boundary information of the first block 1002 is provided by a first subset of blocks in the first image tile 1000A. Optionally, each of the first subset of blocks precedes or follows the first block 1002 in the first sequence of blocks. In addition, in some embodiments, the first image tile 1000A includes a second subset of blocks that do not share any blocks with the first subset of blocks, and the first block 1002 is encoded independently of whether the first encoder kernel 800A has obtained or encoded any of the second subset of blocks. Optionally, the second subset of blocks includes a second block 1004.

[0191] In some embodiments, the server system 114 determines that the boundary information of the first block 1002 is not provided by a third block of a second image tile that is different from the first image tile and is assigned to the corresponding encoder core 800. The first block 1002 is encoded independently of whether the corresponding encoder core assigned to process the second image tile has received or encoded the third block. Said another way, when the first block 1002 is encoded by the first encoder core 800A, the third block may have been encoded by the corresponding encoder core, not yet stored and available in the data buffer 144, or is waiting in the data buffer 144 to be encoded in the corresponding encoder core.

[0192] In some embodiments, each image tile 1000 of the image frame includes a corresponding sequence of blocks, and each block is composed of one or more macroblocks, each macroblock being a basic processing unit of a corresponding image and video compression format based on a linear block transform (e.g., discrete cosine transform (DCT)). That is, each of the blocks of the first sequence in the first image tile includes multiple macroblocks (e.g., a single macroblock, two or more macroblocks). In the example, the macroblock is composed of 16×16 pixels and is further subdivided into transform blocks (e.g., 16×16 luminance (Y) blocks and 8×8 chrominance (Cb and Cr) blocks) or prediction blocks (e.g., 4×4 or 16×16 blocks for inter-frame prediction or intra-frame prediction).

[0193] The first block 1002 needs to be stored in the data buffer 144 until it is encoded, or until any block using the first block 1002 as boundary information is encoded. In other words, the first block 1002 can be cleared from the data buffer 144 already after the first block 1002 itself and any block using the first block 1002 as boundary information are encoded. Therefore, the data buffer 144 is reserved for storing other blocks in the image frame and does not need to have a size that can accommodate all blocks in the image frame. As a result, the size of the data buffer 144 can be reduced while still encoding the image frame in the encoder 142 in an efficient manner (mainly because the encoder 142 no longer needs frame data to be available for the entire image frame).

[0194] Figures 11A-11C 1 are example image tiles 1100, 1120, and 1140 defined according to a rendering specification of frame data of an image frame associated with an online interactive session according to some embodiments. As explained above, the frame data of the image frame may be segmented into a plurality of image tiles according to the rendering specification. Fig.11A In some embodiments, according to the first rendering specification, the frame data of the image frame is divided into a single row of image tiles 1100. Each image tile 1100 includes a plurality of pixel columns extending from the top to the bottom in the image frame. In a specific example, media encoding is performed in the encoder 142 based on the standard VP8 codec, and according to the VP8 codec, the first rendering specification defines that each image frame encoded by the encoder 142 is divided into eight tiles, which are arranged in a single row and span the entire height of the image frame. In some embodiments, the basic processing unit of the encoder 142 includes one or more macroblocks 1102. Each of the tiles 1100 includes a first number of macroblock columns and a second number of macroblock rows of the image frame. Each tile 1100 extends from the top to the bottom of the image frame, so the second number of macroblock rows of the corresponding image tile 1100 is equal to the number of macroblock rows in the image frame.

[0195] refer to Fig. 11B In some embodiments, according to the second rendering specification, the frame data of the image frame is divided into a single column of image tiles 1120. Each image tile 1100 includes a plurality of pixel rows extending from the left edge to the right edge of the image frame. In some embodiments, the basic processing unit of the encoder 142 includes one or more macroblocks 1102, and according to the second rendering specification, the frame data of the image frame is divided into a single column of tiles 1120. Each of the tiles 1100 includes a first number of macroblock columns and a second number of macroblock rows of the image frame. Each tile 1100 extends from the left side to the right side of the image frame, so the first number of macroblock columns of the corresponding image tile 1100 is equal to the number of macroblock columns in the image frame.

[0196] refer to Fig. 11C In some embodiments, according to the third rendering specification, the frame data of the image frame is divided into an array of image tiles 1140. The array of image tiles 1140 includes two or more rows of image tiles and two or more columns of image tiles (e.g., two rows and five columns of image tiles).

[0197] In some embodiments, the number of tiles segmented from the frame data is equal to the number of encoder cores 800 in the encoder 142. In the example, the image frame includes eight tiles, and the encoder 142 includes eight encoder cores 800. In some cases, each of the plurality of encoder cores 800 may be assigned to process a different tile of the plurality of tiles 1100 segmented from the frame data of the image frame. Alternatively, in some cases, the encoder core 800A may be assigned to process two or more tiles (e.g., Fig.11A 1100A and 1100C, Fig. 11B 1120A and 1120C in ), without assigning one of the multiple encoder cores 800 to process any tile.

[0198] In some embodiments, the number of tiles segmented from the frame data may be greater than the number of encoder cores 800 in the encoder 142. At least one encoder core 800 may be assigned to process two or more tiles (e.g., Fig.11A 1100A and 1100C, Fig. 11B In some cases, even if the number of image tiles is greater than the number of cores, none of the multiple encoder cores 800 may be assigned to process any tile.

[0199] Furthermore, in some embodiments, the number of tiles segmented from the frame data may be less than the number of encoder cores 800 in the encoder 142. In some cases, each of the subset of encoder cores 800 is assigned to process a different tile of the plurality of tiles segmented from the frame data of the image frame. At least one encoder core 800 is not assigned to process any tile, while in some cases, encoder core 800A is still assigned to process two or more tiles (e.g., Fig.11A 1100A and 1100C, Fig. 11B 1120A and 1120C).

[0200] In some embodiments, the number of tiles segmented from the frame data is independent of the number of encoder cores 800 in the encoder 142. Alternatively, in some embodiments, the rendering specification is associated with the number of encoder cores 800 in the encoder 142, and the number of image tiles is determined according to the number of encoder cores 800 in the encoder 142.

[0201] In some embodiments, the boundary information required for encoding the first block 1002A of the first image tile 1000A includes one or more blocks ( Fig.11A). Optionally, each of the one or more blocks is directly adjacent to the first block 1002A, or is located at any position in the first image tile 1000A. In the example, the one or more blocks include all eight blocks surrounding the first block 1002, including the upper left block A, the upper block B, the upper right block C, the left block D, the right block E, the lower left block F, the lower block G, and the lower right block H. In another example, the one or more blocks only include adjacent blocks, that is, the upper block B, the left block D, the right block E, and the lower block G. After receiving the information of the one or more blocks from the data buffer 144, the first block 1002A is encoded. At the same time or after the first block 1002A is encoded, the second block 1004A is obtained from the data buffer 144, so the second block 1004 is also obtained after the one or more blocks are obtained as the boundary information of the first block 1002A.

[0202] In addition, in some embodiments, the boundary information required for encoding the first block 1002B of the first image tile 1000A includes one or more blocks. Because the first block 1002B is close to the boundary of the first image tile 1000A, the one or more blocks used to encode the first block 1002B belong to two or more image tiles. Optionally, each of the one or more blocks is directly adjacent to the first block 1002A, or is located at any position in one of the two or more image tiles. In the example, the one or more blocks include all eight blocks surrounding the first block 1002B, including the upper left block A', the upper block B', the upper right block C', the left block D', the right block E', the lower left block F', the lower block G', and the lower right block H'. Reference Fig.11A , the one or more blocks include a first subset of blocks (eg, blocks A', B', D', F', and G') of the first image tile 1100A and a second subset of blocks (ie, C', E', and H') of the second image tile 1100B. Fig. 11C , the boundary information required to encode the first block 1002B of the third image tile 1140C includes one or more blocks, and the one or more blocks also include four subsets of blocks. Each of the four subsets of blocks belongs to the first, second, third and fourth image tiles (i.e., blocks A' and B' in the first image tile 1140A; block C' in the second image tile 1140B; blocks D', F' and G' in the third image tile 1140C; and blocks E' and H' in the fourth image tile 1140D).

[0203] In another example, one or more blocks corresponding to the first block 1002 include only adjacent blocks, ie, an upper block B', a left block D', a right block E', and a lower block G'. Fig.11A , the one or more blocks include a first subset of blocks (eg, blocks B', D', and G') of the first image tile 1100A and a second subset of blocks (ie, E') of the second image tile 1100B. Fig. 11C , the boundary information required for encoding the first block 1002B of the third image tile 1140C includes one or more blocks, and the one or more blocks further include three subsets of blocks. Each of the three subsets of blocks belongs to the first, third and fourth image tiles respectively (i.e., block B' in the first image tile 1140A, blocks D' and G' in the third image tile 1140C, and block E' in the fourth image tile 1140D). Before the first block 1002B can be encoded by the corresponding encoder kernel, the first block 1002B still needs to wait for information of one or more blocks to be received from the data buffer 144. While or after the first block 1002B is encoded, the second block 1004B is obtained from the data buffer 144, so after obtaining one or more blocks as boundary information of the first block 1002B, the second block 1004B is also obtained.

[0204] Figures 12A-12C 1200, 1220, and 1240 are example raster scans of frame data of an image frame associated with an online interactive session according to some embodiments. The GPU 140 generates each picture element (e.g., pixel) of the image frame according to a predetermined raster order and stores data corresponding to the corresponding picture element in a data buffer 144, thereby making the data of the corresponding picture element available to the encoder 142 according to the predetermined raster order. When the data of each picture element is available, that is, according to the predetermined raster order, the encoder 142 thereby extracts the frame data of the image frame by raster scanning of the data buffer. As described above, each encoder core 800 is configured to sequentially encode blocks of a corresponding image tile, and for example, the first encoder core 800A obtains and encodes a series of blocks (e.g., a first block 1002 and a second block 1004) of a corresponding first image tile 1000A. Specifically, each block of the image tile is encoded when the GPU 140 provides the information required for encoding the block, regardless of whether other blocks of the image frame not required for encoding the block are available in the data buffer. That is, in the example, the frame data includes data of picture elements (e.g., pixels) corresponding to the first block 1002 and the second block 1004, and the first block 1002 and the second block 1004 are obtained while the data of the picture elements corresponding to the first block 1002 and the second block 1004 are extracted according to a predetermined raster order.

[0205] refer to Fig. 12A, according to the first predetermined raster order, the GPU 140 generates each pixel of the image frame row by row, and stores the data corresponding to the corresponding pixel in the data buffer 144 row by row. When the frame data of the image frame becomes available in the data buffer 144, they are also extracted row by row from the data buffer 144 according to the first predetermined raster order. Each row of the image frame includes data from two or more image tiles. For example, the frame data of the image frame is divided into a single row of image tiles 1100 (for example, including 5 image tiles 1202-1210). The pixel data on each odd row of the image frame is made available from the left side to the right side of the image frame, that is, for the image tiles 1202, 1204, 1206, 1208 and 1210 are available in sequence. On the contrary, the pixel data on each even row of the image frame is available in the opposite order or from the right side to the right side of the image frame, that is, for the image tiles 1210, 1208, 1206, 1204 and 1202 are available in sequence.

[0206] In some embodiments, to encode the first block 1002, the first encoder core 800A obtains the first block 1002 and corresponding boundary information including one or more blocks AH. Optionally, a first subset or all of the one or more blocks AH belong to the same first image tile 1202, which includes the first block 1002. Optionally, a second subset of the one or more blocks AH (e.g., Fig. 12A The blocks C, E and H in FIG. 1 belong to a second image tile 1204 that is different from the first image tile 1202. Each of the first block 1002 and the one or more blocks AH includes one or more picture elements, for example, 4, 16 or 256 pixels. Fig. 12A In the example of , the first encoder core 800A has to obtain data of at least all image elements of the first block 1002 and blocks AF before encoding the first block 1002. The first encoder core 800A obtains the first block 1002 after extracting the first four lines of the image frame from the data buffer 144. However, the first encoder core 800A does not immediately encode the first block 1002. Instead, the raster scan of the data buffer 144 continues, and when the raster scan reaches the sixth line of the image frame, the first encoder core 800A starts encoding the first block 1002 after it completely obtains the data of block F.

[0207] As more and more rows of picture elements become available and are extracted from the data buffer 144, the encoder encoder core 800 of the encoder 142 begins encoding blocks in their corresponding image tiles in parallel from the top to the bottom of the image frame without waiting for the frame data for the entire image frame to be available. This parallel encoding mechanism shortens the overall waiting time of the encoder 142 and reduces the encoding time required for the image frame. In an example, the encoder 142 identifies a predefined frame rate (e.g., 60fps) for encoding a sequence of image frames of a low-latency stream and determines a first time interval (e.g., 16.67ms) corresponding to the predefined frame rate. During each first time interval, if the encoding time required for the image frame is reduced, the remaining time left in the first time interval can be extended to encode a subset of the image frames of a second sequence of normal-latency streams, thereby allowing concurrent encoding of two or more media streams, for example, with Figure 8C The game scene shown in FIG. 1 is associated with two different media streams displayed simultaneously on two separate client devices.

[0208] refer to Fig. 12B , according to the second predetermined raster order, GPU 140 sequentially generates image tiles 1202-1210 of the image frame, for example, generating the first image tile 1202 before the second image tile 1204 adjacent to the first image tile 1202. For each image tile, GPU 140 generates the pixels in the corresponding image tile row by row, and stores the data corresponding to the pixels in the corresponding image tile row by row in data buffer 144 until all the pixels of the corresponding image tile pixels are generated and stored. When the frame data of the image frame becomes available in the data buffer 144, they are also extracted from the data buffer 144 by encoder 142 according to the second predetermined raster order. Specifically, encoder 142 sequentially extracts the image tiles 1202-1210 of the frame data of the image frame, and for each image tile, the pixels of each image tile row by row after GPU 140 makes them available. For example, when GPU 140 makes data for pixels in first image tile 1202 available in data buffer 144 in a zigzag raster order from top to bottom, the data for the pixels is fetched from data buffer 144 in the same zigzag raster order from top to bottom by corresponding encoder core 800A. As a result, after data for the entire first image tile 1202 becomes available and is fetched, GPU 140 continues to sequentially generate and store data for pixels in each of the other image tiles 1204-1210 in a zigzag raster order from top to bottom, and encoder 142 follows GPU 140 to fetch data for pixels in the corresponding image tiles from data buffer 144 in a zigzag raster order from top to bottom.

[0209] exist Fig. 12BIn some embodiments shown in , the first encoder kernel 800A obtains the first block 1002 and corresponding boundary information including one or more blocks AH. The first encoder kernel 800A has to obtain data of at least all image elements of the first block 1002 and blocks AF before encoding the first block 1002. Optionally, a second subset of the one or more blocks AH (e.g., Fig. 12B 1204, which is different from but adjacent to the first image tile 1202. The first encoder kernel 800A obtains the first block 1002 after the first encoder kernel 800A extracts the first four rows of the first image tile 1202 from the data buffer 144. However, the first encoder kernel 800A does not immediately encode the first block 1002. Instead, the raster scan of the data buffer 144 continues, and when the raster scan reaches the second image tile 1204 of the sixth row, the first encoder kernel 800A starts encoding the first block 1002 after it completely obtains the data of block F.

[0210] In some embodiments not shown in the present application, the first image tile 1202 includes the first block 1002 and one or more blocks AH. After the first encoder kernel 800 completely obtains the picture elements of the blocks AH and the first block 1002 within the first image tile, the first encoder kernel 800 encodes the first block 1002. To encode the first block 1002, the raster scan is limited to the first image tile 1202 independent of the other image tiles 1204-1210, for example, the first encoder kernel 800A does not need to wait until the raster scan is completed for all blocks in the first image tile 1202 or reaches any block in the second image tile 1204.

[0211] As more and more image tiles and rows of picture elements in each tile become available and are extracted from the data buffer 144, the encoder core 800 of the encoder 142 begins encoding blocks from top to bottom within their corresponding image tiles without having to wait for frame data for the entire image frame to be available. Fig. 12A The first predetermined raster order shown in allows the assigned encoder cores processing the image tiles to complete encoding of the corresponding image tiles at completion times substantially close to each other. Fig. 12BThe second predetermined raster order shown in allows the first encoder core 800A to start and complete encoding the first image tile 1202 earlier than other encoder cores 800 that are assigned to process image tiles available after the first image tile 1202. After encoding the blocks in the first image tile 1202, the first encoder core 800A can be assigned to encode another image tile (e.g., image tile 1210) in the same image frame or a subset of a different image frame, thereby allowing image frames of two or more media streams to be encoded simultaneously, for example, as shown in FIG. Figure 8C The game scene shown in FIG. 1 is associated with two different media streams displayed simultaneously on two separate client devices.

[0212] refer to Fig. 12C , each image tile 1202-1210 in the image frame includes a plurality of segments, and each segment includes a first number of rows. According to a third predetermined raster order, the GPU 140 generates the segments in the image tiles 1202-1210 row by row, for example, generating the first segment 1212 of the first image tile 1202 before the first segment 1214 of the second image tile 1204. For each segment of each image tile, the GPU 140 generates the first number of pixel rows in the corresponding segment row by row, and stores the data corresponding to the pixels in the corresponding segment of the image tile row by row in the data buffer 144 until all pixels of the corresponding segment are generated and stored. When the frame data of each segment of the image frame becomes available in the data buffer 144, they are also extracted from the data buffer 144 by the encoder 142 according to the second predetermined raster order.

[0213] Specifically, the encoder 142 sequentially extracts the segments of the image tiles 1202-1210 of the frame data of the image frame row by row. For each segment in each image tile, the encoder 142 extracts the pixels of each segment row by row in a zigzag raster order after the GPU 140 makes the pixels of each segment available. For example, while the pixel data in the first segment 1212 of the first image tile 1202 is made available in the data buffer 144 in a zigzag raster order from top to bottom by the GPU 140, the corresponding encoder core 800A extracts the pixel data from the data buffer 144 in the same zigzag raster order from top to bottom. As a result, after having made data for first segment 1212 of first image tile 1202 available and fetched, GPU 140 continues to generate and store pixel data in first fragment 1214 of second image tile 1204 in a zigzag raster order from top to bottom, and encoder 142 follows GPU 140 to fetch pixel data in first segment 1214 of second image tile 1204 from data buffer 144. After all first segments of image tiles 1202-1210, second segment 1216 of first image tile 1202 following first segment 1212 becomes available and fetched.

[0214] In a way, the third predetermined raster order is Fig. 12A and 12B It should also be noted that the zigzag raster order is only an example of an internal raster order applied to scan pixels in each segment of the image tiles 1202-1210, and the internal raster order is not limited to the zigzag raster order.

[0215] exist Fig. 12C In some embodiments shown in , the first encoder kernel 800A obtains the first block 1002 and corresponding boundary information including one or more blocks AH. The first encoder kernel 800A has to obtain data of at least all picture elements of the first block 1002 and blocks AF before encoding the first block 1002. Optionally, a second subset of the one or more blocks AH (e.g., Fig. 12B1204, which is different from but adjacent to the first image tile 1202. After the first encoder kernel 800A extracts the fourth segment of the first image tile 1202 from the data buffer 144, the first encoder kernel 800A obtains the first block 1002. However, the first encoder kernel 800A does not immediately encode the first block 1002. Instead, the raster scan of the data buffer 144 continues, and when the raster scan reaches the fifth segment of the second image tile 1204, the first encoder kernel 800A starts encoding the first block 1002 after it has completely obtained the data of block H.

[0216] Fig.13 1 is a flowchart illustrating an example method 1300 for encoding a media stream according to some embodiments. The method 1300 is optionally implemented by a computer stored in a computer memory or a non-transitory computer readable storage medium (e.g., Figure 4 The computer-readable storage medium may include a magnetic or optical disk storage device, a solid-state storage device such as a flash memory, or other one or more non-volatile storage devices. The instructions stored on the computer-readable storage medium may include one or more of the following: source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Some operations in method 1300 may be combined and / or the order of some operations may be changed.

[0217] The method 1300 is performed by the server system 114 to encode a media stream. The server system 114 includes a GPU 140, a plurality of encoder cores 800, a data buffer 144, and a memory storing programs executed by the GPU 140 and the encoder cores 800. When at least a portion of the frame data is made available, the encoder core 800 identifies (1302) at least a portion of the frame data of an image frame associated with the online game session and stored in the data buffer 144. Optionally, the frame data has a first resolution.

[0218] Multiple encoder cores 800 are allocated (1304) to process multiple image tiles of frame data simultaneously and in parallel. Multiple image tiles are defined according to a rendering specification. In some embodiments, according to the rendering specification, the frame data of the image frame is divided into a row of image tiles. For example, the rendering specification definition corresponding to VP8 divides each image frame into eight tiles arranged in a single row, and each tile extends from top to bottom in the corresponding image frame. In other words, each image tile includes multiple macroblock columns of the image frame and multiple macroblock rows of each image tile, and the number of macroblock rows of each image tile is equal to the number of macroblock rows of the image frame. Each macroblock is a processing unit of a corresponding image and video compression format based on a linear block transform, and optionally includes an array of 16×16 pixels or 64×64 pixels. In some embodiments, the multiple image tiles of the frame data include multiple image tiles, and the number of image tiles is determined according to the number of processing cores in the multiple encoder cores.

[0219] The plurality of image tiles include (1306) a first image tile (e.g., image tiles 1000A, 1100A, 1120A, 1140A, and 1202) assigned to a first encoder kernel 800A. In some embodiments, the first encoder kernel 800A is assigned to process a third image tile that is not adjacent to the first image tile after encoding the first image tile. The first image tile includes a first sequence of blocks, and each of the first sequence of blocks includes a plurality (e.g., 1, 2, and 4) of macroblocks.

[0220] The first encoder kernel obtains (1308) from the data buffer a first block 1002 of a first sequence of blocks of a first image tile assigned to the first encoder kernel. The first encoder kernel obtains (1310) boundary information for the first block. In some embodiments, the boundary information for the first block is provided by a first subset of blocks in the first image tile. In addition, the first image tile includes a second subset of blocks that do not share any blocks with the first subset of blocks, and the first block is encoded independently of whether the first encoder kernel has obtained or encoded any of the second subset of blocks. In some embodiments, the first encoder kernel determines that the boundary information for the first block is not provided by a third block of a second image tile different from the first image tile, and encodes the first block independently of whether a corresponding encoder kernel assigned to process the second image tile has received or encoded the third block.

[0221] The first encoder kernel encodes (1312) the first block 1002 in the first sequence of blocks based on the boundary information while or before acquiring the second block 1004 following the first block in the first sequence of blocks from the data buffer 144. The second block is required in the boundary information. Optionally, the second block is directly adjacent to the first block. Optionally, the second block is not directly adjacent to the first block. In some embodiments, the first block is encoded independently of whether the first encoder kernel completes acquiring any other encoded blocks following the first block in the first sequence of blocks of the first image tile. In some cases, the second block is prepared by the GPU 140 and loaded into the data buffer 144, while the first block is obtained from the data buffer by the first encoder kernel.

[0222] In addition, in some embodiments, the first image tile further includes a third block. While the first encoder core 800A is encoding the first block, the third block is loaded into the data buffer 144 by the GPU 140. The third block is encoded after the encoding of the first and second blocks.

[0223] The encoded first block 1002 is then transmitted 1314 to a client device of a user associated with the online gaming session. In some implementations, the encoded first block 1002 is transmitted after all blocks of the image frame have been encoded. In some implementations, the encoded first block 1002 is transmitted after a subset of blocks in the image frame have been encoded.

[0224] In some embodiments, the encoder 142 extracts frame data of the image frame by raster scanning of the data buffer 144 according to a predetermined raster order. The frame data includes data of pixels corresponding to the first and second blocks, and the first and second blocks are obtained while the data of pixels corresponding to the first and second blocks are extracted according to the predetermined raster order. In addition, in some embodiments, the frame data of the image frame is extracted row by row from the image frame according to the predetermined raster order, and each row of the image frame includes data from two or more image tiles. Alternatively, in some embodiments, when data from each image tile is extracted by corresponding raster scanning, the frame data of the image frame is also extracted, and a plurality of image tiles are sequentially scanned to extract their corresponding data. In addition, the predetermined raster order includes an internal raster order, and the raster scanning within each image tile follows the internal raster order.

[0225] Alternatively, in some embodiments, each image tile includes a plurality of segments, each segment also including a first number of pixel rows. The segments of the plurality of image tiles of the image frame are scanned row by row according to a predetermined raster order, and the pixels in each segment of each image tile are scanned according to an internal raster order. In another example, a first row number of a first image tile is extracted by raster scanning. After extracting the first row number of the first image tile, the encoder 142 extracts the first row number of a second image tile by raster scanning. The first row number of each of the first image tile and the second image tile is followed by the second row number. After extracting the first row number of the second image tile, the encoder 142 extracts the second row number of the first image tile by raster scanning. After extracting the second row number of the first image tile, the encoder extracts the second row number of the second image tile by raster scanning. The predetermined raster order includes an internal raster order, and the raster scan within each of the first image tile and the second image tile follows the internal raster order.

[0226] Reference above Figures 12A-12C Discuss more details about extracting frame data for an image frame from a data buffer.

[0227] It should be understood that the Fig.13 The specific order of operations in is exemplary only and is not intended to indicate that the described order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize various methods to manage the processing power of the server system as described herein.

[0228] In summary, some embodiments of the present application are directed to GPU virtualization supporting low latency and soft real-time workloads, and more specifically, to a method for managing processing power of a server system having one or more processing cores that also include multiple processing slices. Upon receiving a request to initiate an online game session, the server system assigns each processing slice of the processing core to a subset of the online game sessions to be executed thereon. A first processing slice is assigned to a first subset of the online game sessions that includes a first game session and a second game session. At the first processing slice, a time-sharing processing schedule for the first subset of the online game sessions is determined. According to the time-sharing processing schedule, the first game session and the second game session share a duty cycle of the first processing slice and are dynamically executed in parallel according to the real-time data processing needs of the first game session and the second game session.

[0229] Some embodiments of the present application are directed to concurrent rendering of multiple media streams, and more specifically, to a method for simultaneously rendering multiple media streams including at least a low-latency stream and a normal-latency stream. The low-latency stream corresponds to an online interactive session and has a faster response rate and lower transmission delay than the normal-latency stream. An encoder processor identifies a predefined frame rate for encoding a sequence of image frames of the low-latency stream. For each image frame of the low-latency stream, a time interval is determined according to the predefined frame rate, and the image frame is encoded within the time interval. Based on determining whether the remaining time left in the time interval meets the normal delay requirement associated with the normal-latency stream, the encoder processor optionally generates a subset of image frames of the normal-latency stream during the remaining time of the time interval.

[0230] Some embodiments of the present application are directed to a multi-core hardware accelerator for real-time video encoding, and more specifically, to a method for encoding a media stream. An encoder processor identifies frame data of an image frame associated with an online game session and allocates its encoder cores to process multiple image tiles of the frame data simultaneously and in parallel. The image tiles include a first image tile assigned to a first encoder core and including a first sequence of blocks. The first encoder core obtains a first block of the first image tile and boundary information of the first block. While or before obtaining a second block following the first block in the first sequence of blocks, the first encoder core encodes the first block in the first sequence of blocks based on the boundary information and transmits the encoded first block to a client device of a user associated with the online game session.

[0231] Reference has been made in detail to various embodiments, examples of which are illustrated in the accompanying drawings. In the above detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention and the described embodiments. However, the present invention may be practiced without these specific details. In other cases, well-known methods, processes, components, and circuits have not been described in detail in order to avoid unnecessarily obscuring aspects of the embodiments.

[0232] It will be understood that although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first device may be referred to as a second device, and similarly, a second device may be referred to as a first device without changing the meaning of the description, as long as all occurrences of the first device are consistently renamed and all occurrences of the second device are consistently renamed. The first device and the second device are both devices, but they are not the same device.

[0233] The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the claims. As used in the description of the embodiments and the appended claims, the singular forms "one", "an" and "the" are also intended to include plural forms unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more associated listed items. It will also be understood that when used in this specification, the terms "include" and / or "comprising" specify the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0234] As used herein, the term “if” may be interpreted to mean “when,” “upon,” or “in response to determining,” or “upon determining,” or “in response to detecting,” a stated conditional precedent is true, depending on the context. Similarly, the phrase “if it is determined that [a stated conditional precedent is true],” or “if [a stated conditional precedent is true],” or “when [a stated conditional precedent is true],” may be interpreted to mean, “after determining that a stated conditional precedent is true,” or “in response to determining that a stated conditional precedent is true,” or “upon determining that a stated conditional precedent is true,” or “after detecting that a stated conditional precedent is true,” or “in response to detecting that a stated conditional precedent is true,” depending on the context.

[0235] For the purpose of explanation, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise form disclosed. In view of the above teachings, many modifications and variations are possible. The embodiments are selected and described in order to best explain the principles of the invention and its practical application, so that others skilled in the art can best utilize the invention and various embodiments with various modifications to suit the specific purpose intended.

Claims

1. A method for simultaneously providing media streams, comprising: At a server system including an encoder and a memory storing a program for execution by the encoder: Generating, by the encoder, a plurality of media streams including a low-delay stream and a normal-delay stream, wherein the low-delay stream corresponds to an online interactive session and has a faster response rate and lower transmission delay than the normal-delay stream, comprising: identifying a predefined frame rate for encoding a first sequence of image frames of the low-latency stream; For each image frame in the first sequence of image frames of the low-latency stream: determining a first time interval corresponding to the predefined frame rate; encoding the image frame during the first time interval; determining a remaining time left in the first time interval after encoding the image frame; generating a subset of image frames of a second sequence of the normally delayed stream during the remaining time of the first time interval in response to determining that the remaining time left in the first time interval satisfies the normal delay requirement associated with the normally delayed stream; and Based on determining that the remaining time left in the first time interval does not meet the normal delay requirement, generating any image frames of the second sequence of image frames is suspended during the remaining time of the first time interval.

2. The method according to claim 1, further comprising: transmitting the low latency stream to a first client device of a game player for enabling the game player to perform a plurality of interactive actions during the online interactive session; and The normally delayed stream is sent to one or more second client devices to allow one or more viewers to comment on the online interactive session in real time.

3. The method according to claim 2, further comprising: obtaining a player demonstration video stream captured by the game player while the game player is participating in the online interactive session; and The player demonstration video stream is merged into the normally delayed stream so that the player demonstration video stream is played to the one or more viewers simultaneously with the normally delayed stream of the online interactive session.

4. The method according to claim 3, further comprising: The player demonstration video stream is enabled to be displayed as an embedded window while displaying the normally delayed stream on the one or more second client devices.

5. The method according to claim 3, wherein: The player demonstration video stream includes real-time commentary created by the game player while the game player is participating in the online interactive session.

6. The method according to any one of claims 2 to 5, further comprising: obtaining a player demonstration audio stream recorded by the game player while the game player is participating in the online interactive session; and The player demonstration audio stream is merged into the normally delayed stream so that the player demonstration audio stream is played simultaneously to the one or more viewers along with the normally delayed stream of the online interactive session.

7. The method according to any one of claims 1 to 5, wherein: The low-latency stream and the normal-latency stream are independent of each other.

8. The method according to any one of claims 1 to 5, wherein: The predefined frame rate for encoding the first sequence of image frames of the low-delay stream is determined based on a first threshold delay.

9. The method according to claim 8, wherein: The normal delay flow has a second delay time that is less than a second threshold delay, and the second threshold delay is greater than the first threshold delay.

10. The method according to any one of claims 1 to 5, wherein: The low-latency stream and the normal-latency stream correspond to the online interactive session; In response to a player action of a game player associated with the online interactive session, the low-latency stream and the normal-latency stream respectively have a first response rate and a second response rate for rendering a result of the player action; and The first response rate is faster than the second response rate.

11. The method according to any one of claims 1 to 5, wherein: The low-delay stream includes a first image frame, and a subset of image frames of the second sequence of the normal-delay stream corresponding to the first image frame includes a portion of image frames in the normal-delay stream, including: determining a first remaining time left in the first time interval after encoding the first image frame; and Based on determining that the first remaining time left in the first time interval satisfies the normal delay requirement associated with the normal delay stream, the portion of the image frame in the normal delay stream is generated during the remaining time of the first time interval.

12. The method according to any one of claims 1 to 5, wherein: The low-delay stream includes a second image frame, and a subset of the image frames of the second sequence of the normal-delay stream corresponding to the second image frame includes two or more image frames in the normal-delay stream, including: determining a first remaining time left in the first time interval after encoding the second image frame; and Based on determining that the first remaining time left in the first time interval satisfies a normal delay requirement associated with the normal delay stream, the two or more image frames in the normal delay stream are generated during a remaining time of the first time interval.

13. The method according to any one of claims 1 to 5, further comprising: determining a second remaining time remaining in the first time interval after generating a subset of image frames of the second sequence of the normally delayed stream during the remaining time of the first time interval; and Based on determining that the second remaining time left in the first time interval satisfies a third requirement associated with a third stream, a subset of image frames of the third stream is generated during the second remaining time of the first time interval.

14. A server system comprising: Encoder; and A memory storing one or more programs for execution by the encoder, the one or more programs comprising instructions for executing the method according to any one of claims 1-13.

15. A non-transitory computer-readable medium storing one or more programs for execution by an encoder of a server system, the one or more programs comprising instructions for performing the method according to any one of claims 1-13.

Images