Content streaming system and method

Abstract

An interactive content distribution system includes an edge server configured to receive interactive content streamed from a master server and stream the interactive content to a client device, the edge server configured to receive data streamed from the client device, and to stream the interactive content to the client device. The edge server is configured to combine the interactive content stream and the camera feed stream together to generate a shared stream, and the edge server is configured to cause the shared stream to be distributed to one or more audience devices.

Description

TECHNICAL FIELD

[0001] The present invention relates to a content streaming system and method. BACKGROUND

[0002] Content providers such as video streaming services and similarly some websites typically have a primary or origin server that provides its content, and these servers can for example be geographically located close to the service provider.

[0003] However, for providers with a global reach, this model can result in undesirable latency for users that are far away from the primary server. Therefore, so-called edge servers can be deployed on a regional basis to cache content from the primary server for delivery to users in its region, thereby reducing the latency for these users.

[0004] However, while this approach works well for non-interactive content, where the edge server acts as a passive store for a copy of the content originating from the primary server and can be updated on a scheduled basis in a similar manner to a virtual warehouse, it is not suitable for interactive content such as video games, where the content is generated by the primary server in real-time in response to input from the user; in this case, the edge server would simply appear to be another node forwarding live content between the primary server and the client, but in particular game applications are very sensitive to latency, so this problem can be serious when there is a geographical separation between the primary server and the user, and therefore an equivalent to this edge server for interactive content would be desirable.

[0005] The present invention seeks to alleviate or mitigate this problem. SUMMARY

[0006] Various aspects and features of the present invention are defined in the appended claims and within the text of the accompanying description.

[0007] In a first aspect, there is provided an interactive content distribution system according to claim 1.

[0008] In another aspect, there is provided a method of interactive content distribution according to claim 14. BRIEF DESCRIPTION OF DRAWINGS

[0009] A more complete understanding of the present disclosure and the many attendant advantages thereof will readily be had by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:

[0010] - Figure 1 is a schematic diagram of a server-client system known in the art.

[0011] - Figure 2is a schematic diagram of an interactive content provision system according to an embodiment of the present specification.

[0012] - Figure 3 is a schematic diagram of an interactive content provision system according to an embodiment of the present specification.

[0013] - Figure 4 is a schematic diagram of an interactive content provision system according to an embodiment of the present specification.

[0014] - Figure 5 is a schematic diagram of a client device system according to an embodiment of the present specification.

[0015] - Figure 6 is a flow diagram of a method of interactive content provision according to an embodiment of the present specification. DETAILED DESCRIPTION

[0016] An interactive content distribution system and method is disclosed. In the following description, numerous specific details are presented in order to provide a thorough understanding of embodiments of the application. However, it will be apparent to one skilled in the art that the present application can be practiced without requiring all of the specific details. In other instances, well-known specific details have been omitted in order to avoid obscuring the present application.

[0017] Reference is now made to the drawings, in which like reference numerals refer to like or corresponding parts throughout the several views, Figure 1 A conventional server-client arrangement is shown, comprising a server 1000 in communication with a client device 10 via a long distance network 404 such as the internet.

[0018] The time taken for data to travel from the server to the client can be relatively long, partly as a pure function of distance, but more importantly because the hops (connections within the internet linking the main server to the client) tend to increase the time and slow the effective speed of the data.

[0019] Thus, for a streamed video game, in which the current game image is to be displayed as it is received, the latency of that image is both a function of the distance travelled (which can be thought of as contributing to the front end latency), and a function of the data size, because the large amount of data travelling slowly needs longer to arrive in its entirety (which can be thought of as contributing to the back end latency).

[0020] In conventional edge server systems, in effect, this problem is bypassed by providing a copy of the source data closer to the user, so the distance is shorter, there are fewer hops, making the data more responsive and faster.

[0021] However, for interactive content, this approach is not possible because the source content is created in real-time in response to user interaction. While more master servers closer to the user can be provided, this is not an edge server system.

[0022] Edge server

[0023] Thus, in embodiments of the present specification, an edge server system is provided that improves latency for at least some portions of streamed interactive content, such as video games.

[0024] Reference Figure 2 In embodiments of the present specification, an interactive content providing system (or "edge server system" elsewhere herein, e.g., a system including edge servers as well as master / source servers for providing interactive content to clients) includes an adjusted master server 1000', an edge server 1010, and optionally a client device 10 (e.g., when a client joins to access content provided by the edge server system).

[0025] In this configuration, the master server still streams interactive video game content to the edge server, represented by the dashed line. Obviously, there is scope to optimize the path between these two servers as they will be persistent points of communication, and this can itself contribute to latency and speed (and thus to front and train edge latency).

[0026] It will be appreciated that there can be multiple edge servers deployed at different locations geographically and / or according to network topology, and one edge server is selected for this purpose that is suitable for the end client of this interactive content.

[0027] The edge server can then be utilized to reduce the amount of data streamed from the master server in one or more successive more interactive modes.

[0028] First mode

[0029] In a first mode, the game can have non-interactive elements (although typically they can be skipped by the user), such as so-called cut-scenes. Conventionally, the master server streams these identically to any other part of the game content.

[0030] However, in instances of the present specification, the master server can instead output data identifying these cut-scenes, where these occur, rather than the cut-scenes themselves (such requests, commands, or metadata are represented by the solid line); the edge server can then provide these from a local cache.

[0031] In this case, the cut scene can be a pre-recorded video associated with the game, or can be a scripted animation within the game. In the latter case, the resulting scripted animation can be recorded and played back in a similar manner to the pre-recorded video at the edge server. The main server can then send an instruction to the edge server that it should stream the relevant recorded content from its cache, rather than the resulting video itself.

[0032] Optionally, the edge server can respond to the instruction to confirm whether it has the relevant recorded content. If not, the server can generate and output it in the usual way, whilst instructing the edge server to record the stream as well. Then, when another instance of the game on the main server again encounters the cut scene, that instance will not need to generate its own stream.

[0033] In this way, the edge server can build up a repository of such cut scenes. This can be particularly useful for games in which different versions of the main character are available, and so multiple versions of a scripted cut scene can be possible.

[0034] The first mode described above relates to non-interactive content interspersed between periods of interactive content.

[0035] Meanwhile, the second mode relates to non-interactive content provided in parallel with interactive content.

[0036] Second mode

[0037] One example of this is audio; whilst the user can interactively change position and viewpoint within the game, and so change the image streamed, this is therefore interactive, typically some music, some ambient sound and some dialogue can continue independently of this interaction, or be modified in only a relatively simple manner (e.g. changing the relative volume levels on the respective channels). This can therefore be considered as non-interactive content running in parallel with the interactive content.

[0038] In practice, such content is often interactive in that it is triggered by a change in the game state, but once triggered / started, it is played back in a non-interactive manner, rather than being generated frame-by-frame or in real-time as with the image in the game.

[0039] Thus, in this second mode, the edge server can cache non-interactive content that is provided in parallel with interactive content. In other words, it can cache pre-recorded elements of content (i.e., inherently non-interactive content components) that are then used interactively (i.e., triggered / selected and played back in response to evolving game play and user actions), but are non-interactive during use (i.e., simply played back). Notably, even such non-interactive content can often be interrupted and skipped, and as noted above, can be modified in playback by changing channel volume or applying effects such as reverb.

[0040] Thus, in this mode, the main server can stream generated images interactively, but in a similar manner to the cutscene scenario, does not stream (or all) audio that accompanies them. Instead, the server can output data to the edge server that identifies the audio data to be produced and its timing, in a manner that is generally similar to how it would internally output a request for an audio asset from an audio file repository stored locally at the main server to be played back at a particular time.

[0041] The edge server can then receive the request as well as the stream from the main server, add the identified audio, and relay it to the client. The edge server can also implement channel level adjustments (e.g., to change the audio levels of stereo, 5.1 or 7.1 sound, e.g., in response to main server instructions to represent a change in position or distance relative to a virtual sound source).

[0042] The edge server can also implement environmental processing on the audio, such as reverb, or cache audio to which such processing has already been applied, selecting the appropriate version based on instructions included in or associated with the server stream.

[0043] By avoiding the need to include at least some dialog, music, environmental sound and / or common sounds such as footsteps or gunfire from the data stream from the main server, the entire stream from the main server can be smaller on a frame-by-frame basis.

[0044] In addition to freeing up bandwidth at the main server and potential congestion, this can also reduce the tail latency of the stream, as for most networks between the main server and the edge server, the total time to complete the data associated with a current content frame will generally be shorter. As a result, the completed image will appear on the client's screen sooner.

[0045] If the first mode involves completely non-interactive content, and the second mode involves non-interactive content initiated in parallel with interactive content, the third mode involves interactive content that is not currently being interacted with.

[0046] Third mode

[0047] This typically happens when the viewpoint of the game is controlled by the game itself rather than the user, and so typically most of the scene is static or predictable / scripted. Thus, for example in platform games and so-called metroidvania games, most of the on-screen environment is static for the duration of the user's play, and typically flips between static environments every few seconds or minutes. Similarly, dialogue scenes with non-player characters (NPCs) typically take a fixed camera viewpoint, so most of the image can be predetermined. Similar observations apply to items such as in-game menus, skill trees, maps, etc. which can change as the game progresses (e.g. more of the map becomes visible) but tend to be static at the time of display.

[0048] Thus, in the third mode, the edge server is provided with cached image data for known views of the game, e.g. respective levels of a platform game, or a continuous shot of the background including NPC dialogue, or even a continuous shot of NPC dialogue for some (e.g. N most common) or some (e.g. top M most common) or all dialogue trees for all NPCs.

[0049] The main server can then provide the edge server with information indicating whether image information or background image information from the cache is required, and only stream those parts of the image which diverge from the cached version. For example, in a platform game, these can include the main character, the current enemy, and any parts of the scene which include moving elements of the environment or elements affected by lighting effects, for example. Meanwhile, the remainder of the image need not be included in the stream.

[0050] Which parts of the image are included in the stream and which are not can be indicated, for example, by using image tiles or other addressable chunks of the image. The interactive content can be streamed to the edge server on a tile-by-tile or chunk-by-chunk basis, and then the missing chunks added to the stream by the edge server. Alternatively, the interactive content can be streamed as the whole image with, for example, black / blank regions, where the cached content will be used, then the streamed image updated by the edge server as appropriate. This can be done while the stream is still encoded (e.g. with larger encoded cached data replacing smaller encoded blank data) or after the image is decoded, merged with the cached image, and re-encoded and sent.

[0051] In this way, the amount of data streamed from the main server can be significantly reduced, again freeing up bandwidth and potential congestion at the main server, and again reducing the tail latency of the stream, since for most networks between the main server and the edge server, the total time to complete the data associated with the current content frame is generally short. As a result, the completed image will appear on the client's screen sooner.

[0052] As mentioned above, this approach can be used when a large part of the scene is predictable / scripted. In this case, it can thus be used for cut scenes using in-game scripted animations featuring a main character whose appearance has been configured by the user, either through a face / body editor in the game, and / or by selecting the clothes, weapons, etc. they are equipped with.

[0053] In this case, a part of the cut scene including the customised character can be sent from the main server, and the rest can be added at the edge server.

[0054] A similar implementation can be pre-recorded videos merged into the game; this can happen when the user is navigating a skill tree, and an animated video shows what the new skill will do. At other times, it is an element of an environment such as a TV show. Again, the main server can instruct where the video is to go within the streamed image (optionally with appropriate scaling and / or warping), and the edge server can merge it.

[0055] As an extension of the third mode, as an alternative or supplement to the techniques of the third mode, the edge server can optionally cache the last complete image sent to the client. This can then be signalled to the edge server if the next generated image does not include a change of field of view (e.g. because the user has not moved their character or viewpoint between successive frames), and only those parts of the next generated image that differ from the previous image need to be sent. This can again take the form of only sending a subset of tiles or blocks containing changed areas, or an image with unchanged areas masked out to be replaced at the edge server with image data. Alternatively, instead, it can take the form of incremental data encoding the changes between successive images; the edge server would then need to compute the resulting new image.

[0056] Further extensions of this mode can use the motion vectors for the motion between the cached video image and the next video image to offset the image data in the cache and still use it even if the user does move their viewpoint slightly between frames. In this case, the relative position of the static elements of the two images can be easily indicated by the panning and / or zooming metadata, while any new image data that is brought into view can be sent from the main server. Dynamic elements of the image, such as moving characters, can be transmitted from the main server as before, but can also optionally be encoded using a motion-aware encoding scheme such as H.264.

[0057] In each case, this will again reduce the size of the overall stream from the main server, as it will only need to send the frame-to-frame differences in the image.

[0058] In addition to freeing up bandwidth at the main server and potential congestion, this can also reduce the tail latency of the stream, as for most networks between the main server and the edge server, the total time to complete the data associated with the current content frame will generally be shorter. As a result, the completed image will appear on the client's screen sooner.

[0059] Fourth mode

[0060] Finally, in a fourth mode, the edge server can interact with the main server to offload some of the rendering requirements of the main server. For example, most games have a foreground region that is typically composed of polygons, textures are applied to the polygons, and the polygons are typically interactive in the sense that they are known geometry within the environment - but beyond a certain rendering distance, this can be replaced with elements that simply provide a background. A common example would be a so-called skybox that provides a spherical or hemispherical background image of the distant environment (e.g. the sky, and possibly distant mountains). This fills the screen everywhere there is no foreground content.

[0061] In the fourth mode, the skybox is not rendered by the main server, or is rendered at low resolution, or is normally rendered but is not included in the transmitted stream in each case (e.g. masked using z-axis information within the game environment that indicates the farthest extension of foreground objects). This again reduces the size of the overall stream from the main server, with the attendant benefits described elsewhere in this document.

[0062] The main server also sends information to the edge server indicating the image viewpoint (e.g. view direction and field of view, and typically also the location in the game world), the edge server then selects the appropriate element (e.g. skybox), and the appropriate orientation of the skybox relative to the viewpoint, and adds the skybox image data to the streamed image to complete it.

[0063] The method need not be limited to skyboxes, although this can represent a relatively large proportion of the transmitted images in some games; it can also be applied to heads-up displays or other elements of constant overlay on the image, which then need not be included in the main server image, or at lower resolution (e.g. if this facilitates transparent operation of the game).

[0064] It can thus also be applied to, for example, a part of the geometry in a game, which in some games implements a similar or complementary function to a skybox, such as a stadium and / or the spectators in it in a football game, which are not obviously reused or crowd animation generic, for example, if they are at a sufficient distance from the camera viewpoint / foreground action. A small number of variations can be made (e.g. depending on the respective team, weather, day or night match, etc.), and the main server can request the most appropriate variation.

[0065] It will be appreciated that the main server and edge server can operate in any of the above modes, sequentially and / or in parallel (i.e. using different modes for different parts of the content, or for different image frames, or within different parts of the same image frame, as appropriate).

[0066] Thus, in general, for each of the four modes, the edge server caches game assets (video, images, skyboxes, overlays, etc.), and receives dynamically generated content (i.e. content that is not predetermined and common to most or all instances of the game) from the main server, such as the current player and enemy actions, or the viewpoint at the time of the set or just for the previous frame, which has not been cached in the edge server. The edge server then typically combines the received content with the cached content, as appropriate, usually in response to metadata indicating relevant aspects of the current game state, before passing the combined result to the end client.

[0067] Thus, advantageously, the majority of the network path from the main server to the edge server is traversed only by the smaller dynamic component of the content, reducing bandwidth utilisation and congestion, and the duration between the leading and trailing edges of the data, reducing the total time to complete reception of the data.

[0068] Provided that the time taken by the edge server to fill in the rest of the content is shorter than the time saved by sending only the partial content from the main server, the result is a net reduction in latency when the complete content is delivered to the end client.

[0069] The above caveat can be tested as part of the process; thus, if the client has a high quality connection (e.g. fast, with high bandwidth) between the client and the main server, optionally, the system can revert to the traditional streaming model, streaming only the raw complete output from the main server (even though the edge servers are still hops / relays on the path between them).

[0070] Cache option

[0071] The main server can deliver content to multiple edge servers. While in principle all edge servers can be provided with all content required to implement some or all of the techniques / modes herein, for a large number of games, this can be a large amount of data.

[0072] Notably, for fast implementation at the edge servers, this data should preferably be available for fast calling, e.g. within RAM or equivalent cache, rather than e.g. stored on HDD or raid array.

[0073] This in turn imposes a practical limit on the number of assets that can be stored for use by the edge servers.

[0074] Thus, in embodiments of the present specification, the content streaming system evaluates which assets are available e.g. in accessible / “working” memory (i.e. memory capable of delivering assets fast enough to contribute to latency reduction).

[0075] This evaluation can make use of one or more of the following metrics:

[0076] - which games are most frequently accessed through the edge server

[0077] - this can be used to provide an initial set of assets

[0078] - where the server is located

[0079] - this can be used to select geographically localised versions of resources, e.g. language of dialogue

[0080] - which specific assets are most frequently used by the edge server

[0081] - e.g. footstep audio can be frequent, while specific lines of dialogue can not be

[0082] - equivalent server bandwidth cost x frequency of asset

[0083] - small but frequently used assets can reduce more total bandwidth than large but rarely used assets.

[0084] - allocation of bandwidth saving

[0085] - a clipping scenario can save a large amount of bandwidth for 30 seconds, but can only be done once per hour; thus, it can not help with general congestion or tail latency as much as assets that save less bandwidth, but do so almost continuously. The relative value of these assets can depend on additional factors, such as network conditions at the main server and / or edge servers, quality of service thresholds, current user activity, etc.

[0086] Alternatively or additionally, other metrics not directly related to assets can influence the operation of the system, such as current main server load, current edge server load, current actual latency and latency sensitivity (general or at this time) of the client game, user bandwidth, and network congestion. Thus, for example, a clipping scenario can save a large amount of bandwidth for 30 seconds, but it can be latency insensitive if the user does not need to react to it, and thus does not need to be provided by an edge server, unless there can be network congestion at the main server, as it is running close to capacity, in which case the reduction in bandwidth due to not outputting it at the main server is generally beneficial.

[0087] The edge server can track which assets are used and the frequency of use to determine the effective bandwidth of each periodic value of the corresponding asset, and keep the assets with the highest values in working memory.

[0088] To enable this cache to evolve, the cache can include a short-term and long-term component, where new cache items received from the main server are evaluated relative to the long-term items.

[0089] Optionally or additionally, in cases where some items are stored in working memory and some items are in slower memory (at the edge server or at the main server or at a third server, not shown), tracking by the main server of the number of requests to use assets can be used to determine the upgrade of assets from memory to memory, or conversely, the downgrade from memory to memory.

[0090] Again, assets can be selected for inclusion in memory based on several metrics, including only the frequency or number of requests in a period; or the bandwidth cost of the asset x the frequency or number of requests in a period.

[0091] Thus, for example, on one edge server, a particular game is very popular, and thus the edge server has received a set of assets to use with the techniques herein. It can also be provided with an initial subset of assets to use in working memory (e.g., corresponding to welcome and menu screens, tutorial levels, etc.), either identified by the developer, for example, or empirically identified, for example, during Q&A testing.

[0092] When a player on a client device streams a game, the master server requests use of assets at the edge server according to any of the techniques herein, and the edge server can track which assets are requested for use; in the case that a first asset must be denied use by the requesting edge server (e.g., because it is in slow storage), and the bandwidth cost of the first asset x the frequency or number of requests for that first asset in the period exceeds the equivalent calculation for a second asset currently in working memory, then it is more efficient that the second asset be demoted to slow storage, and the first asset be promoted to working memory (assuming there is no spare memory to accommodate both).

[0093] By comparing the actual utilization of assets in working memory to the potential utilization of assets not in working memory (e.g., in slow storage or not yet fetched from the master server), the edge server can continuously maintain a good and preferably optimal clustering of assets in memory using the techniques herein to reduce the total bandwidth and congestion at the master server, and to reduce the tail latency for users.

[0094] Variant

[0095] The above embodiments assume that some content can be added to the stream at the edge server.

[0096] However, in principle some content can be added at the client, alternatively or additionally to some content added at the edge server.

[0097] Reference is now also made to Figure 3 In this case, for example, the master server 1000" or the edge server 1010' can track which assets are most efficiently used, and download them to the client 10'. The client can then add them to the stream image itself. Advantageously, this can also reduce the bandwidth, congestion, and tail latency in the network between the edge server and the client (which can be the slowest part of the journey).

[0098] It can also free up working memory of the edge server to provide other assets to an incoming stream in the same game or other games, such that there can be even greater bandwidth and latency reduction between the master server and the edge server.

[0099] Thus, in some embodiments of the specification, the techniques herein can be implemented by use of assets at the edge server, the client, or both. The choice of what to implement can depend on the developer's choice, the relative network conditions between the edge server and the client, and the computing power and memory capacity of the respective client. In this regard, different clients in contact with the same edge server can operate differently.

[0100] Thus, in some embodiments of the specification, the techniques herein can be implemented by use of assets at the edge server, the client, or both. The choice of what to implement can depend on the developer's choice, the relative network conditions between the edge server and the client, and the computing power and memory capacity of the respective client. In this regard, different clients in contact with the same edge server can operate differently. Figure 3In this case, the edge server 1010' can estimate that a subset of assets are most common for the game that is currently being played on the client device 1010', and send copies of these assets to the client device.

[0101] The edge server then relays the request from the master server to the client device to incorporate any of these assets into the stream, rather than doing so itself. At the same time, optionally, it can include other assets that it now has free memory to use.

[0102] The client device then incorporates the requested assets into the stream or decoded partial images as appropriate in a similar manner to that previously described for the edge server.

[0103] Alternatively or additionally, when the client has received the selected assets, the edge server can then inform the master server that these assets are now at the client device. The master server can then transfer the use of the techniques described herein from the edge server to the client device, entirely for that device, at least for those assets. In this case, the master server can route the streamed data directly to the client, as well as use the requests / commands of the client holding the assets according to any of the techniques herein.

[0104] It will be appreciated that a hybrid approach can also be used, in which some data is streamed to the client with its commands (optionally using the edge server as a passive hop, so that the total path time remains similar), and some data is streamed to the edge server with commands via the edge server (e.g. where the master server knows that the client device does not have assets that the edge server can provide).

[0105] Thus, in embodiments of the present specification, the master server (1000', 1000'') can output a stream of partial images and / or audio and requests added by one or both of the edge server and the client device. In the case that only a subset of the assets is currently available at the edge server or the client device, this is received by the server, which can then simply omit the partial images / audio within that subset, but also indicate which other partial images / audio can be replaced from the wider set of assets, so as to facilitate updating of the subset when usage trends change, all in accordance with any of the techniques described elsewhere herein.

[0106] Meanwhile, in embodiments of the present specification, an edge server 1010, 1010' can receive from the main server a stream of partial images and / or audio and a request to add those streams by the edge server. In cases where the edge server has limited memory and can only do so for a subset of possible assets, it can inform the main server which parts of the image and / or audio it can currently reconstruct. It can also maintain a record of which other parts of the image and / or audio can be replaced from a wider set of assets in order to swap assets with higher potential utility for those with lower utility in memory, all according to any of the techniques described elsewhere herein. The edge server then forwards the reconstructed image and / or audio to the end client, optionally with a request for further reconstruction if the client is using similar techniques.

[0107] Similarly, in embodiments of the present specification, a client device can receive complete streamed images and / or audio from the main server, or can receive complete or reconstructed streamed images and / or audio from an edge server, or can receive partial images and / or audio from either server and a request to merge the assets into partial images and / or audio that the client device has previously received, as according to any of the techniques described elsewhere herein.

[0108] User streaming

[0109] So far, the techniques described herein relate to when a game is generated on a main server, and reducing the overall latency of the game that is streamed down from the main server to an edge server and from the edge server to a client device. This can involve providing all or part of the image from the edge server rather than from the main server, and can even involve the edge server performing some of the computations that contribute to image generation on behalf of the main server.

[0110] However, there are other scenarios that are traditionally assumed to have the game generated at a client device, and then streamed up to a distribution server. These scenarios typically involve so-called streamers, i.e. people who share their gaming experience on various streaming sites.

[0111] Typically, a streamer will use a screen recorder (either as software within the client device or intercepting its output) to capture the game image. They can also use a camera to record themselves. They then typically composite these two image sources on a computer, with the video image comprising a relatively small portion (e.g. typically less than ¼, ½, ¾ or 10th of the screen), and encode and stream the result up to a streaming service. The compositing computer can alternatively or additionally add other graphical overlays, such as the streamer's logo. Meanwhile, the streaming service itself can also composite the received stream, e.g. with a live chat box for viewers to chat in.

[0112] This approach has a number of problems. Firstly, the local composition and encoding of the game image, optional camera feed and any other graphical adornments (e.g. the streamer’s logo) is computationally heavy. If done on the device running the game, it can impact the quality of the game (e.g. frame rate). If done on a separate computer, it can introduce latency for viewers compared to active players of the game. However, and perhaps more importantly, most private individuals have relatively low upload bandwidth compared to download. This means that the uploaded stream is typically not of very high quality. For example, the game can generate HD or 4K quality images, but the stream from the user’s computer or device can be only SD or low quality / highly compressed HD. Additionally, in a cloud gaming application with a thin client on a low power device (e.g. a smartphone), this high encoding computational cost can result in lower battery life, resulting in a lower quality of experience for the player.

[0113] Furthermore, typically the streaming service then decodes the stream in order to add other features such as chat, and then transcodes the content for different user requirements; and transcoding of low quality images can be sub-optimal and further impact the quality of the viewer’s experience.

[0114] Accordingly, such streamers prefer to play their games on a host server which has much higher bandwidth than their own server, and is able to provide high quality images to the stream distributor.

[0115] In this case, while the images from the host server will still be displayed to the user of the client device, if the user then composes and streams the game as before, this will defeat the potential benefits of the approach.

[0116] Accordingly, reference is now made to Figure 4 In embodiments of the present specification, this composition and transmission to the streaming server can be performed by the edge server.

[0117] The host server (1000, 1000', 1000") generates interactive content (e.g. a game) and streams the generated interactive content to the edge server 1020. Optionally, this can involve any of the techniques described elsewhere herein, e.g. in relation to any of the described modes or caching strategies of the edge server. Alternatively, the stream from the host server to the edge server can be conventional and not supplemented with any caching of supplemental material at the edge server to supplement the host server’s streaming output.

[0118] In any case, the edge server then streams the image of the interactive content to the client device (10, 10'). The user of the client device can then play the game, sending input commands back to the main server (typically via the edge server).

[0119] Typically, the client device (or computer used for streaming) is operatively coupled to the camera 12, and can upload only the video feed for compositing with the game images to the edge server (e.g., typically in parallel with the game control inputs, but without the game images); furthermore, because this is typically a small insertion (as noted above, typically less than 1 / 4, 1 / 6, 1 / 8, or 1 / 10 of the resolution of the game images), the per-frame bandwidth can be significantly less than the per-frame bandwidth of the game images, and thus this video can be uploaded from the client's device or computer with good quality using the upload bandwidth available to the user, and in any case without any impact on the quality of the game images, since these images are not uploaded from the client side at all. It will be appreciated that the uploaded video will also typically have associated audio from the streamer, which will also be up-streamed to the edge server.

[0120] The edge server now has access to the high-quality game images down-streamed from the main server (and optionally completed at the edge server), as well as the high-quality video of the streamer up-streamed from the client device. Separately, the edge server can also relay the game control signals (e.g., keystrokes, button and / or joystick inputs, motion inputs, and optionally voice commands) from the client to the main server, or these can go directly from the client to the main server.

[0121] Optionally, the main server, edge server, and client can cooperate to ensure synchronization of the game images and the streaming video, particularly if the round-trip delay between the edge server and the client is greater than the duration of a game image frame (typically, but not necessarily, 1 / 60 of a second).

[0122] For example, the main server or edge server can add a frame number, timestamp, or similar unique identifier to each respective image frame. The video frames captured by the user's camera can then be associated with the unique identifier of the frame that was displayed to the user on the user's client device at the time the video frame was captured.

[0123] The edge server can use the unique identifiers associated with the game images and the video images to synchronize them, so that the composite image has the two components properly synchronized. It can be assumed that the uploaded audio is synchronized with the uploaded video, either separately or as part of the video stream.

[0124] The edge server typically performs the compositing of these two sources. Even in cases where the anchor does not include a camera feed, they typically include other elements to be composited, such as an anchor logo, etc., which can be uploaded from the client and / or cached at the edge server. Thus, in the following, when referring to the composited game image and video image, "video image" is a generic term for the camera image and / or any other streamer-sourced image composited onto the game image.

[0125] The server can include multiple nodes or processors (not shown) for respective tasks, e.g. a first node with the function of providing a relay / downstream game image cache, optionally a second node providing an upstream video image cache function (or this can also be done by the first node), and optionally another node performing the compositing function, although this can also optionally be performed by any of the nodes of the edge server, or by another compositing server local to the edge server. The client streamer can provide standard controls (e.g. via a streaming API or other convention) providing information about the streamer intended placement and optionally the shape or scaling of the video feed and any other elements as well as an API key. This information can also include whether any green screen background has been removed or needs to be removed at the edge server before compositing.

[0126] The edge server can then communicate with a third party streaming distribution service, or be part of the streaming distribution service itself.

[0127] In a first instance, the edge server can send the composited game image and video image to a third party streaming service (e.g. streaming server 1040) along with any credentials or streaming control data uploaded from the client streamer, to allow this composited stream to be used with its streaming account.

[0128] Alternatively, the edge server can send the game image and video image as separate streams to a third party streaming service along with synchronization information and any credentials or streaming control data, so that the third party streaming service can composite the images with any overlays of its own, such as chat feeds and / or streamer logos. This minimizes the number of compositing steps. It also allows separate transcoding of the game image and video image before compositing, if required by different end user audiences, which in turn can improve the overall image quality in the final composite.

[0129] Other optional elements that can be added to the stream include graphical overlays of input devices to show the desired control inputs being provided by the streamer. When the streamer is playing a game, the input can be highlighted based on the control inputs being forwarded by the edge server to the host server. This can be useful, for example, in the case where the streamer is providing a tutorial, so the view of the input can be useful to other players.

[0130] In the case where the edge server communicates in this way with third parties, the network link between the two can be optimized in any way, and will typically have much higher bandwidth available than the client streamer in any case. Again, this serves to improve image quality and reduce latency.

[0131] The stream media server can then provide the stream to a plurality of viewer devices 10A, 10B,... 10N, which can be of different types and can receive the encoded stream or in some cases a differently transcoded stream.

[0132] It will also be appreciated that the stream server can also implement any of the techniques described herein to cache content received from the edge server; in this case, the edge server itself can also operate partly like the host server in that it instructs what content needs to be cached at the stream server to complete the image. Common content can be the logo of the streamer and the like.

[0133] In a second instance, the edge server is part of a streaming distribution service. In this case, the functionality of the third party streaming service is implemented by the edge server, optionally at another node of the edge server, or by a distribution server local to the edge server.

[0134] In this case, the edge server can composite together the game image, the video image, any streamer overlays and any streaming service overlays (logo, chat functionality, etc.), and optionally transcode the game image and video image streams separately, or once composited together, or once all elements are composited together.

[0135] The edge server can then provide the stream to a plurality of viewer devices 10P, 10Q,... 10X, which can be of different types and can receive the encoded stream or in some cases a differently transcoded stream.

[0136] It will be appreciated that in the above embodiments, optionally, the host server can operate transparently; that is, it is unadjusted with respect to implementing the user streaming techniques disclosed herein. In particular, frame synchronization can be performed by the edge server without any input from the host server. This allows the benefits of the user streaming techniques to be enjoyed on a legacy cloud streaming system.

[0137] Alternatively, it can provide a frame number, timestamp, or other frame ID for the purpose of synchronization of the composition of the stream by the edge server or any server and optionally implementing any other technique described herein.

[0138] At the same time, it will be appreciated that when streaming a game from a service that does not support the above techniques, the client device or companion streaming computer (as the case can be, collectively referred to as a client streaming device) can operate in a conventional manner, but can then switch to the techniques when they are supported, and optionally when the uplink bandwidth of the client streaming device is below a threshold (e.g., a threshold required for a predetermined image quality at an intended streaming resolution).

[0139] As described elsewhere herein, connections between various servers of the streaming system can seek to optimize their connections between them. Optionally, this optimization can be based on quality of image feedback from receiving servers or clients and / or quality of experience feedback from users, which is used to tune networking parameters / routing. Optionally, the feedback is correlated to parameters / routes by a machine learning model (e.g., supervised or semi-supervised model) that is trained with such feedback to tune various parameters / routes.

[0140] Variant

[0141] The above embodiments relate to streaming by a single streamer. However, optionally (e.g., for e-sports broadcasts), the above techniques can be applied to multiple streamers.

[0142] In this case, advantageously, each streamer uploads much less data (typically only their own live video and input controls thereof), and this is to the edge server; the edge server then relays the input controls to the main server of the game instance in which the e-sport event is running. This tends to reduce overall latency (as the competitors upload less time, and only to the edge server - hereafter the edge server can forward the inputs by, e.g., high-bandwidth / optimized means) and latency variability between players (e.g., the range of significant differences in latency between the competitors and the edge server is less, and again reduced if the edge server uses the same routing / means to route inputs to the main server for all competitors).

[0143] This makes cloud-hosted e-sports more responsive and more fair in terms of the technical infrastructure of the game.

[0144] Furthermore, if the game images are from the main server and the video images are from the competitors’ client devices, the broadcast results will be better, as described elsewhere herein.

[0145] As a result of lower and more equal latency, the result will be a better experience for the competitors, and for the audience, a better quality visual effect.

[0146] In this case, the edge server can thus receive multiple streams of the respective instance of the game and / or view, and optionally the corresponding video stream of the competitor, and compose these streams together with any other overlays and information, as described elsewhere herein.

[0147] Optionally, in the case that the competitors are geographically distant, they can use respective edge servers, which then communicate using high bandwidth / optimised means.

[0148] Similarly, optionally, the esports director can access a user interface to select game image streams and video streams (and any other graphical assets) and select which elements to compose. Thus, the director can act as a composition director and issue commands to the edge server as to how to compose and what to compose. Thus, for example, a game stream from the main server created for the director in spectator mode in the game (which does not necessarily correspond to a view from any competitor) can be composed with video streams from one or more competitors, for example to create an overhead view of a skirt in the game, and the reactions of the competitors involved as potential one of multiple shared streams.

[0149] It will be appreciated that the above methods can be used for an esports running over the internet, but also for a situation where some or all of the parties (main server, edge server, competitor clients, director client) are on a local area network.

[0150] Hardware

[0151] Reference is now also made to Figure 5 which illustrates an example of a client device (10, 10') but which can also (in physical or virtual form) be used as an example of at least part of a main server generating interactive content, and / or as an example of at least part of an edge server caching content, and / or as an example of at least part of a streaming server distributing shared streams to client audiences. The client device (10, 10') can for example be a computer or a console.

[0152] The client device comprises a central processor or CPU 20. The client device also comprises a graphics processing unit or GPU 30 and RAM 40. Two or more of the CPU, GPU and RAM can be integrated as a system on a chip (SoC). Further storage can be provided by a disk 50, either as an external or internal hard disk drive, or as an external solid state drive, or as an internal solid state drive.

[0153] The client device can send or receive data via one or more data ports 60 (such as USB ports, Ethernet ® ports, Wi-Fi ® ports, Bluetooth ® ports, or the like). It can also optionally receive data via an optical drive 70. Audio / visual output from the client device is typically provided through one or more A / V ports 90 or one or more data ports 60. Where components are not integrated, they can be connected as appropriate by dedicated data links or via the bus 100.

[0154] An example of a device for displaying images output by the entertainment system is a head-mounted display "HMD" 120 worn by the user 1. Another example would be a TV or integrated display (not shown).

[0155] Interaction with the client device is typically provided using one or more handheld controllers 130 and / or, in the case of an HMD, one or more VR controllers (130A-L, 130R).

[0156] Overview

[0157] In the overview embodiments of the present specification, the interactive content distribution system comprises the following.

[0158] The edge server (1010, 1010', 1020) is configured to receive interactive content streamed from the main server (1000', 1000'') and to stream that interactive content to a client device (10, 10'), as described elsewhere herein.

[0159] The edge server is configured to receive data streamed from the client device, as described elsewhere herein. That data stream is typically a camera (12) feed, but optionally or additionally can include any other source from the streamer to be composited with the interactive content stream (e.g. a logo, chat data, etc.).

[0160] The edge server is configured to composite the interactive content stream and the data stream together (e.g. by a third party service, or a node of the edge server or a server local to the edge server) to generate a shared stream (e.g. a stream to be distributed to the client audience), as described elsewhere herein.

[0161] The edge server is configured to cause the shared stream to be distributed to one or more audience devices (e.g. by a third party service, or a node of the edge server or a server local to the edge server), as described elsewhere herein. Optionally, any one or more of these can also transcode the stream to multiple resolution-bitrate pairs.

[0162] Examples of this summary embodiment implementing the methods and techniques described herein (e.g., by using suitable software instructions) are contemplated within the scope of the present application, including but not limited to:

[0163] - the data stream comprises a camera (12) feed, as described elsewhere herein;

[0164] - the edge server is configured to provide to the client device a frame identifier (e.g., frame number, timestamp, or other unique ID) of at least some frames (e.g., every frame, or every other frame, or every I-frame, or every Nth frame, etc.) of the interactive content streamed to the client device, as described elsewhere herein;

[0165] - the edge server is further configured to receive, in association with at least some frames of the camera feed, a frame identifier corresponding to a frame of the interactive content displayed at the client device when the respective frame of the camera feed was captured, as described elsewhere herein; and

[0166] - the edge server is configured to synthesize the interactive content stream and the camera feed stream together based on the one or more interactive content stream frame identifiers associated with the respective one or more frames of the video stream (e.g., directly based on the associated frame IDs, or counting from the last frame provided if not for each frame, if appropriate), as described elsewhere herein;

[0167] - the edge server is configured to receive one or more graphical resources (e.g., a streaming logo, a custom border, etc.) from the client device, and the edge server is configured to incorporate at least one of the graphical resources into the shared stream (e.g., by having it synthesized, typically at the same time as the interactive content stream and the camera feed stream are synthesized together), as described elsewhere herein;

[0168] - the edge server is configured to receive an interactive content input command from the client device, and the edge server is configured to forward the interactive content input command to the host server for use by the host server to affect a state of the interactive content (e.g., to control a character in a game and / or a virtual camera position and / or view direction), as described elsewhere herein;

[0169] - in this case, optionally, the edge server is configured to have a graphical representation of the one or more interactive content input commands incorporated into the shared stream (e.g., by being synthesized), as described elsewhere herein;

[0170] - the edge server is configured to receive one or more streaming parameters (e.g., a video feed location and / or scale for compositing with interactive content) from the client device, and the edge server is configured to generate a shared stream based on the one or more received streaming parameters, as described elsewhere herein;

[0171] - the edge server composites the interactive content stream and the camera feed stream together to generate the shared stream (and optionally any graphical assets and / or representations of input commands), as described elsewhere herein;

[0172] - the edge server is configured to provide one selected from a list consisting of the shared stream, the interactive content stream, and the camera feed stream (and optionally any graphical assets and / or representations of input commands) to a separate streaming distribution service in order to distribute the shared stream to one or more spectator devices, as described elsewhere herein;

[0173] - the edge server (optionally equivalent to its nodes or local servers) is configured to distribute the shared stream to one or more spectator devices, as described elsewhere herein;

[0174] - the edge server comprises two or more nodes or local servers respectively assigned to one or more roles selected from a list comprising: receiving interactive content streamed from a host server, receiving a camera feed streamed from a client device, compositing the streams together to generate a shared stream, forwarding interactive content input commands to the host server, incorporating graphical representations of one or more interactive content input commands into the shared stream, providing the shared stream to a third-party streaming distribution service, and providing the shared stream as a streaming distribution service to a plurality of spectator clients, as described elsewhere herein;

[0175] - the edge server is configured to receive a plurality of streams of interactive content, the edge server is configured to receive a plurality of camera feeds streamed from a plurality of client devices, the edge server is configured to receive a compositing command from a compositing director; and the edge server is configured to composite one or more of the streams of interactive content and one or more of the streams of camera feeds together to generate a shared stream in response to the compositing command from the compositing director, as described elsewhere herein;

[0176] - the edge server is further configured to store at least some cached content that is predetermined content of the generated interactive content by the host server, the edge server is configured to receive data from the host server indicating the cached content to be provided by the edge server to the client device, and the edge server is configured to provide the streamed generated interactive content and the indicated cached content to the client device, as described elsewhere herein; and

[0177] - In this case, optionally, the edge server is configured to synthesize the interactive content stream, the indicated cached content, and the camera feed stream together (e.g., by a third party service, or a node of the edge server, or a server local to the edge server) to generate a shared stream (e.g., a stream assigned to a client viewer), as described elsewhere herein.

[0178] Further, it will be appreciated that the interactive content distribution system can include one or more primary servers, one or more additional edge servers, one or more other streaming distribution servers, and / or one or more client devices of viewers of the shared stream.

[0179] It will be appreciated that the interactive content distribution system can be implemented in hardware, suitably adapted by software instructions or by including or replacing dedicated hardware, as applicable, to implement the techniques or methods described elsewhere herein.

[0180] Accordingly, the required adaptations to existing parts of equivalent equipment can be implemented in the form of a computer program product comprising processor implementable instructions stored on a non-transitory machine readable medium, such as a floppy disk, an optical disk, a hard disk, a solid state disk, a PROM, a RAM, a flash memory or any combination of such or other storage media, or in hardware as an ASIC (application specific integrated circuit) or FPGA (field programmable gate array) or other configurable circuit adapted to the adaptation of conventional equivalent equipment. Separately, such a computer program can be transferred via a data signal on a network, such as an Ethernet network, a wireless network, the Internet or any combination of such or other networks.

[0181] Turning now to Figure 6 In the method of interactive content distribution outlined in the summary of the present specification, the method comprises the following steps:

[0182] In a first step s610, the interactive content streamed from the primary server is received at the edge server and streamed to the client device, as described elsewhere herein.

[0183] In a second step s620, data streamed from the client device is received at the edge server, as described elsewhere herein. As previously mentioned, this data stream is typically a camera (12) feed, but can alternatively or additionally include any other source from the streamer that is to be synthesized with the interactive content stream (e.g., a logo, chat data, etc.).

[0184] In a third step s630, the interactive content stream and the camera feed stream are composited together (e.g., by a third party service, or a node of an edge server or a server local to an edge server) to generate a shared stream (e.g., a stream to be distributed to spectator devices), as described elsewhere herein.

[0185] And in a fourth step s640, the shared stream is distributed to one or more spectator devices (e.g., by a third party service, or a node of an edge server or a server local to an edge server), as described elsewhere herein.

[0186] It will be appreciated that the first and second steps can occur in any order or simultaneously, and typically together in a continuous manner.

[0187] It will be apparent to those skilled in the art that variations of the above-described methods corresponding to the operation of the various embodiments of the apparatus as described and claimed herein are considered to be within the scope of the present application, including but not limited to:

[0188] - the method comprises the steps of providing to the client device frame identifiers of at least some frames of the interactive content streamed to the client device, receiving from the client device a frame identifier associated with at least some frames of the camera feed, the frame identifier corresponding to a frame of the interactive content displayed at the client device when the corresponding frame of the camera feed was captured, and compositing together the interactive content stream and the camera feed stream based on the one or more interactive content stream frame identifiers associated with the corresponding one or more frames of the video stream, as described elsewhere herein.

[0189] The foregoing discussion discloses and describes exemplary embodiments of the present application. As will be understood by those skilled in the art, the present application can be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the disclosure of the present application is intended to be illustrative, but not limiting, of the scope of the application and other claims. The disclosure, including any readily discernible variants of the teachings herein, defines and establishes the scope of the preceding claims, such that no inventive subject matter is dedicated to the public.

Claims

1. An interactive content distribution system comprising: an edge server configured to receive interactive content streamed from a host server and to stream the interactive content to a client device; the edge server configured to receive data streamed from the client device; the edge server configured to synthesize an interactive content stream and a data stream together to generate a shared stream; and the edge server configured to cause the shared stream to be distributed to one or more viewer devices.

2. The system of claim 1, wherein, the data streamed from the client device comprises a camera feed.

3. The system of claim 2, wherein: the edge server is configured to provide to the client device frame identifiers for at least some frames of the interactive content streamed to the client device; the edge server is configured to receive frame identifiers in association with at least some frames of the camera feed, the frame identifiers corresponding to frames of the interactive content displayed at the client device when respective frames of the camera feed were captured; and the edge server is configured to cause the interactive content stream and the camera feed stream to be synthesized together with synchronization based on one or more interactive content stream frame identifiers associated with respective one or more frames of the video stream.

4. The system of any preceding claim, wherein: the edge server is configured to receive one or more graphical resources from the client device; and the edge server is configured to incorporate at least one of the graphical resources into the shared stream.

5. The system of any preceding claim, wherein: the edge server is configured to receive interactive content input commands from the client device; and the edge server is configured to forward the interactive content input commands to the host server for use by the host server to affect a state of the interactive content.

6. The system of claim 5, wherein: the edge server is configured to have a graphical representation of one or more interactive content input commands incorporated into the shared stream.

7. The system of any preceding claim, wherein: the edge server is configured to receive one or more streaming parameters from the client device; and the edge server is configured to cause the shared stream to be generated based on one or more received streaming parameters.

8. The system of claim 2 or claim 3, wherein: the edge server synthesizes the interactive content stream and the camera feed stream together to generate the shared stream.

9. The system of any preceding claim, wherein: the edge server is configured to provide to a separate streaming distribution service one selected from a list consisting of: i. the shared stream; and ii. the interactive content stream and the data stream, in order to cause the shared stream to be distributed to one or more viewer devices.

10. The system of any preceding claim, wherein: The edge server is configured to distribute the shared stream to one or more viewer devices.

11. The system of any preceding claim, wherein: The edge server comprises two or more nodes or local servers respectively assigned to one or more roles selected from a list consisting of: i. receiving interactive content streamed from a host server; ii. receiving a camera feed streamed from the client device; iii. compositing the streams together to generate a shared stream; iv. forwarding interactive content input commands to the host server; v. incorporating the graphical representation of one or more interactive content input commands into the shared stream; vi. providing the shared stream to a third party streaming distribution service; and vii. providing the shared stream to a plurality of viewer clients as a streaming distribution service.

12. The system of any preceding claim, wherein: The edge server is configured to receive a plurality of streams of interactive content; The edge server is configured to receive a plurality of camera feeds streamed from a plurality of client devices; The edge server is configured to receive a compositing command from a compositing director; and The edge server is configured to composite one or more of the streams of interactive content and one or more of the streams of camera feeds together to generate a shared stream in response to the compositing command from the compositing director.

13. The system of any preceding claim, wherein: The edge server is further configured to store at least some cached content, the cached content being predetermined content of the interactive content generated by the host server; The edge server is configured to receive data from the host server indicating cached content for provision by the edge server to the client device; and The edge server is configured to provide the client device with the stream generated interactive content and the indicated cached content.

14. The system of claim 13, wherein: The edge server is configured to composite the stream of interactive content, the indicated cached content and the stream of camera feeds together to generate a shared stream.

15. A method of interactive content distribution, comprising the steps of: receiving at an edge server interactive content streamed from a host server and streaming the interactive content to a client device; receiving at the edge server data streamed from the client device; compositing the stream of interactive content and a camera feed stream together to generate a shared stream; and distributing the shared stream to one or more viewer devices. The data streamed from the client device comprises a camera feed.

17. The method of claim 16, comprising the steps of: providing to the client device a frame identifier of at least some frames of the interactive content streamed to the client device; 16. The method of claim 15, wherein, ​ ​ ​ receiving, from the client device in association with at least some frames of the camera feed, frame identifiers corresponding to frames of interactive content displayed at the client device when respective frames of the camera feed were captured; and based on one or more interactive content stream frame identifiers associated with respective one or more frames of the video stream, compositing together the interactive content stream and the camera feed stream with synchronization.

18. A computer program comprising computer-executable instructions adapted to cause a computer system to perform the method of any one of claims 15 to 17.

Images