Video streaming system and method

Abstract

A video streaming method comprises the steps of in a videogame, identifying as an item of interest 310, 320, 330, 340 an in-game element that is within a field of view of a virtual camera of the video

Description

BACKGROUND OF THE INVENTION Field of the invention The present invention relates to a streaming system and method Description of the Prior Art Traditionally, videogames have been played on computers and consoles physically located where the end user is playing, whether this is at home or in an arcade. However, with the advent of fast and responsive internet, and the profusion of smart devices able to stream content from the internet, it has become possible to play videogames on computers and consoles physically remote from the end user, who then relies on a stream to their local smart device (which may itself be a computer or console) to view the video game output (and indeed stream control information back in order to interact with it). Two particular aspects of videogames that are regularly seen as important are firstly their graphical quality (resolution / detail), which provides immersion and visual appeal, and secondly their frame rate, which provides responsiveness, and (for example in the case of head mounted displays) can mitigate nausea. However high quality video and high frame rate video, both separately and together, pushes up network bandwidth when streamed. A limited network bandwidth can therefore impact one or both of the quality and framerate of the streamed videogame. A solution to this problem is foveated rendering, where the user's gaze is tracked, and quality is maintained at the gaze point but drops off elsewhere in the image, reducing the overall bandwidth cost of each image; this allows for a relatively higher frame rate whilst maintaining a subjective appearance of quality for the end user, for the same bandwidth budget. However, foveated rendering can be difficult to implement on a streaming platform due to relative lag between uploading the current gaze position, foveated rendering of the image, and downloading that image - during which time the user's eye position may have changed. This can be mitigated by increasing the foveated region in the render, but this in turn increases bandwidth requirements again. Furthermore, not all end user smart devices are suitable for or capable of gaze tracking; in these cases foveated rendering is not possible. The present invention seeks to mitigate or alleviate this problem. SUMMARY OF THE INVENTION Various aspects and features of the present invention are defined in the appended claims and within the text of the accompanying description. In a first aspect, a video streaming method is provided in accordance with claim 1. In another aspect, a video streaming method is provided in accordance with claim 14. In another aspect, a video streaming server is provided in accordance with claim 16. In another aspect, a video streaming server is provided in accordance with claim 17. In another aspect, a video streaming system is provided in accordance with claim 18. BRIEF DESCRIPTION OF THE DRAWINGS A more complete appreciation of the disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein: Figure 1 is a schematic diagram of a video streaming system in accordance with embodiments of the present description. Figure 2 is a schematic diagram of an image from a videogame for streaming in accordance with embodiments of the present description. Figure 3 is a schematic diagram of an image from a videogame for streaming in accordance with embodiments of the present description. Figure 4 is a flow diagram of a method of video streaming in accordance with embodiments of the present description. Figure 5 is a flow diagram of a method of video streaming in accordance with embodiments of the present description. DESCRIPTION OF THE EMBODIMENTS A video streaming system and method are disclosed. In the following description, a number of specific details are presented in order to provide a thorough understanding of the embodiments of the present invention. It will be apparent, however, to a person skilled in the art that these specific details need not be employed to practice the present invention. Conversely, specific details known to the person skilled in the art are omitted for the purposes of clarity where appropriate. Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views, Figure 1 shows a video steaming system 10 in accordance with embodiments of the present description. The system comprises a video streaming server 100, which may be a computer or console, or comprise a plurality of such computers or consoles, as physical and / or virtual machines, for example for a cloud based videogame streaming service. A computer or console of the server runs a videogame 110 for streaming, and generates a video output. A streaming video encoder 120 then encodes the video output and broadcasts (streams) it over a network 300 (typically the internet) to an end user client device 200. As noted previously the end user client device can be any suitable device, including a smartphone, smart TV, tablet, computer, videogame console, or a controller unit with display capabilities. The system may thus comprise the server alone (e.g. when waiting to stream, or encoding material in advance of streaming), the server and one or more client devices, and / or the server, one or more client devices, and a network (e.g. a local area network, in the case of a e-sports competition or the like). In embodiments of the present description, the streaming video encoder 120 seeks to allocate available bandwidth so that a region of interest (Rol) within the image is allocated relatively more bandwidth than a secondary or peripheral region of the image. As noted elsewhere herein, gaze tracking can be used to identify an Rol for the purposes of foveated rendering, and optionally alternatively or in addition can be used to identify a similar Rol for the purposes of bandwidth allocation. However where gaze tracking is not available (either at all, or temporarily for example due to a camera being occluded), or where it may not be appropriate (for example in local cooperative play with multiple users of the same display) then an alternative strategy for determining a region of interest is required. To a first approximation, the Rol may be assumed to be in the centre of the image; in videogames where users have control over the virtual camera viewpoint, users tend to orient themselves towards their current goal or object of interest within the virtual environment. To a second approximation, the Rol may be based upon analysis of the image, optionally in conjunction with one or more proceeding images; in this case, regions of the image showing greatest change in terms of motion, colour, and / or luminance (once gross motion such as panning has been accounted for) may be considered to indicate one or more regions of interest. Similarly, a so-called saliency map of the image, which identifies the relative degree of importance of some or all pixels in the image to the human visual system, may be generated. Again this typically relates to intra-image changes and rates of change in terms of colour and luminance. These two approaches can be combined to provide a third approximation so that the changes in terms of image content are weighted by their proximity to the centre of the image, thus favouring movement or change close to the centre of the image as the most likely region of interest. In each case, optionally the Rol for the proceeding N images (where N is one or more) may also be used to predict or constrain the Rol in the current image, so that the region maintains a similar size, tracks smoothly and / or avoids jitter when substantially located in the same place or corresponding to the same image element(s); this can serve to reduce unnecessary variability in the region of interest from frame to frame, which in turn can improve both the subjective experience for the user, and the efficiency of the encoder, particularly if it uses inter-frame information in the encoding scheme. However, these approaches are insufficient for some important videogame scenarios. As an example, consider the case where a player is attempting to snipe (shoot) another target player or non-player character ('NPC'), who is hidden behind a specific object; the player will wait for their target to reveal itself, it which point a responsive frame rate and / or an accurate representation of the target will both be desirable for the player in order to shoot the enemy, possibly before the enemy shoots them (or hides back behind cover). However, whilst waiting for the target player or NPC to reveal themselves, the player is effectively looking at a static part of the environment. Furthermore, if the player themselves are seeking shelter and using a 'peek' function or similar to glance at the wider environment, the static part of the environment where the target player is hiding may or may not be near the centre of the player's field of view in the image. As a result, the estimates for the Rol described previously herein cannot be assumed to work; the hidden target player cannot be assumed to be in the centre of view, and does not provide any change in view when hidden, and may not represent the largest change in view if they poke their head out past the object they are hiding behind. As a result it is possible that using such estimates will actually result in this important region of the image being allocated relatively low bandwidth when streamed (and possibly also relatively low quality / resolution if a differentiated rendering scheme is also being used). Clearly this is undesirable. Accordingly, in embodiments of the present description, the videogame 110 and the streaming video encoder 120 interact to provide the streaming video encoder with a better approximation of where the Rol might be. In particular, the videogame has knowledge of the game state, both in terms of what objects (e.g. including players, and / or non-player characters) are in the environment (whether hidden or visible), and in terms of the current goal is or aims of the player either generally in terms of the genre (e.g. multiplayer shooter) or in terms of a specific quest (e.g. find a particular object or person), or in terms of a character build (e.g. melee character, sniper character, healer), and / or in terms of a current action or behaviour (e.g. talking / attaching / hiding). Therefore potentially the videogame can identify what element(s) of the game world are salient to the user's likely current intentions, based on the current game context, and provide a location of the element(s) relative to the current image to the streaming video encoder, as a game context saliency map. Hence in the case of a player attempting to shoot another target player or NPC, the videogame can be set to mark such target players or NPCs as being of interest. Consequently the videogame can identify to the streaming video encoder the location of such a target within field of view encompassed by the current image, even if the target is not fully or partially visible within the image due to full or partial occlusion. As a result, the image streamed to the user will have high quality in the area where the target player is hiding, enabling precise and responsive actions by the current player during game play. In embodiments of the present description objects in the virtual environment are optionally tagged as objects of interest responsive to the current game state. The tag may for example be a flag associated with an object ID, or with a so-called hurt box associated with the object. Tagged objects in the virtual environment may be whole objects in the sense of complete representations of objects (e.g. a ball) or more generally elements of the virtual environment, which may include all or part of one or more such objects. The objects of interest may therefore also more generally be considered items of interest, which may encompass all or part of such objects. However for the purpose of explanation herein, reference is made to identifying objects as objects of interest, whilst the techniques also apply more generally to elements and items as described above. In-game logic may be used to determine whether to set the flag to indicate that an object is of interest. In some cases, it may be inherent in the nature of the object; for example in a multiplayer game, optionally all other players may be considered objects of interest. In other cases, an object may become an object of interest due to meeting one or more in-game conditions or becoming associated with a predetermined attribute. Hence for example in a multiplayer game, optionally only the player associated with a predetermined condition (e.g. currently in possession of a football, currently pointing a weapon at you, and the like), or a predetermined attribute (e.g. currently having the highest score, or currently critically low on health), may be considered an object of interest. In other cases, an object may become an object of interest based on a wider contextual aspect of the game; for example an NPC may not be an object of interest unless a player has selected to follow a particular quest for which the NPC has key information. In this case, typically there will already be logic in place for the NPC to only deliver the relevant information when the quest is being followed; similar logic can be used to selectively flag the NPC as an object of interest. In other cases, an object may become an object of interest based upon a contextual aspect of the current player, or more generally the player's in-game avatar; hence for example if the player has selected a certain character class or in-game faction, or as a member of a particular team, then this may affect what objects within the environment are of interest; for example in a shooting game, members of other factions or teams may be of interest, whereas in a cooperative game, members of the player's own team may be of more interest. It will be appreciated that the determination of whether an object becomes object of interest may be based upon any suitable combination of the above considerations. For example, for a player having a melee-based character class, a non-player character within striking range may be of more interest than a non-player character outside striking range. Meanwhile for a player having a character class that can attack from a distance, such as a sniper, such a distance-based criteria may not be relevant. Similarly for example in a football game there is a mixture of cooperative and competitive play, and the objects of interest may vary dynamically depending on factors such as who has possession of the ball, which players are within unobstructed passing distance of the player, which opponents are running toward the player, the general direction of travel of the ball, and the like. Meanwhile the ball itself may be considered to be inherently of interest. It will therefore also be appreciated that potentially there may be more than one Rol identified within an image; unlike in the case of gaze tracking where the Rol is (only) where the user is looking, in the present embodiments there may be multiple regions of interest that the user is considered to be more likely to be looking at. It will also be appreciated that there is also scope for ranking objects according to relative interest, so that the streaming video encoder can include or exclude objects according to a current bandwidth budget, and / or alter the relative quality accordingly. In this case, a value rather than a flag may be associated with the object as described elsewhere herein. In this way, objects can have an interest value calculated for them, for example according to one or more of their inherent nature, current condition, current behaviour, relation to a contextual aspect of the game, and / or relation to a contextual aspect of the player. Referring now to figure 2, this illustrates an image from a multiplayer videogame of the 'capture the flag' variety. In such games typically two teams each have the aim of carrying a flag to the opposing teams goal; the person carrying the flag is typically a prime target, as they will drop the flag if they are shot. Meanwhile other players on each team seek to either capture the flag, support and protect a teammate carrying the flag, or sabotage the attempts of the opposing team to capture it back. In the image, a first player 250 is running toward the opposing team's goal whilst carrying the flag 260. Meanwhile, a second player 270 is hiding behind a wall. The current player is represented by the point of view of the image, and hence has a clear view of the player with the flag, and a heavily occluded view of the second player. They can also see a health pack 280 in front of the wall but partially occluded by a rock. In this case, the flag may be considered to be of inherent interest within the game. In one example instance, the current player is on the opposing team to the first player, and so attacking the first player will be a primary aim of the current player. As a result, first player 250 is also likely to be of high interest. In this case, if the second player is on the same team as the current player, then whilst they are a competitor for the flag they are not a direct threat and so will be of less interest, but potentially of more interest than the background environment. Conversely, if the second player is on the same team as the flag holder, then they are likely to attack the current player, and so will be of interest; potentially more interest than if they were on the same team as the current player, but potentially of less interest than the current flag holder. Meanwhile in another example instance, the current player is on the same team as the first player, and so defending the first player whilst they are holding the flag will be a primary aim of the current player. In this case, if the second player 270 is on the opposing team then they are a high risk to the first player (and the current player), and so will be of high interest. Finally another example instance, the current player has taken in-game damage, and their health is below a threshold value at which another hit might cause them to die. In this case, due to their health being below the threshold value, the health pack will be of high interest. Hence according to a combination of inherent features, properties of other players and objects, goals of the game, and properties of the current player's character at the time the image is generated, different objects in the same image may have different relative interest to the current player. As noted above, interest may be indicated by a flag associated with the object(s), or a value associated with the object(s). The game 110 may convey interest in an object or objects to the streaming video encoder 120 in several ways. In an embodiment of the present description, an application programming interface ('API') is provided by the game engine or more generally the game. This can pass information relating to regions of interest within the image to the streaming video encoder. In one instance, this may be a mask of similar dimensions to the image, in which rendered elements of the image corresponding to the locations of the objects of interest are represented in the mask by a flag (optionally whether or not the objects themselves are wholly or partially obscured in the image). Where instead the locations of the objects of interest are represented in the mask by a value, then in effect the mask becomes a heat map of interest instead. This information may optionally be gathered for example prior to the so-called Z-culling step, so that occluded or partially occluded objects of interest can also be readily identified. Figure 3 illustrates possible regions of interest associated with the scenarios outlined previously herein. The regions may optionally exactly correspond with the in-game objects; this would be the most efficient implementation in that the minimum image area is identified for high quality, thereby maximising the bandwidth available. In practice, however, this may be less desirable; for example, where the player or NPC 270 is hiding behind the wall, increasing the quality of the silhouette of the NPC (mostly corresponding to wall) would in effect highlight the position, and possibly the pose, of the hidden character. This would be unwelcome in many games, spoiling surprises and, in multiplayer games, possibly amounting to cheating. Accordingly, optionally the Rol 330, 340 may be a larger region encompassing the object of interest, optionally offset (i.e. not centred) on the object, further optionally with a randomly selected offset, in order to create a region of confusion that obscures the precise location of occluded or partially occluded objects whilst still enabling improvement of the quality of image in their location / area. Similarly in practice, objects are both of complex shape and typically of dynamic shape (due to changes in pose and / or virtual camera angle) and position between image frames. Consequently the streaming video encoder 120 may have difficulty efficiently encoding the exact object shape with intra-image encoding schemes and / or inter-image encoding schemes. Typically such schemes are tile or block-based, and may also assume some continuity of shape (or tile / block) between frames. Accordingly, regions of interest for objects of interest may correspond to bounding boxes for the objects, with the box boundaries corresponding to tile or block boundaries in the image. Hence for example if an encoding scheme uses 16x16 pixel tiles or blocks, then regions 31D and 320 (and indeed 330 and 340) may be rectangles whose dimensions and starting points are all multiples of 16 and server to minimally bound the objects of interest. Optionally, they may bound the object with an N-tile deep buffer, where N is 1 or more, so that the object can move into the buffer on the next frame, if the encoding scheme uses inter-frame data and would be more efficient in this case. It will also be appreciated that where an encoding scheme uses tiles or blocks, then the mask or heatmap referred to elsewhere herein can be of a corresponding resolution; hence for an encoding scheme that uses 16x16 pixel tiles or blocks, the mask or heat map can have dimensions 16 times smaller than the image, so that each pixel in the mask or heatmap corresponds to one tile or block. Different encoders / codecs may implement regions of interest in different ways. The API may therefore output information identifying the or each Rol in a manner suitable to the relevant codec. This may take the form of a mask or heatmap as described previously, or a mask or heatmap precomputed to form a region (e.g. a rectangular region) bounding an object of interest, either closely, or with a buffer, or with a wider extent that provides a region of confusion, as described elsewhere herein. Alternatively or in addition, the information may comprise a start position and dimensions for the rectangle (or any predefined shape) of the or each Rol. As noted previously, the output may be binary (is of interest / is not) or value based (an interest score, or ranking), for the mask / heatmap and for rectangular regions, however defined. The encoder / codec can then assess which Rol(s) to maintain at a higher quality than other parts of the image within an encoded version of the image, with reference to a current bandwidth budget (that itself may fluctuate in response to the internet connection and overheads from any other transmitted data considered essential to the functioning of the system and / or game). Hence as a non-limiting example the encoder may evaluate how much of the image is identified as being of interest, and based on the proportion allocate bandwidth to the Rol(s) and the remainder of the image. In one example, the encoder may then encode the whole image to a first quality level as a base layer, and then provide an enhancement layer that increases the quality of the Rol(s). Where the Rol(s) have interest values or rankings, the relative quality of encoding (and / or in this example, the number of enhancement layers) may be responsive to the value or ranking. The encoder stops enhancing the Rol(s) once the bandwidth allocation for the image is met. Optionally, the highest scoring or highest ranking Rol may be encoded or enhanced to at least a minimum target quality, before encoding the next highest scoring or ranking Rol, and so on until the bandwidth allocation for the image is met; in this way the lowest scoring or ranking Rol(s) may not be encoded or enhanced to at least the minimum target quality; rather there is a dynamic cut-off that is responsive to the bandwidth cost of enhancing the more important region(s) of interest in the current image; it will be appreciated that since these can vary in size and / or complexity from image to image, the cost can also vary. Optionally, where the difference in quality between the encoded Rol and the remaining image would exceed a threshold difference, the quality of the background image may be increased so that the image fits the bandwidth allocation with only a maximum disparity in quality between the Rol and the rest of the image. This serves to lift the overall image quality, which acknowledges that the system can only assume, but not know, where the user is actually looking in the image. Alternatively or in addition an intermediate quality boundary around the Rol may be implemented. This also serves to reduce any visual discontinuity produced by a large change in quality between adjacent parts of the image. The above techniques may also be used even when gaze tracking is available; the region indicated by the current gaze may be the primary Rol, but for example where an object of interest coincides at least partially with the gaze Rol, it can be assumed to be what is being looked at; consequently the direction, behaviour, or other change in visual property of the object can be factored into the size and shape of the gaze Rol. Similarly, other objects of interest can still be enhanced, potentially to a lesser extent, to allow for the user glancing to another item in a saccade motion, or (depending on the client device) other viewers also looking at the streamed image. Likewise an object of interest, particularly one at least partially overlapping a gaze position, can be used to predict a gaze position when gaze tracking is momentarily interrupted, for example by the user blinking, or placing a finger over a phone camera. In all the above cases, again optionally the Rol for the proceeding N images (where N is one or more) may also be used to predict or constrain the Rol in the current image, so that the region maintains a similar size, tracks smoothly and / or avoids jitter when substantially located in the same place or corresponding to the same image element(s). The above techniques refer to differentiated encoding quality at the encoder, responsive to one or more regions of interest in turn based upon game context. However, it will be appreciated that this approach may also be used alternatively or in addition for rendering quality (e.g. variable resolution rendering, different quality textures, and / or different quality meshes), with the techniques described herein transposed from encoding to rendering. Referring back to Figure 1, it will be appreciated that the game 110 and the streaming video encoder 120 may be implemented by two separate hardware units, within the same or separate servers. Alternatively they may be implemented within the same hardware unit, for example by re-configuring the video output of a videogame console to generate the encoded stream. Variant embodiments The embodiments herein assume that the game is able to provide an indication of one or more regions of interest in response to a current game state of the videogame, for example via a game or gameengine API. However, for legacy games, or games that do not adopt the techniques herein, it would still be desirable to identify an Rol based on game context. As noted elsewhere herein, it is possible to approximate regions of interest from image data alone, for example to identify regions of the image showing greatest change in terms of motion, colour, and / or luminance (once gross motion such as panning has been accounted for). However, this does not take account of game context per se. Accordingly, in a variant embodiment of the present description, the encoder 120 (or a separate Rol estimation processor, not shown) performs an Rol estimation function, as described below. A trainable image saliency model (for example, a convolutional neural network) is trained using a large scale image database (discussed later herein) to predict game-context based regions of interest in images or image sequences. The convolutional neural network may comprise multiple convolutional layers, with different dilation rates, to capture features at different scales. This may be of particular benefit when regions of interest include partly occluded objects, as per the scenarios in Figures 2 and 3. The large scale image database comprises images generate by games that do implement the game context based Rol techniques described elsewhere herein, and so provide both example input image data and target game context based Rol data. The trainable image saliency model can then learn correlations between these Rols and image data, potentially including a sequence of two or more images, and / or depth, motion vector, and / or other data from the game that may be accessible by the encoder or Rol processor, including any metadata for example relating to some or all of the game state. Hence the image data, Rol data, and any other relevant data generated for streaming according to the techniques herein can be used to train an image saliency model to estimate such Rol data for games that do not generate their own Rol data according to these techniques. The encoder 120 or separate Rol estimation processor can then use a run time instance of the image saliency model to perform the Rol estimation function by outputting an estimate of a region of interest for the rendered image that is (predicted to be) responsive to a current game state of the videogame Optionally, image saliency models may be trained for specific genres of game (so that aspects of the image more salient to that particular genre are more readily learned), or may be trained for specific titles or series; for example the latest game in a series may use the techniques herein, and be used to train an image saliency model for graphically similar and gameplay similar earlier legacy games in the same series. Similarly, an image saliency model may be trained for one title, for example where one computing platform has the computational budget to generate the Rol information (or has a more precise representation of the game environment to do this with), whilst the same title on another computing platform does not, or does not implement these techniques for whatever reason; in this way streamed instances of the game on one platform can train an image saliency model to boost the streamed appearance of the game on a different platform. Further optionally, early access or alpha / beta releases of a game may generate the Rol information, at a potential cost to performance, to train an image saliency model to use with a final release version of the game that does not generate Rol information, or generates simplified Rol information that uses also used as input in the image saliency model. Simplified Rol information may for example comprise metadata describing the current in-game location, and / or the presence and possibly location of one or more key objects in the image. It will be appreciated that the or each image saliency model does not need to be trained whilst the game images are being streamed, but can be trained for example by the same streaming servers when they are not currently streaming and / or generating game outputs. Alternatively or in addition the training may occur on separate systems. The trained image saliency model(s) can then be operated at runtime by the encoder or Rol estimation processor to improve the streaming of games that are not implementing the techniques herein. Separately, whether the Rol is provided by the game, or estimated by a trained saliency model, it may alternatively or in addition be used for other purposes. In particular, it may used to boost the effective resolution / detail of the image in the Rol, for example using a neural network trained to upscale images and provided by the encoder or a upscale processor (not shown) between the game and the encoder. Again, this training can be genre, series, or title specific. This may be particularly effective for legacy games that have lower resolution or detail than more recent titles. By limiting the upscaling to the Rol (and optionally to a lower degree, and region around the Rol) the image quality can be boosted further in a computationally efficient manner (which may be of benefit to a streaming service with many instances of such games to support in parallel), and then encoded preferentially in the same Rol for streaming. Optionally the upscaling neural network may be implemented on the receiving client device, if the Roi information is also transmitted as metadata. This distributes the computational burden of upscaling between server and client. The relevant neural network for upscaling can be transmitted when the user first selects the relevant game. Summary Referring now to Figure 4, in a summary embodiment of the present description, a video streaming method comprises the following steps. In a videogame, in a first step s410 identify as an item of interest an in-game element that is within a field of view of a virtual camera of the videogame (but optionally not visible or only partially visible in the resulting image), in response to a current game state of the videogame, as described elsewhere herein. In a second step s420, output from the videogame to an encoder as a region of interest within an image rendered from the field of view of the virtual camera, a region corresponding to the item of interest, as described elsewhere herein. And at the encoder, in a third step s430 encode the image for streaming, wherein the encoder encodes the image within the region of interest at a higher quality than the image outside the region of interest, as described elsewhere herein. It will be apparent to a person skilled in the art that variations in the above method corresponding to operation of the various embodiments of the apparatus as described and claimed herein are considered within the scope of the present invention, including but not limited to that: the current game state comprises one or more selected from the list consisting of objects within the game's environment (optionally whether visible or wholly or partially hidden from view); a current goal or aim for the player; a genre of the game; an active quest of the game; one or more properties of a current player's in-game character; and one or more properties of an ingame action (or behaviour), as described elsewhere herein; the identifying step comprises using one or more rules selected from the list consisting of whether an element is identified as being of inherent interest in the game; whether an element meets a predetermined in-game condition; whether an element has a predetermined in-game attribute; and whether an element has a specified role in a current quest, to identify an in-game element as an item of interest, as described elsewhere herein; the identifying step comprises selecting an element, responsive to a property of the player's ingame character, to identify an in-game element as an item of interest, as described elsewhere herein; a plurality of elements are identified as items of interest, and the encoding step comprises encoding the image within at least two regions of interest at a higher quality than the image outside these regions of interest, as described elsewhere herein; an interest value or rank is calculated for an item of interest, and the encoding step comprises encoding the image within the region of interest at a quality responsive to the interest value or rank, as described elsewhere herein; at least one of the higher quality within the region of interest and the quality outside the region of interest are selected so that the difference in quality inside and outside the region of interest is less than or equal to a predetermined maximum difference in quality, as described elsewhere herein; the encoding step comprises encoding the image within a bounding region bounding the region of interest at an interim quality lower than the high quality within the region of interest and higher than the quality outside the bounding region, as described elsewhere herein; the region of interest, corresponding to and encompassing the item of interest, is not centred on the item of interest, optionally when the item of interest is wholly, or further optionally partially obscured, or is less than a threshold size in the image, as described elsewhere herein; the region of interest, corresponding to and encompassing the item of interest, has a position and dimensions responsive to the positions and dimensions of encoding blocks or tiles of the encoder, as described elsewhere herein; the outputting step comprises outputting the or each region of interest via an application programming interface from the game to the encoder, as described elsewhere herein; the method comprises the steps of receiving player gaze information, and defining at least a first gaze region of interest responsive to the player gaze information, and where a gaze region of interest and an item of interest at least partially overlap, using the item of interest to predict a gaze region of interest in the event that player gaze information is interrupted, as described elsewhere herein; and the method comprises the step of, at the videogame, rendering the image within the region of interest at a higher quality than the image outside the region of interest, as described elsewhere herein. It will be appreciated that the above methods may be carried out on hardware suitably adapted as applicable by software instruction or by the inclusion or substitution of dedicated hardware. Thus the required adaptation to existing parts of a device may be implemented in the form of a computer program product comprising processor implementable instructions stored on a non-transitory machine-readable medium such as a floppy disk, optical disk, hard disk, solid state disk, PROM, RAM, flash memory or any combination of these or other storage media, or realised in hardware as an ASIC (application specific integrated circuit) or an FPGA (field programmable gate array) or other configurable circuit suitable to use in adapting the conventional equivalent device. Separately, such a computer program may be transmitted via data signals on a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks. Accordingly, in another summary embodiment of the present description, a video streaming server (100) comprises the following. Firstly, a videogame processor (not shown, but typically a CPU of a computer or console) configured (for example by suitable software instruction) to run a videogame (110) and to identify as an object of interest (250, 260, 270, 280) an in-game object that is within a field of view of a virtual camera of the videogame, in response to a current game state of the videogame, as described elsewhere herein. Secondly, an output processor (not shown but typically a CPU and / or GPU of a computer or console) configured (for example by suitable software instruction) to output to an encoder (120) as a region of interest (310, 320, 330, 340) within an image rendered from the field of view of the virtual camera, a region corresponding to the object of interest, as described elsewhere herein. Thirdly, the encoder (120) (typically either a separate device, or a component of the same computer or console as the videogame processor and output processor), configured (for example by suitable software instruction) to encode the image for streaming with a higher quality within the region of interest than for the image outside the region of interest, as described elsewhere herein. Instances of the video streaming server of this summary embodiment implementing the methods and techniques described elsewhere herein (for example by use of suitable software instruction) are also envisaged within the scope of the application. Referring now to Figure 5, in a variant summary embodiment of the present description, a video streaming method comprises the following steps. In a first step s510, in a videogame, generate an image rendered from a field of view of a virtual camera, as described elsewhere herein; In a second step s520, provide at least the rendered image as input to a trained image saliency model, wherein the image saliency model has been previously trained on similar inputs from a videogame, and also on target outputs generated by the videogame that indicate a region of interest in response to a current game state of the videogame, as described elsewhere herein; In a third step s530, obtain an output from the trained image saliency model that estimates a region of interest for the rendered image that is responsive to a current game state of the videogame, as described elsewhere herein; In a fourth step s540 output to an encoder the region of interest within the rendered image, as described elsewhere herein; and In a fifth step s550 at the encoder, encode the image for streaming, wherein the encoder encodes the image within the region of interest at a higher quality than the image outside the region of interest, as described elsewhere herein. It will be apparent to a person skilled in the art that variations in the above method corresponding to operation of the various embodiments of the apparatus as described and claimed herein, for example in the context of the previous summary embodiment, are considered within the scope of the present invention. Similarly, it will again be appreciated that the above methods may be carried out on hardware suitably adapted as applicable by software instruction or by the inclusion or substitution of dedicated hardware. Accordingly, in a variant summary embodiment of the present description a video streaming server (100), comprises the following. Firstly, a videogame processor (not shown, but typically a CPU of a computer or console) configured (for example by suitable software instruction) to run a videogame (110) that generates an image rendered from a field of view of a virtual camera, as described elsewhere herein; Secondly, an estimation processor (not shown, but typically a CPU of a computer or console, or corresponding to the encoder 120) configured (for example by suitable software instruction) to provide at least the rendered image as input to a trained image saliency model, wherein the image saliency model has been previously trained on similar inputs from a videogame, and also on target outputs generated by the videogame that indicate a region of interest in response to a current game state of the videogame, as described elsewhere herein, Wherein the estimation processor is configured (for example by suitable software instruction) to obtain an output from the trained image saliency model that estimates a region of interest for the rendered image that is responsive to a current game state of the videogame; Thirdly, an output processor (not shown, but typically a CPU of a computer or console) configured (for example by suitable software instruction) to output to an encoder (120) the region of interest within the rendered image; And fourthly the encoder 120, configured (for example by suitable software instruction) to encode the image for streaming, wherein the encoder encodes the image within the region of interest at a higher quality than the image outside the region of interest. Instances of the video streaming server of this variant summary embodiment implementing the methods and techniques described elsewhere herein (for example by use of suitable software instruction) are also envisaged within the scope of the application. It will be appreciated that embodiments of both the summary embodiments and variant summary embodiments may be combined as appropriate, for example to accommodate streaming both of games that do implement the techniques herein, and legacy games that do not. Hence also referring again to Figure 1, a video streaming system (10) may comprises a video streaming 5 server (100) according to either summary embodiment, and as described elsewhere herein; and at least a first client device (200) configured to receive and decode the encoded image. The foregoing discussion discloses and describes merely exemplary embodiments of the present invention. As will be understood by those skilled in the art, the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, 10 the disclosure of the present invention is intended to be illustrative, but not limiting of the scope of the invention, as well as other claims. The disclosure, including any readily discernible variants of the teachings herein, defines, in part, the scope of the foregoing claim terminology such that no inventive subject matter is dedicated to the public. 15

Claims

1. A video streaming method, comprising the steps of:in a videogame, identifying as an item of interest an in-game element that is within a field of view of a virtual camera of the videogame, in response to a current game state of the videogame;outputting to an encoder as a region of interest within an image rendered from the field of view of the virtual camera, a region corresponding to the item of interest; andat the encoder, encoding the image for streaming, whereinthe encoder encodes the image within the region of interest at a higher quality than the image outside the region of interest.

2. The video streaming method of claim 1, in which the current game state comprises one or more selected from the list consisting of:i. objects within the game's environment;ii. a current goal or aim for the player;iii. a genre of the game;iv. an active quest of the game;v. one or more properties of a current player's in-game character; andvi. one or more properties of an in-game action.

3. The video streaming method of claim 1 or claim 2, in which the identifying step comprises using one or more rules selected from the list consisting of:i. whether an element is identified as being of inherent interest in the game;ii. whether an element meets a predetermined in-game condition;iii. whether an element has a predetermined in-game attribute; andiv. whether an element has a specified role in a current quest,to identify an in-game element as an item of interest.

4. The video streaming method of any preceding claim, in which the identifying step comprises:selecting an element, responsive to a property of the player's in-game character, to identify an in-game element as an item of interest.

5. The video streaming method of any preceding claim, in which a plurality of elements are identifying as items of interest; andthe encoding step comprises encoding the image within at least two regions of interest at a higher quality than the image outside these regions of interest.

6. The video streaming method of any preceding claim, in which an interest value or rank is calculated for an element of interest; andthe encoding step comprises encoding the image within the region of interest at a quality responsive to the interest value or rank.

7. The video streaming method of any preceding claim, in which:at least one of the higher quality within the region of interest and the quality outside the region of interest are selected so that the difference in quality inside and outside the region of interest is less than or equal to a predetermined maximum difference in quality.

8. The video streaming method of any preceding claim, in which:the encoding step comprises encoding the image within a bounding region bounding the region of interest at an interim quality lower than the high quality within the region of interest and higher than the quality outside the bounding region.

9. The video streaming method of any preceding claim, in which:the region of interest, corresponding to and encompassing the item of interest, is not centred on the item of interest.

10. The video streaming method of any preceding claim, in which:the region of interest, corresponding to and encompassing the item of interest, has a position and dimensions responsive to the positions and dimensions of encoding blocks or tiles of the encoder.

11. The video streaming method of any preceding claim, in which the outputting step comprises: outputting the or each region of interest via an application programming interface from the game to the encoder.

12. The video streaming method of any preceding claim, comprising the steps of:receiving player gaze information, and defining at least a first gaze region of interest responsive to the player gaze information; andwhere a gaze region of interest and an item of interest at least partially overlap, using the item of interest to predict a gaze region of interest in the event that player gaze information is interrupted.

13. The video streaming method of any preceding claim, comprising the step of:at the videogame, rendering the image within the region of interest at a higher quality than the image outside the region of interest.

14. A video streaming method comprising the steps of:in a videogame, generating an image rendered from a field of view of a virtual camera;providing at least the rendered image as input to a trained image saliency model, whereinthe image saliency model has been previously trained on similar inputs from a videogame, and also on target outputs generated by the videogame that indicate a region of interest in response to a current game state of the videogame;obtaining an output from the trained image saliency model that estimates a region of interest for the rendered image that is responsive to a current game state of the videogame;outputting to an encoder the region of interest within the rendered image; andat the encoder, encoding the image for streaming, whereinthe encoder encodes the image within the region of interest at a higher quality than the image outside the region of interest.

15. A computer program comprising computer executable instructions adapted to cause a computer system to perform the method of any one of the preceding claims.

16. A video streaming server (100), comprising:a videogame processor configured to run a videogame (110) and to identify as an item of interest (250, 260, 270, 280) an in-game element that is within a field of view of a virtual camera of the videogame, in response to a current game state of the videogame;an output processor configured to output to an encoder (120) as a region of interest (310, 320, 330, 340) within an image rendered from the field of view of the virtual camera, a region corresponding to the item of interest; andwherein the encoder is configured to encode the image for streaming; andthe encoder is configured to encode the image within the region of interest at a higher quality than the image outside the region of interest.

17. A video streaming server (100), comprising:a videogame processor (110) configured to run a videogame that generates an image rendered from a field of view of a virtual camera;an estimation processor configured to provide at least the rendered image as input to a trained image saliency model, whereinthe image saliency model has been previously trained on similar inputs from a videogame, and also on target outputs generated by the videogame that indicate a region of interest in response to a current game state of the videogame,the estimation processor being configured to obtain an output from the trained image saliency model that estimates a region of interest for the rendered image that is responsive to a current game state of the videogame;an output processor configured to output to an encoder (120) the region of interest within the rendered image; andat the encoder, encoding the image for streaming, whereinthe encoder encodes the image within the region of interest at a higher quality than the image outside the region of interest.

18. A video streaming system (10), comprising:a video streaming server (100) according to claim 16 or claim 17; andat least a first client device (200) configured to receive and decode the encoded image.18

Images