Rendering method and system

Abstract

A rendering method for a streaming server comprises the steps of rendering a first image for streaming at a predetermined quality and estimating locations of one or more regions of the first image hav

Description

The present invention relates to a rendering method and system. In cloud gaming, a videogame runs on a real or virtual videogame console as part of a server / cloud system that then streams the outputs of the video game to a respective client, whilst relaying uploaded inputs from that client to the console. The advantage of this approach is that the client device does not need to have the same computing or graphical capabilities as the console in order to still play the videogame; it just needs to be able to stream the resultant video / audio and upload the user's inputs. However, it does mean that the client device, and more broadly the client / server system, is dependent upon the quality of the network between them and the quality of the encoding used to transmit the video. Embodiments of the present description seek to mitigate or alleviate this dependency. SUMMARY OF THE INVENTION Various aspects and features of the present invention are defined in the appended claims and within the text of the accompanying description. In a first aspect, a rendering method for a streaming server is provided in accordance with claim 1. In another aspect, a streaming server is provided in accordance with claim 16. In another aspect, a streaming system is provided in accordance with claim 17. BRIEF DESCRIPTION OF THE DRAWINGS A more complete appreciation of the disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein: Figure 1 is a schematic diagram of an entertainment device representative of a server or a client in accordance with embodiments of the present description. Figure 2 is an illustration of an image for encoding and streaming. Figures 3A - 3d are examples of image errors caused by encoding. Figure 4 is a schematic diagram of stages in a rendering pipeline in accordance with embodiments of the present description. Figure 5 is a flow diagram of a rendering method for a streaming server in accordance with embodiments of the present description. DESCRIPTION OF THE EMBODIMENTS A rendering method and system are disclosed. In the following description, a number of specific details are presented in order to provide a thorough understanding of the embodiments of the present invention. It will be apparent, however, to a person skilled in the art that these specific details need not be employed to practice the present invention. Conversely, specific details known to the person skilled in the art are omitted for the purposes of clarity where appropriate. Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views, Figure 1 shows an example of an entertainment system 10 such as a computer or console. The entertainment system 10 may be considered an example of either a server / cloud device, or a client device, depending on how it is configured. The entertainment system 10 comprises a central processor or CPU 20. The entertainment system also comprises a graphical processing unit or GPU 30, and RAM 40. Two or more of the CPU, GPU, and RAM may be integrated as a system on a chip (SoC). Further storage may be provided by a disk 50, either as an external or internal hard drive, or as an external solid state drive, or an internal solid state drive. The entertainment device may transmit or receive data via one or more data ports 60, such as a USB port, Ethernet® port, Wi-Fi® port, Bluetooth® port or similar, as appropriate. It may also optionally receive data via an optical drive 70. Audio / visual outputs from the entertainment device are typically provided through one or more A / V ports 90 or one or more of the data ports 60. Where components are not integrated, they may be connected as appropriate either by a dedicated data link or via a bus 100. Examples of a device for displaying images output by the entertainment system include a head mounted display 'HMD' 120 worn by a user 1, a TV (not shown), and a portable screen 140. Interaction with the system is typically provided using one or more handheld controllers 130, 140, and / or one or more VR controllers (130A-L,R) in the case of the HMD. It will be appreciated that as a client device, the entertainment device may not require further storage, an optical drive, or the like, if the game that might otherwise use them is being streamed to the client. Conversely, as a server / cloud device, the entertainment device typically will not be operably coupled to a display device, or receiving inputs directly from a controller. In this latter case, the inputs from the client's controller may be received either by an OS or helper app of the entertainment device, or by a further element of the server / cloud device, and then provided to the entertainment device or the game running thereon as if input locally. Lossy Codecs Meanwhile the video and optionally audio output by the entertainment device is provided to an encoder, for streaming to the client, rather than being output to a display. In embodiments of the present description, the encoder is a lossy encoder. Lossy encoders typically attempt to retain equivalent visual quality to the source image when there is sufficient bandwidth to do so, but successively degrade quality within the image when bandwidth is limited. This is typically done so that the frame rate of the video comprising the images can be maintained even as the quality of the images in the video drops. Examples of lossy video codecs include H.261, H.264, HEVC / H.265, MPEG-2, and MPEG-4, among others. The losses introduced by such codecs can take many forms - a reduction in colour depth or palette size, a reduction in effective resolution, a reduction of high spatial frequency components, increased quantisation of image data, and the like. Similarly, these losses can occur to different degrees in different parts of an image. Codecs often work on a tile-by-tile basis, where the tiles may be regularly sized macroblocks or similar, or may be differently sized (and / or potentially differently shaped). If compressing to a bandwidth budget, some tiles may need more compression to meet a target budget, or equivalently some tiles may be more greatly affected by a compression technique (for example a tile containing a lot of high frequency components may be particularly affected by compression schemes that rely on reducing such components). In any event, lossy encoding can result in a variety of different forms of image degradation, to various degrees, within an encoded image. Ideally, when there is sufficient bandwidth available, the losses may be negligible or hard to detect visually, but may become an issue if there is insufficient bandwidth. What counts as sufficient bandwidth or insufficient bandwidth may in turn vary depending for example on the resolution and / or framerate of the streamed video. Turning now to Figure 2, this is an example of a video image generated at the server for subsequent encoding. The image comprises a mix of plain, shiny, and specular surfaces, and some basic colours. Turning next to Figures 3A-D, these show one section of the image in more detail. Figure 3A shows the original image. Figure 3B shows the original image but with a reduced colour depth (as may occur for example due to quantisation) - in this case, the character's eyes start to look unevenly coloured, and the specular surface looks more gritty. In addition, colour gradation between elements (e.g. due to antialiasing) is significantly reduced. Meanwhile areas of the image with simple or plain textures (e.g. the black parts of the character's face panel, and the background) are largely unaffected. Figure 3C shows the same image but with high frequency components of the image reduced (as may occur in many frequency-domain based codecs). The loss of high frequency components particularly affects sharp (spatially compact / high frequency) transitions, such as between the character's antenna and background, or some parts of the character's face panel and the adjoining regions. The specular surface has lost a lot of definition, as it is a high frequency texture. Meanwhile colour gradation between elements (e.g. due to antialiasing) is less affected. Finally, some artefacts may be introduced within otherwise plain regions (for example the torso of the character), typically because a flat texture may in fact be the summation of several different frequency components, and if a high frequency component is removed then in effect its contribution to the flatness of that texture is lost. Finally, Figure 3D shows the original image but with reduced resolution (as may occur in some codecs, or where upscaling is required at the client side). Again this reduces high frequency components but does so uniformly / independently of the content and so the whole image looks blurred. In this case, areas of the image with simple or plain textures (e.g. the black parts of the character's face panel, and the background) are largely unaffected, whilst the edges of objects and fine details are obviously reduced in quality. Hence different parts of the image may be affected differently by any one codec, and different codecs may affect a part of an image in different ways. Codec correction Conventionally, some codecs attempt to correct for this using an approach similar to the following: Upon encoding the original image, the server then decodes it again as if it was the client device, to obtain the version of the image that may be displayed by the client. The server then compares this decoded version of the image to the original image (for example by simple subtraction of respective pixel values) to determine an error correction image. This image typically has a much smaller value range over most of its area (e.g. most of the image does not have large errors) and so can be efficiently quantised or otherwise compressed and sent as a separate error correction layer that can itself be decoded and added back onto the original decoded image to improve its overall quality. Optionally such a server can then also compare a decoded error-corrected version of the image with the original image to identify a second-order error correction image for encoding to further correct the image, and potentially so-on, to create multiple error-correction or enhancement 'layers' that can be transmitted to the client when bandwidth permits to improve the image quality. As bandwidth increases or decreases, the number of layers can increase or decrease. This strategy can be used, for example, in the H.264 codec. Rendering method Embodiments of the present description seek to make use of any suitable techniques to locate one or more regions of the original image having errors caused by encoding for streaming, such as the comparison example above; e.g. a comparison between the original image and the image eventually displayed by the client device (whether or not enhancement layers are part of the current codec's approach, though clearly if used then the comparisons are already available). Any other techniques, for example training a machine learning system to identify image features that tend to subsequently be encoded / compressed with errors, may also be considered. Accordingly, in embodiments of the present description, the server generates the encoded version of the currently output image from the console, for streaming to the client device. The server then performs its own decoding of the encoded image, using the same decoder as the client device would use, to generate the decoded image that would be displayed by the client. This decoded image may take account of any enhancement layers etc., if used, that are being sent in response to current network bandwidth conditions. More generally, the server generates an estimated equivalent of the image that is displayed by the client device according to whatever encoding / decoding approach was used for that image. The server then compares this with the original image. As noted above, for some codecs this comparison may already be available. The server is then able to detect (e.g. based on the residual error values) where in the image the codec has effectively reduced the quality of the original image. This information may then be used by the videogame console to alter how it generates the next image for output. In particular, it can reduce processing / memory resources directed to areas of the last image that were, or were most, reduced in quality. In other words, where the codec reduces image quality in a preceding original image, the videogame console can reduce image quality in the subsequent original image, on the basis that this reduction will not be noticed as it would have occurred in the codec anyway. The reduction in image quality may be proportional to the residual error values in the in the comparison, so that small error values result in small reductions in quality, or only error values above a threshold amount result in corresponding reductions in quality. Depending on how reductions in image quality are implemented in the generated image, these may allow for continuous or small-step reductions, proportional to codec based error, or may for example relate to disabling a rendering feature or making large stepwise reductions in such a feature if the effect of this would be masked by the codec's reduction in image quality. Referring now to Figure 4, the generation of an original image by the videogame console typically involves some or all of the following elements of a rendering pipeline. After the game logic (game state) of the game has been updated, the pipeline comprises some or all of the following (and potentially other steps not included here), in roughly, but not necessarily, the order given. Geometry update (410) The geometry of the environment to be rendered may be updated if there are new elements to show, or if elements have changed. Similarly, changes in character animation / pose may be defined. Consequently, where for example the edges of an object within an image are reduced in quality, then the number of polygons defining that edge may optionally be reduced for the next image Z-buffer update (420) What aspects of the environment are visible is updated depending on the geometry and the virtual camera viewpoint - parts that are unseen need not be processed, or only processed to a limited extent. This would typically not be altered in response to changes in image quality, although a reduction in polygon count in the geometry update could change what is visible at the edges of the affected object(s). Texturing (430) Texture maps are used to apply basic graphical imagery to the geometry of the scene. Typically different textures are used as a function of distance from the viewpoint. In regions where the image quality is reduced, a lower quality version of the texture may be used - for example by changing the so-called LOD (level of detail) in that region, and / or using a lower resolution texture for example from a series of equivalent textures in a mip-map. In regions where colour depth / palette size is reduced, textures may be simplified (e.g. truncating lowest significant bits of the colour channel(s)), if this provides a net benefit to memory and / or computing resources. Similarly of a codec results in a blocky region, a texture with a lower effective resolution (e.g. a similar effective resolution to that of the resulting blockiness) may be selected. Static Lighting (440) Lightmaps may be used for direct and indirect lighting, shadows, and reflections. In regions where the image quality is reduced, a lower quality version of the light map may be used, or used for shadows, for example. Equivalently, lower sampling rates for lighting may be used in such regions. Where lighting is used to enhance textures (or at any other point in the rendering process), e.g. for bump-mapping or normal mapping, this can also be reduced or disabled for example in areas where texture quality is also reduced. Dynamic lighting (450) Direct and indirect lighting, shadows, and reflections may be computed for dynamic in-game light sources such as a player's torch. Again in regions where the image quality is reduced, either lower quality mapping or sampling may be used. Atmospherics (460) Effects such as mist or fog, particle effects, translucency (water) and the like. In regions where the image quality is reduced, the processing for these effects may optionally be reduced for example by reducing sampling density of volumetric effects, or simplifying water reflections or the like. As part of or separate to atmospherics, physic simulations relating to interactions between scene elements may similarly be simplified or disabled. Post processing (470) A variety of effects that may be applied, often to the image as a whole, including bloom, exposure, depth of field bokeh, lens effects, and the like. Again in regions where the image quality is reduced, such post processing effects may also be simplified. Similarly anti-aliasing effects applied to the image at this stage or any other can be simplified or disabled for such regions responsive to the degree of error in the decoded image. Upscaling (480) If, in order to achieve a preferred frame rate, the rendering has been performed at a lower resolution than intended, some systems have a machine learning based Al upscaling step to inpaint the rendered image up to the target resolution. Optionally this may be unaffected by changes in quality between the original and decoded image. Alternatively or in addition, if the image or a region thereof has been rendered at a low resolution, only for a sub-set of pixels, and / or with lower textures, then cloning or interpolating between these provides a simpler but lower quality alternative to texturing or rendering at the default quality. This may be useful if the results of the codec are blocky (e.g. a low effective resolution) In each case above, optionally the degree of reduction in rendering quality is coupled to the degree of error in the decoded image, e.g. for the respective regions of the next image. It will be appreciated that one or more steps in the rendering pipeline may be adjusted in response to errors in an image, either in different parts of the image and / or to the same parts. Notably, the above approach provides several benefits. Firstly, in regions where errors have occurred, the above adjustment(s) simplify the resulting image to be encoded. This in turn can make the codec operate more efficiently on the simplified regions, and in turn can result in more bandwidth being available to encode the unaffected or higher quality regions, resulting in an overall better image quality in the eventually decoded image. Similarly, computational and memory overhead saved by simplifying rendering steps in some regions of the image can allow for more computation and memory resource to be used in other regions of the image for the same overall computation and memory budget in the generation of an image frame. Again this can result in overall better image quality in the eventually decoded image. Alternatively or in addition it can enable an increased frame rate if the image is generated more quickly. Image motion To a first approximation, the regions of the previous image that had residual decoding image errors (e.g. above a predetermined actionable threshold) can be re-used as the regions to simplify. As noted elsewhere these regions can be pixel based or tile-based. Where tile based, optionally the error threshold can be on a per-tile basis to estimate how badly the image within that tile is affected. To a second approximation, it can be assumed that residual decoding image errors are in large part due to what is being displayed at those positions, and so an improved approach is to track where those displayed regions move between the previous frame and the new frame to be rendered. It will be appreciated that there is a problem inherent in this improved approach, which is that for an image-based comparison of where corresponding image elements have moved between frames (e.g. so-called optical flow or motion vector analysis), both the old and new images are required, and required to be in a comparable state - however, the purpose of calculating the movement is to determine what parts of the image to generate more simply, i.e. before the new image is complete. This makes the approach difficult. However, this can be addressed in several ways. In a first instance, a motion trajectory can be calculated based on the previous image and one or more earlier images, e.g. using a Kalman filter to predict motion for the new image. In a second instance, at least approximate motion can be compared using early stages of the rendering process, such as the virtual camera position and geometry of the scene. Assuming the texturing and lighting of the image are roughly similar frame-to-frame, then gross motion in the image can be computed. In a third instance, tracking the position of the decoding errors in two or more prior images allows prediction of where the errors will occur in the new image. Typically this will be a simpler process than either of the first two instances, because the errors will be more sparse, particularly if thresholded. Hence for example the positions of errors exceeding a testing threshold can be compared between two or more preceding frames to determine motion, and then errors contiguous with these positions or within a threshold distance can be assumed to move in a similar manner. The use of a threshold can be optional, or can be lowered until a representative error position is found for each region of the image comprising actionable errors (e.g. errors optionally greater than an actionable threshold at which it is useful to simplify the image). If errors contiguous with test positions are used, then only a subset of pixels in each region need be tracked in this manner, making the process even less computationally expensive. In any event, by predicting where image elements that are difficult to encode without threshold levels of loss / error are going to be in the new image, the regions corresponding to the elements can be better identified for simplification of rendering. As noted above with reference to the rendering pipeline, different types of error may prompt different simplifications, for example depending on whether the error relates to colour depth, spatial frequency range, resolution, artefacts, or the like. Variants Having simplified the rendering of at least a first region of the new image, then that image is encoded and transmitted, and also the residual error of that image is calculated as described elsewhere herein so as to identify where to simplify the following image. However, it can be expected that simplification of the new image results in the error in that simplified region being reduced - as such, potentially the system may then estimate that the region (or the image element in that region) is not being significantly affected by the codec (e.g. to a threshold degree), and not simplify it for the subsequent image - this can result in an oscillating condition of large and small errors resulting in alternating simplifications of the image region. This is clearly not desirable. Conversely if the transition from original to simplified regions (or the simplified region itself) create or retain errors, this can trigger yet further simplification either of the existing simplified regions or those bordering them so that the image degrades ever further in a 'race to the bottom'. For example, with reference again to Figure 3C, frequency domain based codecs can introduce artefacts in simple regions of an image, and can have difficulties representing discontinuities in an image, both of which may occur in response to simplification of an image region. To a first approximation one solution to both problems is to provide hysteresis, and assume persistence of an initially detected issue with the original image at least whilst network conditions and / or the image element being simplified remain similar to within predetermined thresholds. However, this in turn can potentially also result in a different 'race to the bottom', where new detected errors result in persistent image simplification due to hysteresis, with no clear mechanism to restore them other than a change in network conditions or the image element disappearing or changing (for example when a character walks off screen, or turns around). As a result progressively more of the image may be simplified or simplified further over time. This is also not desirable, and an image-based mechanism to reset simplification may also be desired. Hence to a second approximation, an alternative (or additional) solution is to retain a 'canary' region at full quality that would otherwise be simplified; this may be a sub-region of a wider region being simplified, or a region that has relatively little error (so keeping it is not too costly) but enough error to reliably detect a subsequent decrease of error. The associated error threshold may be determined empirically. Then, if the error in the or each canary region (or a threshold proportion thereof) drops, this is indicative that conditions have improved (without needing to determine exactly where or how), and the simplification of image elements can be stopped or optionally alternatively scaled down proportional to the improvement - for example regions of the image with the lowest original actionable error are not simplified; if the resulting new error is lower, then regions with higher original actionable errors are not simplified, and so-on until, if conditions allow, no image regions are being simplified. Here, reference to the original actionable error means to the error value that triggered simplification of the image elements - subsequent error values for the simplified image may not be representative of the error that would occur if another original image is encoded. In a similar implementation to the second approximation, if the error in the or each canary region (or a threshold proportion them) drops below the actionable threshold, then either the simplification may be turned off for all regions, or optionally the effective actionable threshold for the other regions can be progressively raised so that regions are progressively output in original form depending on how bad the error for that region was initially. Optionally regions that revert to original form can be treated as additional or replacement canaries. Again if conditions allow, eventually all regions are output as original and not simplified. Alternatively or in addition, an original image can be transmitted periodically; for example if the codec uses I and P frames or similar, the I frame may be original so that each group of pictures starts with original error values. In some cases, simplification, or some types of simplification, may not be desirable even if the original image quality is reduced by the encoding-decoding process. In particular, the areas of change between subsequent images for animation and inter-frame motion within the images is often affected by codec degradation, but at least some of these regions are likely to be of particular importance to the user - for example the movement of their own avatar, or of an enemy within or near their gunsights. Accordingly, the evaluation of whether to simplify a region may also be subject to whether the region or part thereof exceeds a saliency threshold. In this case the measurement of saliency can comprise one or more elements including one or more selected from the list consisting of whether the region comprises or corresponds to a predetermined object (e.g. the player character, or an object directly interacted with by the player), whether the object is in a particular area (e.g. within or within a threshold distance of a targeting reticule, or a goal or event triggering position, or the like), whether the object is engaged in a predetermined animation (e.g. drawing a weapon), and whether the object or region is otherwise indicated to be a focus of user attention, e.g. due to a mouse or cursor position, or due to tracking user gaze. If such a regions exceeds the saliency threshold, then it may not be simplified, or it may only be simplified in ways that do not effect a particular property of the regions or object, such as motion, colour, resolution or the like. The property or properties to conserve may vary depending on the nature of the object / region. For example the edges of the player avatar may be important to preserve so that the user can gauge whether they are going to touch a hazard, and so a region comprising these may for example avoid simplification by reducing high frequency components, geometry, or resolution, but perhaps could still reduce colour depth and texture complexity. It will also be appreciated that in a similar manner to selecting or ruling out simplifications in response to properties of objects / regions of the image, it is also possible to select or rule out simplifications in response to types of or properties of the detected error. Hence for example where colour bleed or colour quantisation is an error, it may be preferable to reduce colour depth to simplify the image rather than for example reduce the resolution. Simplifications corresponding to different sources of image error will be apparent to the skilled person. Summary Turning now to Figure 5, in a summary embodiment of the present description, a rendering method for a streaming server comprises the following steps. In a first step s510, rendering a first image for streaming at a predetermined quality, as described elsewhere herein. In a second step s520, estimating locations of one or more regions of the first image having at least a first error caused by encoding for streaming, as described elsewhere herein And in a third step s530, rendering a subsequent image for streaming, wherein the subsequent image is rendered at a lower quality than the predetermined quality in at least a region corresponding to an estimated location (e.g. a located error in an estimated received first image), as described elsewhere herein. It will be apparent to a person skilled in the art that variations in the above method corresponding to operation of the various embodiments of the apparatus as described and claimed herein are considered within the scope of the present invention, including but not limited to that: The subsequent image is rendered at a lower quality than the predetermined quality in at least a region corresponding to an estimated location (e.g. a located error in the estimated received first image) that exceeds a predetermined error threshold, as described elsewhere herein; the or each region comprises one or more selected from the list consisting of an object in the image, a tile or grid region of the image, and a region of the image subject to a discrete encoding step (e.g. a macroblock), as described elsewhere herein; the image quality in a region is lowered proportional to a level of the corresponding error in the estimated received first image, as described elsewhere herein; this lowering may be due to reducing a quality in one or more rendering steps, potentially by different respective amounts up to an including not performing the respective step. at least a first region of the subsequent image is rendered at a lower quality by altering one or more selected from the list consisting of geometry rendered for the subsequent image, texture quality for the subsequent image, lighting quality for the subsequent image, atmospheric, volumetric, or physics quality for the subsequent image, post processing effects for the subsequent image, and resolution or upscaling for the subsequent image, as described elsewhere herein; in this instance, optionally one or more types of alteration are selected in response to a type of error in the estimated received image, as described elsewhere herein; in this case, optionally types of error include one or more selected from the list consisting of altered colour depth or gamut, altered real or effective resolution, altered spatial frequency distribution, blockiness, and altered edge sharpness, as described elsewhere herein; the method comprises the step of, for a region corresponding to an estimated location for the first image (e.g. a located error in the estimated received first image), estimating the corresponding location of that region in the subsequent image, as described elsewhere herein; in this instance, optionally the step of estimating the location of that region in the subsequent image comprises estimating a motion vector for the region based on its location in the first image and a preceding image (for example based on tracking motion in the image or motion in the location of errors or regions), as described elsewhere herein; similarly in this instance, optionally the step of estimating the location of that region in the subsequent image comprises estimating a change of location based upon one or more selected from the list consisting of a change in geometry and a change in virtual camera position, between the first image and the subsequent image, wherein the estimation occurs prior to a rendering step of the subsequent image that is to be rendered at a lower quality than the predetermined quality for that region, as described elsewhere herein; the step of rendering the subsequent image at a lower quality than the predetermined quality in at least a region corresponding to an estimated location for the first image (e.g. a located error in the estimated received first image) is repeated for further images until one or more selected from the list consisting of an image element in that regions changes by more than a threshold amount, a network condition improves by more than a predetermined amount, and a predetermined period of time has elapsed, as described elsewhere herein; the step of rendering the subsequent image at a lower quality than the predetermined quality in at least a region corresponding to an estimated location for the first image (e.g. a located error in the estimated received first image) comprises retaining a sub-region of that region at the predetermined quality; and the method comprises the steps of evaluating a decoding error in the sub-region, and if the error in the sub-region drops below the predetermined error threshold, improving the rendering quality in that region as a whole in the next rendered image, as described elsewhere herein; the step of rendering the subsequent image at a lower quality than the predetermined quality in at least a region corresponding to an estimated location for the first image (e.g. a located error in the estimated received first image) comprises retaining a sub-region of that region at the predetermined quality, the sub-region being selected according to a salience of an image element within the sub-region, as described elsewhere herein; and the step of estimating locations of one or more regions of the first image having at least a first error caused by encoding for streaming comprises the steps of encoding the first image for transmission to a client, decoding the encoded image as transmitted to generate an estimated received first image, and comparing the first original image to the estimated received image to locate errors in the estimated received first image, as described elsewhere herein. It will be appreciated that the above methods may be carried out on hardware suitably adapted as applicable by software instruction or by the inclusion or substitution of dedicated hardware. Thus the required adaptation to existing parts of an equivalent device may be implemented in the form of a computer program product comprising processor implementable instructions stored on a non-transitory machine-readable medium such as a floppy disk, optical disk, hard disk, solid state disk, PROM, RAM, flash memory or any combination of these or other storage media, or realised in hardware as an ASIC (application specific integrated circuit) or an FPGA (field programmable gate array) or other configurable circuit suitable to use in adapting the conventional equivalent device. Separately, such a computer program may be transmitted via data signals on a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks. Accordingly, and referring again to Figure 1, in a summary embodiment of the present description a streaming server (10) comprises the following. A graphics processor (e.g. GPU 30 and / or CPU 20) configured (for example by suitable software instruction) to render a first image for streaming at a predetermined quality, as described elsewhere herein. An error location estimation processor (e.g. GPU 30 and / or CPU 20) configured (for example by suitable software instruction) to estimate locations of one or more regions of the first image having at least a first error caused by encoding for streaming. The graphics processor is also configured (for example by suitable software instruction) to render a subsequent image for streaming, at a lower quality than the predetermined quality in at least a region corresponding to a located error in the estimated received first image, as described elsewhere herein. Instances of this summary embodiment implementing the methods and techniques described herein (for example by use of suitable software instruction) are envisaged within the scope of the application. For example, the error location processor may encompass or equate to an encoding processor (e.g. GPU 30 and / or CPU 20) configured (for example by suitable software instruction) to encode the first image for transmission to a client, as described elsewhere herein,a decoding processor (e.g. GPU 30 and / or CPU 20) configured (for example by suitable software instruction) to decode the encoded image as transmitted to generate an estimated received first image, as described elsewhere herein, anda comparator processor (e.g. GPU 30 and / or CPU 20) configured (for example by suitable software instruction) to compare the first original image to the estimated received image to locate errors in the estimated received first image, as described elsewhere herein. Meanwhile, in a further summary embodiment of the present description, a streaming system comprises at least one streaming server as described previously herein, and at least one client device configured to receive and decode images received from one of the servers for display, as described previously herein. The foregoing discussion discloses and describes merely exemplary embodiments of the present invention. As will be understood by those skilled in the art, the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting of the scope of the invention, as well as other claims. The disclosure, including any readily discernible variants of the teachings herein, defines, in part, the scope of the foregoing claim terminology such that no inventive subject matter is dedicated to the public.

Claims

1. A rendering method for a streaming server, comprising the steps of rendering a first image for streaming at a predetermined quality; estimating locations of one or more regions of the first image having at least a first error caused by encoding for streaming; andrendering a subsequent image for streaming, whereinthe subsequent image is rendered at a lower quality than the predetermined quality in at least a region corresponding to an estimated location.

2. The method of claim 1, in whichthe subsequent image is rendered at a lower quality than the predetermined quality in at least a region corresponding to an estimated location that exceeds a predetermined error threshold.

3. The method of any preceding claim, in which the or each region comprises one or more selected from the list consisting of:i. an object in the image;ii. a tile or grid region of the image; andill. a region of the image subject to a discrete encoding step.

4. The method of any preceding claim, in which the image quality in a region is lowered proportional to a level of the corresponding error.

5. The method of any preceding claim in which at least a first region of the subsequent image is rendered at a lower quality by altering one or more selected from the list consisting of:i. geometry rendered for the subsequent image;ii. texture quality for the subsequent image;ill. lighting quality for the subsequent image;iv. atmospheric, volumetric, or physics quality for the subsequent image;v. post processing effects for the subsequent image; andvi. resolution or upscaling for the subsequent image.

6. The method according to claim 5, in which one or more types of alteration are selected in response to a type of error in the estimated received image.

7. The method according to claim 6, in which types of error include one or more selected from the list consisting of:i. altered colour depth or gamut;ii. altered real or effective resolution;ill. altered spatial frequency distribution;iv. blockiness; andiv. altered edge sharpness.

8. The method of any preceding claim, comprising the step of:for a region corresponding to an estimated location for the first image, estimating the corresponding location of that region in the subsequent image.

9. The method of claim 8, in which the step of estimating the location of that region in the subsequent image comprises:estimating a motion vector for the region based on its location in the first image and a preceding image.

10. The method of claim 8 of claim 9, in which the step of estimating the location of that region in the subsequent image comprises:estimating a change of location based upon one or more selected from the list consisting of:i. a change in geometry; andii. a change in virtual camera position,between the first image and the subsequent image, whereinthe estimation occurs prior to a rendering step of the subsequent image that is to be rendered at a lower quality than the predetermined quality for that region.

11. The method of any preceding claim, in which the step of rendering the subsequent image at a lower quality than the predetermined quality in at least a region corresponding to an estimated location for the first image is repeated for further images until one or more selected from the list consisting of:i. an image element in that regions changes by more than a threshold amount;ii. a network condition improves by more than a predetermined amount; andill. a predetermined period of time has elapsed.

12. The method of any preceding claim in which the step of rendering the subsequent image at a lower quality than the predetermined quality in at least a region corresponding to an estimated location for the first image comprises:retaining a sub-region of that region at the predetermined quality; and the method comprises the steps of evaluating a decoding error in the sub-region; andif the error in the sub-region drops below the predetermined error threshold, improving the rendering quality in that region as a whole in the next rendered image.

13. The method of any preceding claim in which the step of rendering the subsequent image at a lower quality than the predetermined quality in at least a region corresponding to an estimated location for the first image comprises:retaining a sub-region of that region at the predetermined quality, the sub-region being selected according to a salience of an image element within the sub-region.

14. The method of any preceding claim in which the step of estimating locations of one or more regions of the first image having errors caused by encoding for streaming comprises the steps of: encoding the first image for transmission to a client;decoding the encoded image as transmitted to generate an estimated received first image; and comparing the first original image to the estimated received image to locate errors in the estimated received first image.

15. A computer program comprising computer executable instructions adapted to cause a computer system to perform the method of any one of the preceding claims.

16. A streaming server, comprising:a graphics processor configured to render a first image for streaming at a predeterminedquality;an error location processor configured to estimate locations of one or more regions of the first image having at least a first error caused by encoding for streaming; andthe graphics processor being configured to render a subsequent image for streaming, whereinthe subsequent image is rendered by the graphics processor at a lower quality than the5 predetermined quality in at least a region corresponding to an estimated location.

17. A streaming system, comprising:one or more servers according to claim 16; andone or more client devices configured to receive and decode images received from one of the servers for display.1016

Images