Image rendering system and method

Abstract

An image rendering system configured to generate images for display to a viewer, the system comprising a gaze tracking unit configured to identify a gaze direction of the viewer in dependence upon one

Description

BACKGROUND OF THE INVENTION Field of the invention This disclosure relates to an image rendering system and method. Description of the Prior Art The "background" description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description which may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present invention. In the fields of computer gaming and video content consumption, there has been an increasing demand for immersive content. In many cases, a level of immersion correlates with a display quality associated with the content being viewed - this is because without a high level of display quality it can be challenging for a viewer to consider the content to be particularly realistic and therefore become fully immersed. In some cases, an increased display quality is able to be achieved through the utilisation of increasingly powerful processors and the like. This can enable the generation of high-quality image content by allowing increasingly detailed rendering processes to be performed without introducing too significant a delay in generating the images. However, factors such as the cost and power consumption can mean that this is not a suitable approach for many use cases - for instance, in the case of portable devices such as mobile games consoles or head-mountable display devices (HMDs). In view of this, a number of processing techniques have been proposed which offer an improved level of immersion for a viewer by more efficiently utilising the available processing resources. One example of such a technique is that of foveated rendering, in which a region of the display image is identified as an area comprising the viewer's point of focus and is accordingly rendered at a high level of detail, while areas outside of this region are rendered with a lower level of detail. However, there are a number of drawbacks associated with such a process, such as the requirements for responsive and accurate gaze tracking. It is therefore considered that improvements to such processing techniques would be desirable in providing viewers with an improved and more immersive viewing experience. It is in the context of the above discussion that the present disclosure arises. SUMMARY OF THE INVENTION This disclosure is defined by claim 1. Further respective aspects and features of the disclosure are defined in the appended claims. It is to be understood that both the foregoing general description of the invention and the following detailed description are exemplary, but are not restrictive, of the invention. BRIEF DESCRIPTION OF THE DRAWINGS A more complete appreciation of the disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein: Figure 1 schematically illustrates an entertainment system; Figure 2 shows a simplified side view of the structure of a typical eye; Figure 3 schematically illustrates a graph of visual acuity versus angle for a typical eye; Figure 4 schematically illustrates a foveated rendering process in accordance with the teachings of the present disclosure; Figure 5 schematically illustrates exemplary high-resolution region shapes; Figure 6 schematically illustrates the shapes of Figure 5 with a reference shape overlaid; Figure 7 schematically illustrates an example in which more than two regions are defined; Figure 8 schematically illustrates a system for implementing one or more of the techniques detailed in this disclosure; and Figure 9 schematically illustrates an image rendering method for generating images for display to a viewer. DESCRIPTION OF THE EMBODIMENTS Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views, embodiments of the present disclosure are described. Referring to Figure 1, an example of an entertainment system 10 is a computer or console such as the Sony® PlayStation 5® (PS5). The entertainment system 10 comprises a central processor 20. This may be a single or multi core processor, for example comprising eight cores as in the PS5. The entertainment system also comprises a graphical processing unit or GPU 30. The GPU can be physically separate to the CPU, or integrated with the CPU as a system on a chip (SoC) as in the PS5. The entertainment device also comprises RAM 40, and may either have separate RAM for each of the CPU and GPU, or shared RAM as in the PS5. The or each RAM can be physically separate, or integrated as part of an SoC as in the PS5. Further storage is provided by a disk 50, either as an external or internal hard drive, or as an external solid state drive, or an internal solid state drive as in the PS5. The entertainment device may transmit or receive data via one or more data ports 60, such as a USB port, Ethernet® port, Wi-Fi® port, Bluetooth® port or similar, as appropriate. It may also optionally receive data via an optical drive 70. Audio / visual outputs from the entertainment device are typically provided through one or more A / V ports 90, or through one or more of the wired or wireless data ports 60. An example of a device for displaying images output by the entertainment system is a head mounted display 'HMD' 120, such as the PlayStation VR 2 'PSVR2', worn by a viewer 1. Where components are not integrated, they may be connected as appropriate either by a dedicated data link or via a bus 100. Interaction with the system is typically provided using one or more handheld controllers (130, 130A), such as the DualSense® controller (130) in the case of the PS5, and / or one or more VR controllers (130A-L,R) in the case of the HMD. As discussed above, foveated rendering is an example of a processing technique that can be used to generate immersive content for a viewer. Foveated rendering makes use of the fact that the human eye does not view the environment with a uniform level of detail; this is because the human eye does not have a uniform structure. For instance, the eye is not a perfect sphere, and different parts of the eye have different characteristics (such as varying reflectance or colour). Figure 2 shows a simplified side view of the structure of a typical eye 200; this Figure has omitted features such as the muscles which control eye motion for the sake of clarity. The eye 200 is formed of a near-spherical structure filled with an aqueous solution 210, with a retina 220 formed on the rear surface of the eye 200. The optic nerve 230 is connected at the rear of the eye 200. Images are formed on the retina 220 by light entering the eye 200, and corresponding signals carrying visual information are transmitted from the retina 220 to the brain via the optic nerve 230. Turning to the front surface of the eye 200, the sclera 240 (commonly referred to as the white of the eye) surrounds the iris 250. The iris 250 controls the size of the pupil 260, which is an aperture through 3 which light enters the eye 200. The iris 250 and pupil 260 are covered by the cornea 270, which is a transparent layer which can refract light entering the eye 200. The eye 200 also comprises a lens (not shown) that is present behind the iris 250 that may be controlled to adjust the focus of the light entering the eye 200. The structure of the eye is such that there is an area of high visual acuity (the fovea), with a sharp drop off either side of this. This is illustrated by the curve 300 of Figure 3, with the peak in the centre representing the foveal region. The area 310 is the 'blind spot'; this is an area in which the eye has no visual acuity as it corresponds to the area where the optic nerve meets the retina. The periphery (that is, the viewing angles furthest from the fovea) is not particularly sensitive colour or detail, and instead is used to detect motion. Foveal rendering is a rendering technique that takes advantage of the relatively small size (around 2.5 degrees) of the fovea and the sharp fall-off in acuity outside of that. In particular, the gaze direction of the viewer is used to determine which portion of the display falls within the fovea and then a high-quality rendering process is used for this portion of the display (with a lower-quality rendering process being applied to the rest of the display). High-quality is a relative term which indicates that a particular region (the 'high-quality region') has one or more properties which exceed those of another region (a 'lower-quality region'). Examples of these properties include resolution and level of detail; for example, a high-quality region may be rendered at 4K resolution while a lower-quality region is rendered using HD resolution. The location of the fovea changes with movement of the eye, hence the need for a responsive tracking of the motion of the viewer's eyes - particularly in view of the fact that this motion can often be high velocity. A viewer's eyes typically undergo a large amount of motion during viewing, and this motion may be categorised into one of a number of categories. Saccades, and on a smaller scale micro-saccades, are identified as fast motions in which the eyes rapidly move between different points of focus (often in a jerky fashion). This may be considered as ballistic motion, in that once the movement has been initiated it cannot be altered. Saccades are often not conscious eye motions, and instead are performed reflexively to survey an environment. Saccades may last up to two hundred milliseconds, depending on the distance rotated by the eye, but may be as short as twenty milliseconds. The speed of a saccade is also dependent upon the total rotation angle; typical speeds may be between two hundred and five hundred degrees per second. 'Smooth pursuit' refers to a slower movement type than a saccade. Smooth pursuit is generally associated with a conscious tracking of a point of focus by a viewer, and is performed so as to maintain the position of a target within (or at least substantially within) the foveal region of the viewer's vision. This enables a high-quality view of a target of interest to be maintained in spite of motion. If the target moves too fast, then smooth pursuit may instead require a number of saccades in order to keep up; this is because smooth pursuit has a lower maximum speed, in the region of thirty degrees per second. The vestibular-ocular reflex is a further example of eye motion. The vestibular-ocular reflex is the motion of the eyes that counteracts head motion; that is, the motion of the eyes relative to the head that enables a person to remain focused on a particular point despite moving their head. Another type of motion is that of the vergence accommodation reflex. This is the motion that causes the eyes to rotate to converge at a point, and the corresponding adjustment of the lens within the eye to cause that point to come into focus. Further eye motions that may be observed as a part of a gaze tracking process are those of blinks or winks, in which the eyelid covers the eyes of the viewer. Such motions may be reflexive or intentional, and can often interfere with eye tracking as they will obscure vision of the eye, and the eye is often not stationary during such a motion. In view of the above considerations, the present disclosure seeks to provide a system and method which enables a robust and efficient foveated rendering process to be performed; implementations of the present disclosure can result in a reduced likelihood of the viewer's gaze being in a region of low quality, as well as enabling a lower-frequency (that is, higher latency) foveated rendering process to be utilised. Figure 4 schematically illustrates a foveated rendering process in accordance with the teachings of the present disclosure. This approach may be particularly suitable in the context of foveated rendering during saccades, but may be applied equally in the context of other eye motions as appropriate. A step 400 comprises tracking the viewer's gaze direction and velocity using any suitable gaze tracking arrangement. An example of a suitable gaze tracking arrangement is one in which cameras located near the viewer's eyes are used to capture images of one or both of the eyes, with the images being processed to identify a pupil location. This pupil location can then be used to derive a gaze direction, with the changing location over a number of captured images being used to identify an associated velocity. In some cases, it may also be advantageous to identify a type of motion being performed by the eyes in accordance with any of the types of motion described above - this can be performed instead of, or in addition to, the determination of the velocity. For instance, this could include identifying a saccade. In addition to tracking the viewer's gaze, it may also be advantageous to make predictions about the viewer's gaze motion - these may be based upon the detected positions / velocities prior to the prediction being made, and / or information about the content being viewed which may be indicative of a viewer's likely viewing at one or more times after the prediction is made. A step 410 comprises identifying a region of the display to which the viewer's gaze corresponds. This identified region may be a predefined region of the display into which the viewer's gaze falls, for instance, or a region of a predetermined size about the point upon which the viewer's gaze is detected to be focused upon based upon the tracking of step 400. Any suitable manner of identifying a region based upon the viewer's gaze may be considered appropriate, rather than being limited to these examples. A step 420 comprises defining a shape to be used as the high-quality region in a foveated rendering process. This is based upon at least one aspect of the velocity (direction and / or speed) associated with the viewer's gaze, so as to account for the reduced reliability associated with predictions of a landing point of the gaze when the gaze has a higher movement speed. In particular, the shape defined in this step is defined so as to increase the reliability of the predictions by increasing the likelihood of the gaze falling within the high-quality region once the motion is completed. This can be achieved by enlarging the shape of the high-quality region in the direction of motion; examples of implementations of such a feature are provided below. A step 430 comprises rendering one or more images for display in dependence upon steps 410 and 420. For instance, this may comprise performing a rendering process in which an image is generated comprising a high-quality region with a shape and location determined in dependence upon steps 410 and 420. The rendered image may comprise multiple regions having different levels of quality in some embodiments, such as a high-quality region which is surrounded by a medium-quality region with the rest of the image being low-quality. By considering the motion of the viewer's eye when defining the foveal region, the correspondence between the high-quality region and the foveal region can be improved even when the viewer's eyes are undergoing rapid motion, such as during a saccade. In response to this, it is also possible to reduce the overall size of the foveal region (compared to a typical arrangement which does not provide the shape selection described above) and as such improve the rendering efficiency due to being able to render a relatively smaller region with a high quality. By rendering a smaller region at high quality, it is also considered that the relative level of quality may be increased so as to obtain an improved display quality instead of (or in addition to) the more efficient rendering. As described above, it is common for foveal rendering processes to utilise circular or square regions to define high-quality parts of an image. However, implementations of the method of Figure 4 can lead to the use of other shapes to define the high-quality parts of an image in response to the detection of a 6 changing gaze direction. For instance, an ellipse or a rectangle may be used. Examples of these are discussed below with reference to Figures 5-7; while these examples are illustrated using a circle / ellipse as the base shape which is modified, the same principles may be applied to any shape. Of course, rather than modifying a base shape it is also considered that a new shape may be defined freely in dependence upon the detected gaze motion. In Figure 5, the base shape for a high-quality region of a rendered image is the circle 500; in other words, this relates to an arrangement in which the foveal region is circular when they viewer's gaze is static, and then a modification is applied to this circle 500 when motion of the viewer's gaze is detected. As indicated by the arrow in the top-left corner, the direction of the viewer's gaze motion is towards the right. In a first example, the width of the circle 500 is increased (whilst maintaining a constant height) so as to generate an ellipse that has an increased area in the direction of gaze motion (as well as in the opposite direction). This increased area in the direction of the gaze motion therefore increases the likelihood of the viewer's gaze landing within the ellipse 510 relative to if the circle 500 were used. A second example uses the ellipse 520 which, relative to the circle 500, has an increased width in the direction of gaze motion as well as a reduced height in the direction perpendicular to the direction of the gaze motion. While this may offer a reduced likelihood that the viewer's gaze would land within the ellipse 520 relative to that of the ellipse 510, such a modification to the circle 500 may nevertheless be desirable as the area of the shape (and therefore size of the high-quality region) can be kept substantially constant. This can therefore lead to a more efficient rendering process relative to an arrangement which uses a larger shape for the high-quality region. A third example demonstrates a non-uniform modification to the circle 500; in this case, the shape 530 has been stretched in the direction of gaze motion without adjusting the left side of the shape. This results in a shape which is asymmetrical in the direction of gaze motion. Such an approach may be advantageous in that the overall shape of the circle 500 is largely maintained, leading to a more consistent rendering, whilst also offering the discussed advantages of implementations of the present disclosure. Figure 6 schematically illustrates the modified shapes of Figure 5 with the circle 500 overlaid; the comparison 610 relates to the shape 510 of Figure 5, with the same relation between the pairs 620 / 520 and 630 / 530. These comparisons illustrate how the use of the alternative shapes increases the resilience of the foveated rendering process, as the shapes offer an increased breadth of the high-quality region on the axis of the direction of gaze motion. In other words, the uncertainty of the landing position is focused largely along the axis of the direction of the gaze motion, and as such a high-quality region which is broader along this axis is more likely to contain the eventual landing point of the viewer's gaze. Figure 7 schematically illustrates an example in which more than two regions are defined - for instance, a high-quality region, a medium-quality region, and a low-quality region (which comprises the image area which is not a part of the other two regions). The shapes labelled 700 represent a high-quality region surrounded by a medium-quality region as a base region configuration (in much the same way as discussed with reference to Figure 5). The shapes labelled 710 illustrate an example in which only the high-quality region has its shape modified (in this case, to match the modification represented by the reference 520 of Figure 5). In this example, the medium-quality region is unchanged. This may be advantageous in that the mediumquality region can be modified with a reduced reliance upon a detected gaze motion direction, which can reduce the latency of the rendering process (as this can be rendered before the gaze motion direction is fully identified). The shapes labelled 720 instead illustrate an example in which both the high-quality and mediumquality regions are modified in dependence upon the identified direction of gaze motion. This may be advantageous in that the medium-quality region can obtain the same benefits as the high-quality region in respect of the likelihood of comprising the landing point of the viewer's gaze and substantially maintaining the area of the region. The degree of distortion applied to a base shape (or indeed, the properties of a shape that is defined as a new high-quality region) may be determined in any suitable manner in accordance with the above teachings. In some examples, the detected motion speed can be used to scale or define one or more measurements of the shape as appropriate. For instance, an ellipse is typically defined by its width, a, and its height, b, which respectively represent the extent of the shape along its longest axis and its shortest axis. When modifying or defining an ellipse, each of these factors can be scaled by a respective factor: v ^scaled ~ ^static Cb h — — h ^scaled — „ ustatic V In these equations, ca / b represent a proportionality constant which is specific to each of the width and height measurements; these may be the same in some cases, or they may be different to one another as appropriate to obtain a desired effect. In these equations, v represents a detected velocity of the viewer's gaze. This means that a detected eye motion would lead to a scaling which increases the width and reduces the height; this can be particularly desirable in the cases that the width of the shape is defined as the diameter of the reference circle which is parallel to the direction of the detected velocity. Of course, with a circle as the reference shape of the foveal region there is a free choice as to which direction represents the width and which the height due to rotational symmetry; it is therefore considered that defining the width to be the same direction as the velocity is suitable (in conjunction with the described scaling factors) to achieve a stretch in the direction of the velocity in a manner which maintains computational simplicity. In some cases the parameters shown in the equation above may be modified to as to cause a desired scaling to be performed - for instance, the v terms could be replaced by a composite term which included a constant value (such as c + v / 10) such that a scaling is able to be performed in any desired fashion. In other words, the scalings discussed here are purely exemplary and the specific manner in which they are implemented may be determined freely for a particular implementation of the teachings of the present disclosure. In the case that a non-circular reference shape is used (such as a square), the shape modification may include a rotation of the shape so as to enable a desired stretch. For instance, if the velocity of the gaze is aligned with a diagonal of a square then the square may be rotated forty-five degrees so as to align the width of the shape with the velocity. In other words, a rotation of the reference shape can be performed so as to align the width of the shape with the direction of the velocity. This can then enable a stretch of the width to be performed rather than an enlargement of the entire shape that would otherwise be realised. Of course, this is not essential as a distortion other than a stretch of the width may be desired. Similarly, in some cases it may be considered that the area of the high-quality region should remain constant. In the example of an ellipse which is deformed in line with the multiplicative scaling factor described above, the area is defined as follows: v cb A scaled H X O-scaled ^scaled H X Osbanc X ^static Ca v In this case, it may be appropriate to define a second scaling factor based upon the first scaling factor such that the area is unchanged - in other words, the area of Ascaied is the same as Astatic! V cb ^scaled TC X O,scaieci X bscaieci TC X —— ClSfafic X ~ bsbabic ^static TC X Osbabic X bS[a[ic c, Which may be rearranged to obtain, for a constant area: Ca ^b In other words, by setting the constants to be equal to one another in implementations such as this, the area can be made constant independent of the distortions of the reference shape. In a more general case, the condition of the scaled area being equal to the unsealed area is able to be satisfied when the proportional increase to the width is cancelled out by the proportional decrease in the height. As an equation, this is represented by: Oscaled _ ^static O-static ^scaled Given that the values of the reference shape width and height can be considered to be constant values, they can be omitted from the above so as to arrive at: x Oscaled 7 “scaled Where x is a constant of proportionality which relates to the area of the shape. In line with the above equation, it is considered that it is not necessary to separately calculate the scaling to be applied to the width and the height - one of these can be calculated with the other being defined using this equation. For instance, the scaling of the width may be calculated to arrive at the scaled width and then the scaled height can be determined as being x divided by this value. The above discussion is entirely exemplary, with these teachings able to be applied to a range of conditions (such as the area being able to be scaled by up or down to a certain percentage of the static area during motion) and different shapes (such as rectangles) as appropriate. While shown as a multiplicative scaling factor in these equations, it is of course considered that an additive scaling factor may be considered appropriate. While this would lead to different scaling factors being defined (for instance, using negative scaling factors rather than reciprocals of scaling factors to modify a measurement), the overall function is the same and as such the teachings above can be applied freely to other methods of implementing scaling factors. For example, the width and height of a modified ellipse / circle may be represented by: v Oscaled Ostauc + —— ca V h — fa__ “scaled “static cb In these equations, v represents a detected velocity of the gaze as described above; the scaling is performed such that the width is defined in the direction of the velocity (or the reference shape is otherwise rotated so as to align the width with the velocity direction). As is clear from these equations, the velocity is used as a factor to increase the width and decrease the height of the shape - the constants may be defined in any suitable manner so as to enable this. While the implementation of scaling factors may be performed in the same manner for each viewer, in some cases it may be advantageous to instead personalise the scaling of the size of the high-quality region. For instance, a viewer's gaze tracking history (which may be associated with a viewer profile, for instance, and represented by one or more relevant parameters) can be used to inform the scaling of the region. For example, the viewer's gaze tracking history may be used to measure average saccade duration / motion so as to better inform the prediction of the landing location - with an improved prediction of the landing location, the amount of distortion of the region can be reduced. Any other observed characteristics which inform the prediction reliability can also be used, with increasingly reliable predictions of a landing point typically correlating with smaller scaling factors. This can also be used to inform the nature of the distortion that is applied (or the region shape that is defined), in that characteristics of the viewer's gaze motion can be used to determine whether a modification of the height as well as width of the region is suitable (for example) based upon typical motions or the like. Alternatively, or in addition, the scaling factors applied can be varied based upon one or more rendering parameters associated with the content being viewed. For example, a frame rate or resolution associated with the content may be considered. In the case of a higher frame rate (such as 120 frames per second relative to 60 frames per second), the scaling factors may be reduced (thereby increasing the distortion / deviation relative to the base region shape) as it may be considered that the region can be modified in a more responsive fashion. This therefore means that the location of the region can be updated more frequently, which can lead to an improved placement of the region. In the case of considering the resolution, a higher resolution may correlate with a reduced scaling factor as the rendering of larger high-resolution regions can lead to a significant burden upon an image rendering device. Alternatively, or in addition, the difference between the image quality associated with each of the high- / low-quality (and medium-quality, if applicable) regions may be considered - in the case in which the difference is larger, the scaling factors may be increased so as to increase the likelihood that the landing point of the viewer's gaze falls within the high-quality region. This is because a larger difference in rendering quality (such as a greater difference in resolution) may be more noticeable to the viewer and as such a greater emphasis should be placed on avoiding the viewer viewing the low-quality region. While scaling factors are described above for use in determining a modified shape to be used for the foveated rendering process, in some implementations it is considered instead that a detected gaze motion speed is used to select a predefined shape for use. For instance, two thresholds could be defined - below the first threshold, a reference / standard shape (such as a circle) is used, between the first and second thresholds a first modified shape is used (such as an ellipse which is only slightly broader than it is tall), while above the second threshold a second modified shape is used (such as an ellipse which has a greater width / height ratio than the first modified shape). In other words, while the scale of the modification is proportional to the gaze motion speed it does not have to be directly proportional. Figure 8 schematically illustrates a system for implementing one or more of the techniques detailed in this disclosure. In particular, Figure 8 shows an image rendering system configured to generate images for display to a viewer, the system comprising a gaze tracking unit 800, a region defining unit 810, a rendering unit 820, and optionally a display unit 830. The functionality of these devices may be distributed as appropriate for a given implementation - processing functions associated with the units 800, 810, and 820 may be performed by CPU and / or GPU associated with a games console (such as the entertainment device 10 of Figure 1), for example, a display device, and / or any combination of these and other devices (such as cloud computing hardware) as appropriate. The gaze tracking unit 800 is configured to identify a gaze direction of the viewer in dependence upon one or more images, captured by one or more cameras, of at least one of the viewer's eyes. In some implementations, the gaze tracking unit 800 comprises the cameras which are used to capture the images. The identification performed by the gaze tracking unit 800 may include a detection of both a current (that is, up-to-date) gaze direction of the viewer, a direction of the change of gaze direction (that is, the direction of the gaze motion), and / or a speed of the gaze motion. In some implementations, the gaze tracking unit 800 is configured to specifically detect a saccade performed by the viewer, with the region defining unit 810 being configured to operate in dependence upon this detection. In other words, the defining of the custom region by the region defining unit 810 may be performed only in response to saccades (and optionally micro-saccades) and not in response to other detected eye motions. A saccade may be detected based upon the identification of an abovethreshold movement of the eyes, for example, or a particular pattern of motion (such as one that corresponds to the scanning of an environment). The region defining unit 810 is configured to define the size and / or shape of a first region in dependence upon the tracked gaze direction of the viewer, wherein the size and / or shape of the region are dependent upon at least a detected motion of the viewer's gaze. Examples of this are discussed above with reference to Figures 5-7, in which a shape other than a reference shape is used as the high-quality region when generating an image for display. In a case in which the region defining unit 810 is not configured to define a first region (such as when a below-threshold motion is detected, in implementations in which this condition is applied), a reference region such as a circle or square with a predetermined size may be used instead. As described above, the region may be defined in accordance with the direction of the velocity such that axis representing the width of the region is defined in dependence upon the direction of the velocity, for example, or a rotation of the region is performed so as to align the width of the region with the direction of the velocity. In some implementations the region defining unit 810 may be configured to define the size and / or shape as a modification of a reference region (rather than a definition or selection of a new shape); this reference region may be a circle or a square, for instance, or any other suitable shape for defining a high-quality region in a foveated rendering process. The region defining unit 810 may be configured to define the first region in accordance with a predefined area; this may be the area of a reference shape, for example, so as to ensure a consistent size of the high-resolution region of a rendered image as this can assist with causing processing demands associated with image rendering to be more consistent than in a case in which the size of the region is dynamic. In some implementations, the region defining unit 810 may be configured to define the first region such that the size of the first region is larger in the direction of the detected motion of the viewer's gaze than in a direction perpendicular to the direction of the detected motion of the viewer's gaze. An example of this is shown by the shapes 510, 520, and 530 in Figure 5, for instance, in which each of the shapes is wider (that is, longer in the direction of the gaze motion) than it is tall (that is, the direction perpendicular to the gaze motion). Alternatively, or in addition, the first region may be defined such that the region is asymmetric about an axis perpendicular to the direction of the detected motion of the viewer's gaze. This is exemplified by the shape 530 in Figure 5, in which the shape 'bulges' in the direction of gaze motion so as to no longer be left / right symmetric (with left / right being defined by the axis perpendicular to the direction of the detected motion in this example). In some implementations the region defining unit 810 may be configured to define the size and / or shape of the first region in dependence upon a viewer profile indicating one or more gaze characteristics of the viewer. Alternatively, or in addition, one or more parameters associated with the images being rendered may be considered when defining the first region - examples include a resolution difference between a high-quality and a low-quality region, for example, or a frame rate of the content. In accordance with the discussion with reference to Figure 7, in some implementations the region defining unit 810 may be configured to define a second region which encloses the first region, the 13 second region corresponding to an image quality lower than the first region and higher than the other portions outside of the first region. The second region can therefore be considered to be a mediumquality region, in accordance with the referenced earlier discussion. In some cases, the region defining unit 810 may be configured to define the size and / or shape of the second region in dependence upon the detected gaze motion in much the same manner as the defining of the first region. The rendering unit 820 is configured to render one or more images for display in which a first region is rendered with a higher image quality than portions of the image outside of the first region, wherein the first region is located at a display location corresponding to the identified gaze direction. The higher image quality may correspond to a higher resolution and / or refresh rate, for example. The rendering unit 820 may be implemented in accordance with known foveal rendering techniques as modified in respect of the features described in the present disclosure. The optional display unit 830 comprises a display that is configured to display the images rendered by the rendering unit 820; this display unit 830 may be considered optional in that the rendered images may be stored for display at a later time rather than (or as well as) being displayed immediately. In some implementations the display unit 830 may be a head-mountable display device such as the HMD 120 of Figure 1; advantageously, the HMD may include cameras for performing gaze tracking as discussed with reference to the gaze tracking unit 800. The arrangement of Figure 8 is an example of a processor (for example, a GPU, TPU, and / or CPU located in a games console or any other computing device) that is operable to generate images for display to a viewer, and in particular is operable to: identify a gaze direction of the viewer in dependence upon one or more images, captured by one or more cameras, of at least one of the viewer's eyes; define the size and / or shape of a first region in dependence upon the tracked gaze direction of the viewer, wherein the size and / or shape of the region are dependent upon at least a detected motion of the viewer's gaze; and render an image for display in which a first region is rendered with a higher image quality than portions of the image outside of the first region, wherein the first region is located at a display location corresponding to the identified gaze direction. Figure 9 schematically illustrates an image rendering method for generating images for display to a viewer. This method may be implemented in accordance with the discussion of the arrangement of Figure 8, for example; the arrangement of Figure 1 may be considered to comprise suitable hardware for performing such a method. A step 900 comprises identifying a gaze direction of the viewer in dependence upon one or more images, captured by one or more cameras, of at least one of the viewer's eyes. A step 910 comprises defining the size and / or shape of a first region in dependence upon the tracked gaze direction of the viewer, wherein the size and / or shape of the region are dependent upon at least a detected motion of the viewer's gaze. A step 920 comprises rendering an image for display in which a first region is rendered with a higher image quality than portions of the image outside of the first region, wherein the first region is located at a display location corresponding to the identified gaze direction. An optional step 930 comprises displaying the rendered images. The techniques described above may be implemented in hardware, software or combinations of the two. In the case that a software-controlled data processing apparatus is employed to implement one or more features of the embodiments, it will be appreciated that such software, and a storage or transmission medium such as a non-transitory machine-readable storage medium by which such software is provided, are also considered as embodiments of the disclosure. Thus, the foregoing discussion discloses and describes merely exemplary embodiments of the present invention. As will be understood by those skilled in the art, the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting of the scope of the invention, as well as other claims. The disclosure, including any readily discernible variants of the teachings herein, defines, in part, the scope of the foregoing claim terminology such that no inventive subject matter is dedicated to the public.

Claims

1. An image rendering system configured to generate images for display to a viewer, the system comprising:a gaze tracking unit configured to identify a gaze direction of the viewer in dependence upon one or more images, captured by one or more cameras, of at least one of the viewer's eyes;a region defining unit configured to define the size and / or shape of a first region in dependence upon the tracked gaze direction of the viewer, wherein the size and / or shape of the region are dependent upon at least a detected motion of the viewer's gaze; anda rendering unit configured to render an image for display in which a first region is rendered with a higher image quality than portions of the image outside of the first region, wherein the first region is located at a display location corresponding to the identified gaze direction.

2. A system according to claim 1, wherein the region defining unit is configured to define the size and / or shape as a modification of a reference region.

3. A system according to claim 2, wherein the reference region is a circle or a square.

4. A system according to any preceding claim, wherein the region defining unit is configured todefine the first region such that the size of the first region is larger in the direction of the detected motion of the viewer's gaze than in a direction perpendicular to the direction of the detected motion of the viewer's gaze.

5. A system according to any preceding claim, wherein the first region is defined such that the region is asymmetric about an axis perpendicular to the direction of the detected motion of the viewer's gaze.

6. A system according to any preceding claim, wherein the region defining unit is configured to define the first region in accordance with a predefined area.

7. A system according to any preceding claim, wherein the region defining unit is configured to define the size and / or shape of the first region in dependence upon a viewer profile indicating one or more gaze characteristics of the viewer.

8. A system according to any preceding claim, wherein the region defining unit is configured to define a second region which encloses the first region, the second region corresponding to an image quality lower than the first region and higher than the other portions outside of the first region.

9. A system according to claim 8, wherein the region defining unit is configured to define the size and / or shape of the second region in dependence upon the detected gaze motion.1610. A system according to any preceding claim, wherein the higher image quality corresponds to a higher resolution and / or refresh rate.

11. A system according to any preceding claim, comprising a head-mountable display device configured to display the images rendered by the rendering unit.

12. A system according to any preceding claim, wherein the gaze tracking unit is configured to detect a saccade performed by the viewer, and wherein the region defining unit is configured to define the first region only in dependence upon this detection.

13. An image rendering method for generating images for display to a viewer, the method comprising:identifying a gaze direction of the viewer in dependence upon one or more images, captured by one or more cameras, of at least one of the viewer's eyes;defining the size and / or shape of a first region in dependence upon the tracked gaze direction of the viewer, wherein the size and / or shape of the region are dependent upon at least a detected motion of the viewer's gaze; andrendering an image for display in which a first region is rendered with a higher image quality than portions of the image outside of the first region, wherein the first region is located at a display location corresponding to the identified gaze direction.

14. Computer software which, when executed by a computer, causes the computer to carry out the method of claim 13.

15. A non-transitory machine-readable storage medium which stores computer software according to claim 14.Application No: GB2312833.318Examiner:Steve WilliamsClaims searched: 1-15 [TRUNCATED]Date of search: 15 February 2024Patents Act 1977: Search Report under Section 17Documents considered to be relevant:Category Relevant to claims Identity of document and passage or figure of particular relevance X 1 - 15 US 2020 / 0082794 Al (SONY) See [0071], [0072], and [0117] X 1 - 15 US 2019 / 0156794 Al (SONY) See claim 1 and [0026] X 1 - 15 US 2018 / 0307905 Al (TOBII) See [0106], and [0183] v A 1 - 15 US 2018 / 0224935 Al (TOBII) See [0032], [0034], and [0085] X 1 - 15 US 2022 / 0130350 Al (APPLE) See [0038], [0050] and [0065] v A 1 - 15 US 2021 / 0141449 Al (APPLE) See [0041] and [0049] X 1 - 15 US 2021 / 0248807 Al (FACEBOOK) See claims 1 and 2Categories:X Document indicating lack of novelty or inventive step A Document indicating technological background and / or state of the art. Y Document indicating lack of inventive step if p Document published on or after the declared priority date but combined with one or more other documents of same category. before the filing date of this invention. & Member of the same patent family E Patent document published on or after, but with priority date earlier than, the filing date of this application.Field of Search:19International Classification:Subclass Subgroup Valid From G06T 0015 / 00 01 / 01 / 2011 G02B 0027 / 01 01 / 01 / 2006 G06F 0003 / 01 01 / 01 / 2006 G06T 0007 / 20 01 / 01 / 2017 G06T 0011 / 00 01 / 01 / 2006 H04N 0013 / 383 01 / 01 / 2018

Images