Mesh based graphics method and system

Abstract

A mesh-based graphics method comprises the steps of outputting for display an image of a virtual environment based upon a viewpoint of a virtual camera and receiving information indicating a position

Description

BACKGROUND OF THE INVENTION Field of the invention The present invention relates to a mesh based graphics method and system. Description of the Prior Art In videogames, the geometry of virtual environments and the objects therein are typically defined by meshes. These meshes in turn typically comprise a plurality of tessellating polygons (typically triangles) defining the geometry of the surface of the virtual object. Meshes are typically used to generate images of virtual environments from the point of view of a virtual camera located within the virtual space. To generate such images, graphics rendering processes such as texturing, shading, colouring, and the like are typically performed by CPU(s) and / or GPU(s) on the meshes. Consequently, the number of meshes contributing to the image affects the computational burden of generating the image. Several strategies are used to reduce this burden, including so-called z-culling (where meshes occluded by other meshes from the camera viewpoint are culled and not processed further), and the use of level-of-detail or LoD as a function of distance, to use larger and simpler meshes (and typically also simpler texturing, shading, colouring, and the like) further from the camera viewpoint. However, there is still scope to improve this strategy, thereby further reducing computational overhead. Embodiments of the present invention seek to mitigate or alleviate this problem. SUMMARY OF THE INVENTION Various aspects and features of the present invention are defined in the appended claims and within the text of the accompanying description. In a first aspect, a mesh-based graphics method is provided in accordance with claim 1. In another aspect, a mesh-based graphics system is provided in accordance with claim 13. BRIEF DESCRIPTION OF THE DRAWINGS A more complete appreciation of the disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein: Figure 1 is a schematic diagram of a mesh-based graphics system in accordance with embodiments of the present description. Figure 2 is a schematic diagram of a virtual camera frustum in accordance with embodiments of the present description. Figures 3A to 3C are schematic diagrams illustrating the process of rendering a scene that is within a virtual camera frustum, in accordance with embodiments of the present description. Figure 4 is a schematic diagram of a foveated meshlet processor, in accordance with embodiments of the present description. Figure 5 is a schematic diagram of a foveated virtual camera frustum, in accordance with embodiments of the present description. Figure 6 is a flow diagram of a mesh-based graphics method, in accordance with embodiments of the present description. DESCRIPTION OF THE EMBODIMENTS A mesh based graphics method and system are disclosed. In the following description, a number of specific details are presented in order to provide a thorough understanding of the embodiments of the present invention. It will be apparent, however, to a person skilled in the art that these specific details need not be employed to practice the present invention. Conversely, specific details known to the person skilled in the art are omitted for the purposes of clarity where appropriate. Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views, Figure 1 shows a computer 100 such as a videogame console, configured as a mesh based graphics system. The computer comprises a game processor 110 that maintains a virtual world of a game, and position data for a virtual camera from whose viewpoint images of the virtual world are generated to be displayed to a player of the game. The computer also comprises a gaze tracking unit 120 that receives information about the player's gaze, for example from a sensor (not shown) such as a camera. The gaze tracking unit identifies the player's direction of gaze with respect to the displayed images of the virtual world. The computer further comprises a foveated meshlet processor 130, whose function is described later herein. The computer further comprises a rendering processor 140, which optionally uses information about the player's gaze to perform foveated rendering. The computer also optionally comprises a z-culling processor 150 in the event that z-culling is utilised during the generation of images of the virtual world to reduce the number of meshes to be textured, etc. To explain the function of the foveated meshlet processor, then referring to Figure 2, a camera (or in the present example, a virtual camera 200) has a viewing frustum that is typically a cone or pyramid projecting out along the optical axis of the camera. Most typically, the frustum takes the form of a truncated pyramid (a pyramid rather than a cone because the corresponding images tend to be square or rectangular). The frustum is truncated in that typically objects within a threshold proximity 220 to the camera are not rendered, although in principle objects can be rendered down to a zero distance. Similarly, the frustum can be truncated at a far distance 230, to limit the number of distant objects to be included within the rendering process, and / or responsive to the combination of resulting object size, image resolution, texture detail, and diminishing return on the fidelity of the resulting image caused by objects beyond this far distance. This threshold is often referred to as the 'draw distance' in videogames and also limits the computational burden of generating the image by limiting how may objects to render in the distance. The remaining truncated pyramid 210 is then the camera's working frustum for the purposes of rendering the camera's view of the virtual world, and is referred to hereafter simply as the camera's frustum unless specified. Referring next to Figures 3A to 3C, in Figure 3A two identical objects (as an arbitrary example, footballs) are shown at different positions within the virtual world, and captured within the frustum 210 of the virtual camera. Then in Figure 3B, as part of the process to convert this into a rendered image, a vertex shader is typically used to transfer vertices from the world coordinates occupied by the virtual camera and its frustum into normalised device coordinates ('NDC') via a projection matrix. The result is a cube ('NDC cube') with normalised (e.g. unit) dimensions, in which the virtual world objects have been similarly warped by the projection, so that for example objects close to the camera have become larger than objects further from the camera. Hence in figure 3B the football closest to the camera is now larger than the football further from the camera. Finally in figure 3C, an image can be generated as a 2D render from a front face of the NDC cube, in this case resulting in the closer football appearing larger than and partially occluding the more distant football. Typically the conversion to the NDC cube is implemented by the rendering processor 140, and the culling of occluded mesh elements (e.g. for the more distant ball) is implemented by the z-culling processor 150 prior to the application of textures and the rasterization of the image, in order to the reduce computational load of these stages. In this way, the part of the virtual world captured within the frustum of the virtual camera is translated into a 2D image for output for display to the user. The quality of this 2D image is potentially dependent upon a lot of things, such as the quality / resolution of the textures used in the image, and the number of polygons in the mesh or meshes used to define the objects in the environment within the virtual world, or the parts of those meshes (meshlets) visible in the rendered scene. Embodiments of the present description seek to improve the apparent quality of this 2D image, and / or maintain a quality of this 2D image whilst using fewer computational resources, as described herein. Referring back to figure 1, the gaze tracking unit 120 is used to determine where within the displayed image 310 the player / user is currently looking. This gaze information may be passed to the rendering processor 140 for the purposes of foveated rendering (e.g. where an image is rendered at a higher effective resolution and / or quality in an area corresponding to the current gaze direction of the user); typically this is achieved by using higher quality textures, and rendering at the full resolution of the system, whilst in peripheral regions outside the area corresponding to the user's gaze, (optionally progressively) lower quality textures and lower resolution rendering can be used, without the user noticing. Referring now also to figure 4, alternatively or in addition, in embodiments of the present description this gaze information can be used elsewhere in the graphics pipeline, as described herein. Figure 4 illustrates a foveated meshlet processor 130 comprising an inverse projection processor 132 receiving gaze information data from the gaze tracking processor 120, a frustum selection processor 134 receiving data from the inverse projection processor and the game processor 100, and a mesh selection processor 136 receiving data from the frustum selection processor 134. In operation, the player's gaze intersects with a position in the currently displayed image 310. This position is provided by the gaze tracking processor 122 to the inverse projection processor 132 of the foveated meshlet processor 130. The provided position in turn corresponds directly to a point on an object or environment surface within the NDC cube. This point can then be mapped by the inverse projection processor 132 back to a position in the virtual world of the game by using an inverse projection from the NDC cube to the virtual camera frustum to generate inversely mapped coordinates within the frustum of the virtual camera. This inverse projection may be implemented for example by pixel shaders. It will be appreciated that the inverse projection is the opposite of the transform used to transfer vertices from the world coordinates to normalised device coordinates. If necessary, these inversely mapped coordinates can then in turn be related to the virtual game world by offsetting them based on the virtual camera position in the virtual world (e.g. so that the inversely mapped coordinate origin is consistent with that of the virtual camera frustum), if the coordinates do not already account for this. This then identifies what point in the virtual world coordinates (or just the camera frustum coordinates) is the subject of the user's gaze. Referring now also to Figure 5, the frustum selection processor 134 may then segment the 3D space within the game world into two or more different regions, each one having a separate degree of mesh quality. The aim of this process is to improve the relative mesh data quality (e.g. any or all of increased polygon count (typically together with reduced polygon size), increased mesh complexity, or the like) within a region of the 3D space corresponding to where the user is looking on screen. It will be appreciated that depending on the scene being depicted, this change in mesh quality may be applied to an entire mesh (e.g. that of the ball) or typically meshlet, e.g. the part of the mesh that remains after z-culling (e.g. corresponding to the visible part of the ball), or similarly that part of a larger mesh that is partially within the bounds of the first volume. Hence for example in Figure 5, the user is looking at a ball in the top-right quarter of a rendered image, and upon mapping this back to the virtual world, the ball is in the foreground of the virtual camera frustum. As a result a first 3D segment of space 530 or first 'volume' encompassing the ball located in a top right nearside part of the frustum is selected to use higher quality mesh data (and optionally also higher quality texture data as well). In this case for example it may result in the ball looking more round than before because it is now defined using more polygons. Optionally, a larger second volume 520 encompassing the first volume (or partially encompassing it in the case that the first volume extends to an edge of the frustum, as depicted in Figure 5) may be selected to use a higher quality mesh data (and optionally also a higher quality texture data as well) that is not as high a quality as in the first volume, but still higher than was previously used or would otherwise be used within that second volume. Meanwhile the mesh data for the remainder of the frustum 510 is optionally unaffected. The optional second volume 520 can provide a more smooth stepwise transition in quality between the first volume and the remainder of the frustum, and can also reflect the drop-off in visual acuity of the user as the scene becomes more distant from the point of gaze. As an optional variant, the second volume 520 can maintain the original mesh quality (or use a still less higher quality improvement), and the remainder of the frustum 510 can use a lower mesh quality. This can be used to transfer computational resources from peripheral regions of the frustrum to the first and optionally second volumes, which may be of benefit when the allocated computational budget is fixed or capped, e.g. in order to maintain at least a minimum frame rate. The size and position of the volumes can be determined in any suitable manner. For example to assist with efficient processing, the volumes may be formed from one or more predetermined volumes (e.g. a 3D grid, similar to an array of voxels). These volumes may all be the same size, either within the frustum, or when transformed to the NDC. The volumes may be selected to efficiently fit certain memory and / or processing constraints within the rendering pipeline, for example relating to the resources available to respective threads or parallel processors within a graphics processor. In this case, frustum selection processor 134 may select the first volume to be the predetermined volume in which the point corresponding to the user gaze is found, or the combination of predetermined volumes occupied by the virtual object on which the point corresponding to the user gaze is found. The optional second volume may then be the set of predetermined volumes bounding the or each predetermined volume in the first volume. Alternatively, rather than dividing the frustum into a three dimensional grid of predetermined volumes, the frustum selection processor 134 may define the first volume to be a volume centred on the point within the frustum corresponding to the user gaze. Again the first volume may be of a predetermined size, or the size may be responsive to the size of the object on which the point corresponding to the user gaze is found. Again, the optional second volume may then encompass or partially encompass the first volume by a predetermined amount, or optionally an amount responsive to the size of the first volume (e.g. forming a larger transitional region if the first volume is larger). In either case, in this way the region of space in the virtual game world that, within the frustum of the virtual camera, is being gazed at by the user (as determined by mapping back from the 2D image of the game world that the user is looking at) can have the quality of the mesh(es) / meshlet(s) defining it improved, so that the quality of definition of that part of the game world is improved prior to any subsequent rendering process, whether or not that rendering process also takes account of the gaze of the user (e.g. foveated rendering). This improves the quality of the source model of the game environment in the region of gaze, and thereby provides the basis for improved graphical quality in the rendered image at that point. As was noted previously, graphical quality, including mesh quality, can also be a function of distance from the camera. Hence the initial mesh quality in the foreground of the camera frustum may already be higher than in the background of the camera frustum. Optionally therefore the improvement in mesh quality in the first volume and optional second volume can be made relative to the quality already found within the camera frustum as a function of distance, or more generally as a function of the selected level of detail ('LoD', which may take account of one or more factors other than or in addition to distance, such as relative object importance, transparency, motion, and the like). Hence in an embodiment of the present description, the mesh selection processor 136 may set the level of detail within the first volume to be higher than the current or default level of detail within that first volume, by a first amount. Then optionally the mesh selection processor 136 may set the level of detail within the second volume to be higher than the current or default level of detail within that second volume, by a second amount that is less than the first amount. The level of detail can be indicated in any suitable manner, whether in association with the mesh(es) / meshlet(s) or volume, e.g. using the existing mechanisms for indicating LoD in the game. By way of example only, suppose that there are five levels of detail, with (by convention) 0 being the highest and 4 being the lowest. The mesh selection processor 136 may then allocate levels of detail as follows relative to the current or default level for that volume: Default LoD First volume LoD Second Volume LoD Frustum LoD 0 0 0 0 1 0 0 1 2 0 1 2 3 1 2 3 4 2 3 4 Hence in this case, the current LoD is boosted by two levels for the first volume (up to the maximum available) and by one level for the second volume (up to the maximum available). Other strategies may be similarly considered. For example the LoD of the first volume may be subject to a minimum (e.g. 1) so that even in the far distance, if the user is looking at an object then it is well defined. It will be appreciated that the maximum available LoD may be dynamic and determined by an overall computational budget for the image; as a result when the first volume is sufficiently large then it may only be boosted by one level, or may itself be segmented into a core and outer volume with different levels of quality improvement, in order to manage the computational cost within the overall budget; meanwhile the second volume may be boosted less or not at all, and the quality in the remaining frustum may be reduced. More generally, the quality distribution within the frustum may alter to increase the quality of the first volume and optionally of the second volume potentially at the cost of quality in the remainder of the frustum, optionally to maintain or meet a computational budget. Hence for example in the case that increasing quality requires a transfer of resources to the first volume within a fixed resource budget, the mesh selection processor 136 may then allocate levels of detail as follows relative to the current or default level for that volume: Example Budget First volume LoD Second Volume LoD Frustum LoD 15 0 (Costs 5) 0 (Costs 5) 0 (Costs 5) 13 0 (Costs 5) 0 (Costs 5) 2 (Costs 3) 11 0 (Costs 5) 1 (Costs 4) 3 (Costs 2) 9 0 (Costs 5) 2 (Costs 3) 4 (Costs 1) 7 1 (Costs 4) 3 (Costs 2) 4 (Costs 1) In this case the mesh selection processor 136 tries to meet the target budget, reallocating resource preferentially to the first volume and then to the second volume, subject to maintaining at least a predetermined minimum quality ('4') in the remainder of the frustum. Notably within the first volume the quality remains high even when the budget drops significantly; this can serve to help maintain both apparent graphical quality and for example frame rates, as the depicted scenes very in complexity during game play. Hence more generally the computer 100, typically through the foveated meshlet processor 130, selects at least a first volume within the frustum of the virtual camera in which the relative quality of the mesh(es) / meshlet(s) contributing the virtual environment in that volume is increased. Where both a first volume and a surrounding or partially surrounding second volume are selected, the relative quality for the first volume is greater than for the second volume. The quality may be increased in an absolute fashion (optionally relative to an existing quality at that position), and / or relative to an overall computational budget for generating and / or rendering the scene. The change in relative quality may be achieved by increasing the level of detail (LoD) associated with the mesh(es) / meshlet(s) affected, or via any other mechanism, such as instructing the relevant function in the game to select a higher quality mesh. The first (and optional second) volume may respectively correspond to one or more predetermined volumes that are subsections of the virtual camera frustum and encompass the relevant gaze point or the object that gaze point lies on, or may be centred on that gaze point. Turning now to Figure 6, in a summary embodiment of the present description, a mesh-based graphics method comprises the following steps: In a first step s610, outputting for display an image of a virtual environment based upon a viewpoint of a virtual camera, as described elsewhere herein; In a second step s620, receiving information indicating a position in the displayed image where a user is looking, as described elsewhere herein; In a third step s630, mapping the position corresponding to where the user is looking to a 3D position within a frustum of the virtual camera that generated the displayed image, as described elsewhere herein; In a fourth step s640, selecting a first volume within the frustum that encompasses the 3D position, as described elsewhere herein; In a fifth step s650, increasing the quality of a virtual mesh within the first volume by a first amount, as described elsewhere herein; and In a sixth step s660, outputting for display a second image of the virtual environment based upon a viewpoint of the virtual camera (e.g. the viewpoint for the next rendered frame), as described elsewhere herein. It will be apparent to a person skilled in the art that variations in the above method corresponding to operation of the various embodiments of the apparatus as described and claimed herein are considered within the scope of the present invention, including but not limited to that: the method comprises the steps of selecting a second volume within the frustum that at least partially encompasses the first volume within the frustum, and increasing the quality of a virtual mesh within the second volume by a second amount that is less than the first amount, as described elsewhere herein; the step of increasing the quality of a virtual mesh is achieved by increasing the target level of detail 'LoD' associated with the virtual mesh, as described elsewhere herein; the quality of the virtual mesh is increased by one or more selected from the list consisting of increasing polygon count and increasing mesh complexity, as described elsewhere herein; the quality of the virtual mesh is increased relative to the current quality of the virtual mesh within the frustum, as described elsewhere herein; in this instance, optionally the current quality of the virtual mesh is reduced in response to a computational budget for the second image, as described elsewhere herein; the steps of outputting for display (e.g. the first and second images) comprise projecting mesh coordinates from virtual environment coordinates (or equivalently frustum coordinates) to normalised device coordinates, as described elsewhere herein; in this instance, optionally the step of mapping the position comprises an inverse projection of normalised device coordinates to virtual environment coordinates (or equivalently frustum coordinates), as described elsewhere herein; Similarly in this instance, optionally the step of mapping the position comprises linking the indicated gaze position to a corresponding point in normalised device coordinates, as described elsewhere herein; a volume (e.g. the first or second volume) is comprised of one selected from the list consisting of at least one volume, in a predetermined array of volumes, encompassing the 3D position or an object upon which the 3D position is found; and a volume centred upon the 3D position or an object upon which the 3D position is found, as described elsewhere herein; and the step of outputting for display a second image comprises the step of foveated rendering of the image responsive to received information indicating a position in the displayed image where the user is looking, as described elsewhere herein. It will be appreciated that the above methods may be carried out on conventional hardware suitably adapted as applicable by software instruction or by the inclusion or substitution of dedicated hardware. Thus the required adaptation to existing parts of a conventional equivalent device may be implemented in the form of a computer program product comprising processor implementable instructions stored on a non-transitory machine-readable medium such as a floppy disk, optical disk, hard disk, solid state disk, PROM, RAM, flash memory or any combination of these or other storage media, or realised in hardware as an ASIC (application specific integrated circuit) or an FPGA (field programmable gate array) or other configurable circuit suitable to use in adapting the conventional equivalent device. Separately, such a computer program may be transmitted via data signals on a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks. Accordingly, in a summary embodiment of the present description, a mesh-based graphics system (100) (e.g. a computer or console, or a real or virtual cloud-based equivalent e.g. for streaming), comprises the following features. Firstly a game processor (110) (typically a CPU and / or GPU) configured (for example by suitable software instruction) to maintain a virtual environment, as described elsewhere herein; Secondly a rendering processor (140) (typically a CPU and / or GPU) configured (for example by suitable software instruction) to output for display an image of the virtual environment based upon a viewpoint of a virtual camera, as described elsewhere herein; Thirdly a gaze tracking unit (120) (typically a CPU and / or GPU) configured (for example by suitable software instruction) to receive information indicating a position in the displayed image where a user is looking (e.g. from one or more cameras inside a head-mounted display, or a camera tracking a viewer of a television), as described elsewhere herein; Fourthly a foveated meshlet processor (130) (typically a CPU and / or GPU) configured (for example by suitable software instruction) to map the position corresponding to where the user is looking to a 3D position within a frustum of the virtual camera that generated the displayed image, as described elsewhere herein; The foveated meshlet processor also being configured to select a first volume within the frustum that encompasses the 3D position, and being configured to increase the quality of a virtual mesh within the first volume by a first amount, as described elsewhere herein; and The rendering processor being configured to output for display a second image of the virtual environment based upon a viewpoint of the virtual camera, as described elsewhere herein. Instances of this summary embodiment implementing the methods and techniques described herein (for example by use of suitable software instruction) are envisaged within the scope of the application, including but not limited to that: the foveated meshlet processor is configured to select a second volume within the frustum that at least partially encompasses the first volume within the frustum, and to increase the quality of a virtual mesh within the second volume by a second amount that is less than the first amount, as described elsewhere herein; and the foveated meshlet processor is configured to increasing the quality of a virtual mesh by increasing the target level of detail 'LoD' associated with the virtual mesh, as described elsewhere herein. The foregoing discussion discloses and describes merely exemplary embodiments of the present invention. As will be understood by those skilled in the art, the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting of the scope of the invention, as well as other claims. The disclosure, including any readily discernible variants of the teachings herein, defines, in part, the scope of the foregoing claim terminology such that no inventive subject matter is dedicated to the public.

Claims

1. A mesh-based graphics method, comprising the steps of:outputting for display an image of a virtual environment based upon a viewpoint of a virtual camera;receiving information indicating a position in the displayed image where a user is looking;mapping the position corresponding to where the user is looking to a 3D position within a frustum of the virtual camera that generated the displayed image;selecting a first volume within the frustum that encompasses the 3D position;increasing the quality of a virtual mesh within the first volume by a first amount; andoutputting for display a second image of the virtual environment based upon a viewpoint of the virtual camera.

2. The mesh based graphics method of claim 1, comprising the steps of:selecting a second volume within the frustum that at least partially encompasses the first volume within the frustum; andincreasing the quality of a virtual mesh within the second volume by a second amount that is less than the first amount.

3. The mesh based graphics method of any preceding claim, in which the step of increasing the quality of a virtual mesh is achieved by increasing the target level of detail 'LoD' associated with the virtual mesh.

4. The mesh based graphics method of any preceding claim, in which the quality of the virtual mesh is increased by one or more selected from the list consisting of:i. increasing polygon count; andii. increasing mesh complexity.

5. The mesh based graphics method of any preceding claim, in which the quality of the virtual mesh is increased relative to the current quality of the virtual mesh within the frustum.

6. The mesh based graphics method of claim 5, in which the current quality of the virtual mesh is reduced in response to a computational budget for the second image.

7. The mesh based graphics method of any preceding claim, in which the steps of outputting for display comprise projecting mesh coordinates from virtual environment coordinates to normalised device coordinates.

8. The mesh based graphics method of claim 7, in which the step of mapping the position comprises an inverse projection of normalised device coordinates to virtual environment coordinates.

9. The mesh based graphics method of claim 7 or claim 8, in which the step of mapping the position comprises linking the indicated gaze position to a corresponding point in normalised device coordinates.

10. The mesh based graphics method of any preceding claim, in which a volume is comprised of one selected from the list consisting of:i. at least one volume, in a predetermined array of volumes, encompassing the 3D position or an object upon which the 3D position is found; andii. a volume centred upon the 3D position or an object upon which the 3D position is found.

11. The mesh based graphics method of any preceding claim, in which the step of outputting for display a second image comprises the step offoveated rendering of the image responsive to received information indicating a position in the displayed image where the user is looking.

12. A computer program comprising computer executable instructions adapted to cause a computer system to perform the method of any one of the preceding claims.

13. A mesh-based graphics system (100), comprising:a game processor (110) configured to maintain a virtual environment;a rendering processor (140) configured to output for display an image of the virtual environment based upon a viewpoint of a virtual camera;gaze tracking unit (120) configured to receive information indicating a position in the displayed image where a user is looking;foveated meshlet processor (130) configured to map the position corresponding to where the user is looking to a 3D position within a frustum of the virtual camera that generated the displayed image;the foveated meshlet processor being configured to select a first volume within the frustum that encompasses the 3D position;the foveated meshlet processor being configured to increase the quality of a virtual mesh within the first volume by a first amount; andthe rendering processor being configured to output for display a second image of the virtual environment based upon a viewpoint of the virtual camera.

14. The mesh-based graphics system of claim 13, in which:the foveated meshlet processor is configured to select a second volume within the frustum that at least partially encompasses the first volume within the frustum; andto increase the quality of a virtual mesh within the second volume by a second amount that is less than the first amount.

15. The mesh-based graphics system of claim 13 or claim 14, in which the foveated meshlet processor is configured to increasing the quality of a virtual mesh by increasing the target level of detail 'LoD' associated with the virtual mesh.

Images