Rendering method and system

Abstract

A method for rendering a virtual environment is provided. The method includes identifying one or more static elements in a virtual environment, determining a geometry of the static elements in the virtual environment for a first frame having a first virtual camera position; storing a geometry of a static element of the first frame; for a second frame having the first virtual camera position, determining a geometry of at least a portion of the virtual environment based at least in part on the stored geometry of the static element of the first frame; and for the second frame, determining illumination for the at least part of the virtual environment based at least in part on the geometry of the at least part of the virtual environment determined for the second frame to render the at least part of the virtual environment.

Description

Technical Field

[0001] The present invention relates to a method and system for rendering a virtual environment. Background Art

[0002] Developers of content (such as video games) are constantly striving to provide more realistic virtual environments. The quality of lighting effects often plays a key role in the user's perception of such environments and how realistic they appear to the user.

[0003] However, the process of rendering a virtual environment, and in particular determining the lighting for the virtual environment, is often complex and computationally expensive. For example, while light probes may provide an efficient way to obtain realistic lighting effects in a virtual environment, the process of generating or updating such light probes (which are typically provided in large numbers throughout an environment) may be prohibitive and may require the use of computationally expensive and specialized hardware.

[0004] The present invention seeks to alleviate or mitigate these problems and provide more efficient rendering. Summary of the invention

[0005] Various aspects and features of the present invention are defined in the text of the accompanying description, and include at least:

[0006] - In a first aspect, according to embodiments herein there is provided a method for rendering a virtual environment.

[0007] - In another aspect, a system for rendering a virtual environment is provided according to an embodiment of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present invention will be more readily understood by reference to the following detailed description taken in conjunction with the accompanying drawings, wherein:

[0009] - Figure 1 The entertainment equipment is schematically shown;

[0010] - Figure 2 is a schematic flow chart illustrating a method for rendering a virtual environment;

[0011] - Figure 3 is a schematic flow chart illustrating another method for rendering a virtual environment;

[0012] - Figure 4 schematically illustrates an optical probe; and

[0013] - Figure 5 A system for rendering a virtual environment is schematically shown. DETAILED DESCRIPTION

[0014] A method and system for rendering a virtual environment are disclosed. In the following description, many specific details are presented in order to provide a thorough understanding of embodiments of the present invention. However, it is apparent to those skilled in the art that these specific details need not be employed to practice the present invention. Instead, for the sake of clarity, specific details known to those skilled in the art have been omitted where appropriate.

[0015] In an example embodiment of the invention, a suitable system and / or platform for implementing the methods and techniques herein may be an entertainment device (such as PlayStation video game consoles).

[0016] Referring now to the drawings, wherein like reference numerals designate like or corresponding parts, Figure 1 An example of an entertainment device 10 is shown, which is a computer or console (such as PlayStation (PS5)).

[0017] The entertainment device 10 includes a central processor 20. This can be a single-core or multi-core processor, for example including eight cores (as in the PS5). The entertainment device also includes a graphics processing unit or GPU 30. The GPU can be physically separate from the CPU, or integrated with the CPU as a system on a chip (SoC) (as in the PS5).

[0018] The entertainment device also includes RAM 40, and may have separate RAM for each of the CPU and GPU, or shared RAM (as in the PS5). The or each RAM may be physically separate, or integrated as part of the SoC (as in the PS5). Further storage is provided by a disk 50, as an external or internal hard drive, or as an external solid state drive, or an internal solid state drive (as in the PS5).

[0019] The entertainment device may be suitably connected to the computer via one or more data ports 60 (such as a USB port, port, port, It can also optionally receive data via an optical drive 70.

[0020] Audio / video output from the entertainment device is typically provided through one or more A / V ports 90 or through one or more of the wired or wireless data ports 60 .

[0021] An example of a device for displaying images output by an entertainment device is a head mounted display "HMD" 120, such as a PlayStation VR 2 "PSVR2", worn by user 1. It should be understood that various other devices may be used to display content (e.g., using a conventional television display connected to A / V port 90).

[0022] Where components are not integrated, they may be connected by dedicated data links or via bus 100 as appropriate.

[0023] Typically one or more handheld controllers 130, 130A (such as in the case of the PS5 The controller 130 and / or one or more VR controllers 130A-L, R in the case of an HMD provide interaction with the system. The user typically interacts with the system and any content displayed by the system or the virtual environment rendered by the system by providing input via the handheld controller 130, 130A. For example, when playing a game, the user can navigate in the game virtual environment by providing input using the handheld controller 130, 130A.

[0024] In an embodiment of the present disclosure, the entertainment device 10 generates one or more images of a virtual environment for display (eg, via a television or HMD 120).

[0025] The images generated by the entertainment device 10 are each associated with a virtual camera position. In other words, the entertainment device 10 is configured to output images for display according to the position of the virtual camera.

[0026] It should be understood that the virtual camera has an associated viewpoint relative to the virtual environment such that a portion of the virtual environment is included within the field of view of the virtual camera. The viewpoint is defined by a virtual camera position, which may include the positioning of the virtual camera in the virtual environment and the orientation of the virtual camera relative to the virtual environment. In some cases, the virtual camera position may also include a magnification (e.g., zoom) of the virtual camera.

[0027] A virtual camera can follow a character in a virtual environment, thereby moving relative to the virtual environment. For example, a virtual camera can follow a non-player character (NPC) in a video game, or can be controlled in response to user input to follow a player-controlled character. In some examples, a virtual camera can provide a first or third person viewpoint of a given character in a game. Alternatively, a virtual camera can have a fixed position relative to the virtual environment, and can be controlled in response to input from a viewer (non-playing user) to update the orientation of the virtual camera. Alternatively, the virtual camera can be controlled in response to input from the viewer to allow the viewer to change the position of the virtual camera relative to the virtual environment without the viewer actively participating in the video game. Thus, the position (e.g., position and / or orientation) of the virtual camera can be updated, and the position of the virtual camera can be optionally controlled in response to user input.

[0028] In some examples, the position of the virtual camera can correspond to the tracked movement of the HMD 120 worn by the user, so that the viewpoint of the virtual camera is updated according to the user's head movement. The position and / or orientation of the HMD can be tracked using one or more image sensors and / or one or more inertial sensors (such as accelerometers, gyroscopes, and / or magnetometers). For example, known inside-out and / or outside-in tracking techniques using image sensors mounted on the HMD and / or mounted to capture images including the HMD can be used to track the position and orientation of the HMD. Therefore, in some cases, the viewpoint of the virtual camera can be controlled by the user to move relative to the virtual environment based on the movement of the HMD.

[0029] Operating under appropriate software instructions, an entertainment device or equivalent development kit, computer, or server may implement the methods for rendering a virtual environment as described herein.

[0030] Embodiments of the present disclosure relate to rendering a virtual environment in two stages, including a geometry stage that determines the geometry of the environment, and then a lighting stage that determines the lighting for the environment based on the geometry. Embodiments of the present disclosure allow rendering calculations to be efficiently reused between frames by determining the geometry of static elements in the environment of a first frame and storing (e.g., caching) the results for future use, and then determining the geometry of the environment in a second (e.g., next or any other subsequent) frame having the same virtual camera position as the first frame based on the stored geometry of the static elements of the first frame. Since the virtual camera position does not change between frames, the static geometry of the previous frame can be effectively reused to determine the geometry of the subsequent frame. The method allows the amount of calculation required to render the environment over the course of multiple frames to be reduced, thereby improving the efficiency of rendering the environment. The virtual environment may include a large number of static elements—and in some cases may include primarily static elements (e.g., in a video game). In addition, the virtual environment is typically rendered repeatedly at a high frequency (i.e., frame rate). Therefore, by reusing the determined geometry of static elements between frames and thereby improving the efficiency of rendering multiple frames, the method can provide considerable computational cost savings. In some cases, by improving rendering efficiency, the present disclosure may allow for an increase in the frame rate of content, thereby further improving the immersion of the content for the user.

[0031] As described in further detail below, the present method is particularly applicable to the generation and updating of Light Probes. Light Probes are typically provided in large numbers in a virtual environment and may need to be updated frequently (e.g., to reflect movement of objects in the environment or lighting changes). Light Probes are also static and have a fixed viewpoint, so the present method can allow the stored static geometry to be reused each time the Light Probes are updated, thereby reducing the computational cost of generating and updating Light Probes.

[0032] The present method is also applicable to the rendering of any other environment where the same virtual camera position may be used for multiple frames. For example, in various video games such as sports (e.g., tennis or soccer) games, the same virtual camera position may be reused over the course of the video game (e.g., a behind-the-court camera in tennis or a side-of-the-court camera in soccer). Similarly, for various content, the virtual camera may often be stationary for a period of time—for example, in video games, a user may often stop in a virtual environment for multiple frames to explore their surroundings in detail. Thus, by reusing geometry calculations between frames, the present method allows the computational cost of rendering such environments to be reduced.

[0033] Figure 2 An example of a method for rendering a virtual environment according to one or more embodiments of the present disclosure is shown.

[0034] A virtual environment of content such as a video game typically includes a plurality of corresponding elements (ie, graphical features).

[0035] It should be understood that the term "static element" as used herein relates to an element in a virtual environment that is static (i.e., does not move) relative to an origin or other reference point of the virtual environment. Static elements may remain static during execution of a rendering method, such as during a game session or a portion (e.g., a level) of a game or other content. Static elements may, for example, include scenery of a virtual environment and / or any other element of the environment that does not move. Examples of static elements in a virtual environment include the ground (e.g., a pavement or road), a building, a mountain, or a tree; or portions thereof.

[0036] In contrast, it should be understood that the term "dynamic element" refers to an element in a virtual environment that can move relative to an origin or other reference point of the virtual environment. Examples of dynamic elements in a virtual environment include player characters, vehicles, or non-player characters (NPCs).

[0037] Return to reference Figure 2 , step 210 includes identifying one or more static elements in the virtual environment.

[0038] Static elements may be identified based on predefined attributes associated with elements in the virtual environment. For example, a predefined tag may be assigned to each element in the virtual environment, such as by a developer or other user, to indicate whether the element is static or dynamic—e.g., a game developer may preset which elements in the environment are movable and which are not. Static elements may then be identified based on the predefined tags of the elements in the virtual environment.

[0039] Alternatively or additionally, static elements can be identified by analyzing the movement of elements in the virtual environment over the course of multiple frames of the content. For example, when it is determined that an element has not moved during a predetermined number of frames, the element can be classified as static. The analysis of the movement of the element can be based on a velocity buffer of pixels in each frame, for example. For example, an element with a zero velocity buffer during a predetermined number of frames can be considered static. The analyzed frames can be in the same session (e.g., a game session) of the rendered environment and / or in one or more historical sessions. In some cases, a predefined tag can be assigned to an element based on the analysis of the movement so that in subsequent sessions, the tag can be used to identify static elements, as described above.

[0040] It should be understood that the size of elements in the virtual environment (whether static or dynamic) may vary, and each element may comprise one or more pixels.

[0041] It should also be understood that step 210 may include identifying all or only a subset of the static elements present in the virtual environment. As discussed in further detail below, the geometry of the static elements is stored in step 230. Therefore, the amount of storage required may increase with the number (and / or geometric complexity) of the static elements identified; and in some cases, it may be appropriate to identify only a subset of the static elements in the virtual environment (and store their geometry) to reduce memory usage.

[0042] Step 220 includes determining the geometry of static elements in the virtual environment for a first frame having a first virtual camera position.

[0043] Determining the geometry of the static element may include rendering the static element to determine geometry information of the static element. The geometry information may be determined for each fragment of the static element. The geometry information may include one or more of: a position vector, a color vector, a normal vector, and / or a specular value.

[0044] The geometry information may include depth information of the static element from the viewpoint of the first virtual camera position. For example, the geometry information may include depth information corresponding to the depth information typically stored in the Z buffer. As described below, storing the depth information of the static element allows the determined geometry of the static element to be efficiently reused in subsequent frames.

[0045] In some cases, the geometry information may include information determined in the geometry stage of a deferred shading process for storage in a G-buffer. In some cases, determining the geometry of a static element may include determining data required for pixel shading calculations of the static element. For example, the static element may be rendered by rasterizing the static element into a texture map (also referred to as a texture buffer).

[0046] A first frame may relate to any frame of content, where the term "first" merely indicates a distinction from a subsequent "second" frame. In some cases, a first frame may relate to a frame that is not displayed to any user (e.g., a loading frame), in other words, the geometry of static elements may be determined (e.g., baked) before the content is displayed to a user, or the content is rendered for display to a user. For example, the geometry of static elements may be determined on level load or asset baking in a video game.

[0047] It should be understood that at step 220, lighting for static elements is typically not calculated and only geometry information is collected. Omitting lighting calculations, which are typically more computationally expensive, can allow for reduced computing resource usage. Furthermore, since only static elements are rendered at step 220 and dynamic elements are omitted, the lighting calculated only for static elements may not be accurate for the entire virtual environment.

[0048] Step 230 includes storing the geometry of the static elements of the first frame. In this way, the static element geometry can be cached for future use in subsequent frames.

[0049] The static element geometry may be stored in any suitable type of storage device and / or in any number of storage devices. For example, when the entertainment device 10 executes the method, the static element geometry may be stored in RAM 40 and / or disk 50. Alternatively or additionally, the static element geometry may be stored on one or more external devices, such as on a cloud server.

[0050] In some cases, the type of storage device used to store static element geometry may be selected based on one or more predetermined conditions, such as the frame rate of the content or the amount of memory required to store the geometry. For example, in situations where the frame rate is higher (and therefore a faster retrieval of the stored static element geometry is required) and / or where the amount of memory required is lower, a faster access storage device (such as SRAM) may be used, while in situations where the frame rate is lower or the amount of memory required is higher, a cheaper but slower storage device (such as a disk) may be used.

[0051] In some cases, static element geometry may initially be stored in a buffer (sometimes referred to as a G-buffer) for use in determining lighting for a first frame. The static element geometry may then be transferred from the buffer to a storage device for use in subsequent frames.

[0052] Step 240 includes determining, for a second frame having a first virtual camera position, a geometry of at least a portion of the virtual environment based at least in part on the geometry of static elements of the stored first frame. Reusing the stored geometry of static elements of the first frame in this manner allows for increased efficiency in determining the geometry of the second frame.

[0053] Determining the geometry of the virtual environment (or at least a portion thereof) of the second frame may include determining the geometry of dynamic elements in the virtual environment of the second frame, and combining the stored geometry of static elements of the first frame with the determined geometry of the dynamic elements of the second frame. Determining only the geometry of the dynamic elements of the second frame and combining it with the stored geometry of the static elements of the first frame eliminates the need to repeatedly redetermine the geometry of the entire environment and allows for reduced computational costs of determining geometry for multiple frames.

[0054] One or more dynamic elements in the virtual environment may be identified using the techniques described with respect to identifying static elements at step 210, which may be applied, for example, in reverse to identify dynamic elements. For example, dynamic elements may be identified based on predefined tags assigned to dynamic elements in the environment, and / or by analyzing movement of elements in the virtual environment over the course of multiple frames of the content to identify elements that have moved to at least a predetermined extent.

[0055] Dynamic elements and static elements can be considered as complementary sets of elements in the virtual environment. In other words, any element that is not identified as static can be considered dynamic, and vice versa. Therefore, in some cases, dynamic elements can be identified as all elements that are not identified as static at step 210. It should be understood that, similarly, at step 210, static elements can be identified based on previous identification of dynamic elements, wherein static elements are elements that are not identified as dynamic.

[0056] The geometry of the dynamic elements of the second frame may then be determined using the same techniques as described with reference to determining the geometry of the static elements at step 220 .

[0057] Once determined, the geometry of the dynamic elements of the second frame can be combined with the stored geometry of the static elements of the first frame. The geometry of the static elements and the dynamic elements can be combined based on the relative depths of the static elements and the dynamic elements from the first virtual camera position. The relative depths of the elements can be determined based on depth information (e.g., a depth buffer) determined as part of determining the corresponding geometry of the elements. Based on the relative depths of the static elements and the dynamic elements, portions of the elements that are occluded from the viewpoint of the virtual camera can be discarded to determine the combined geometry of the virtual environment.

[0058] Combining the geometry of static and dynamic elements may include creating a copy of the stored static geometry and writing the dynamic geometry on top of the copy of the static geometry based on the relative depths of the static elements and the dynamic elements, thereby overwriting the copy with the determined geometry of the environment including the static elements and the dynamic elements, while avoiding changing the stored static geometry so that it can be used in other frames.

[0059] It should be understood that step 240 may include determining the geometry of part or all of the virtual environment of the second frame. Typically, the geometry of the portion of the virtual environment within the viewpoint of the virtual camera of the second frame is determined. In some cases, other portions of the geometry may be determined for use in future frames, for example, portions of the environment outside the viewpoint of the virtual camera may be preemptively rendered for quick access in future frames.

[0060] Step 250 includes determining lighting for at least a portion of the virtual environment for the second frame based at least in part on the geometry of at least a portion of the virtual environment for the second frame determined at step 240 to render at least a portion of the virtual environment.

[0061] In an embodiment of the present disclosure, the virtual environment (or portion thereof) is rendered in two stages including a first "geometry" stage and a second "lighting" stage. Step 240 relates to the first geometry stage, while step 250 relates to the second lighting stage.

[0062] Determining the lighting for the second frame based on the geometry of the virtual environment determined at step 240 may include determining one or more output pixels (i.e., pixels for output to a display), and performing a lighting operation on each output pixel. The lighting operations may each include a shading operation. The lighting operations may be performed by a pixel shader that determines the lighting (e.g., direct and / or indirect lighting) for each output pixel based on the determined geometry of the environment.

[0063] Separating the rendering process into a geometry phase and a lighting phase allows for improved efficiency of the rendering environment, because the geometry determined at step 240 provides information related to the fragments to be provided as output pixels for display (e.g., determined based on the depth information of the fragments), and thus the lighting only needs to be determined once for each output pixel. Therefore, the number of lighting operations can be reduced compared to a forward rendering method in which the geometry and lighting phases are not separated.

[0064] Of course, it should be understood that the techniques described herein can be extended to any number of frames. For example, the static element geometry of the first frame can be reused to determine the geometry of multiple subsequent frames with the same virtual camera position, thereby providing further efficiency improvements.

[0065] Figure 3 Another example method for rendering a virtual environment according to one or more embodiments of the present disclosure is shown.

[0066] Step 302 includes determining and storing the geometry of one or more static elements in the virtual environment for a first frame having a first virtual camera position. Figure 2 Steps 210-230 of the method describe a technique to identify static elements and determine their geometry.

[0067] Step 304 includes determining the virtual camera position of the second frame and detecting whether the virtual camera position has changed between the first frame and the second frame (ie, whether the virtual camera has moved).

[0068] If it is determined at step 304 that the virtual camera positions of the first frame and the second frame are the same (ie, the second frame has the same first virtual camera position), the method proceeds to step 306 .

[0069] At step 306, movement of one or more dynamic elements in the virtual environment between the first frame and the second frame is detected. Figure 2 The method describes a technique to detect dynamic elements and their movement, for example, based on a velocity buffer of pixels associated with elements of a virtual environment.

[0070] In some cases, step 306 may include detecting movement of dynamic elements within at least a portion of the environment being rendered. For example, movement of dynamic elements within the field of view of the virtual camera of the second frame may be detected, while movement of dynamic elements outside the field of view may be ignored. As discussed with respect to step 314, if movement of dynamic elements is not detected at step 306, the geometry of the environment determined for the first frame may be reused to determine the lighting for the second frame. Since dynamic elements outside of at least a portion of the environment do not affect the geometry of that portion of the environment of the second frame, detecting movement of dynamic elements only within at least a portion of the environment (e.g., within the field of view of the virtual camera) allows efficient reuse of geometry from the first frame and further reduces the computational cost of rendering the environment.

[0071] It should be understood that detecting movement of dynamic elements in step 306 may include detecting addition of new dynamic elements, removal of previous dynamic elements, and / or any other changes to dynamic elements, such as changes in their shape (e.g., damage to a car element in the virtual environment).

[0072] If movement of at least one dynamic element between the first frame and the second frame is detected at step 306 , the method proceeds to step 308 .

[0073] At step 308, the geometry of at least a portion of the virtual environment is determined for the second frame based at least in part on the geometry of the static elements of the first frame determined and stored at step 302. Figure 2 Step 240 of the method describes a technique for determining the geometry of the environment of the second frame.

[0074] At step 310, lighting for at least a portion of the virtual environment of the second frame is determined based at least in part on the geometry of the second frame determined at step 308 in order to render at least a portion of the virtual environment. Figure 2 Step 250 of the method describes a technique for determining the lighting of the environment for the second frame.

[0075] Therefore, it should be understood that Figure 3Steps 302-310 of the method may correspond to Figure 2 Steps 210-250 of the method. Figure 3 Various further steps of the method relate to further extensions of the rendering method, which allow to further improve the efficiency of the rendering environment.

[0076] If, at step 306 , no movement of the dynamic element is detected between the first frame and the second frame, the method proceeds to step 312 .

[0077] At step 312, a change in one or more light sources in the virtual environment between the first frame and the second frame is detected. The change in the light source may include one or more of the following: the addition of one or more light sources (i.e., adding a light source that does not exist in the first frame to the second frame), the removal of one or more light sources (i.e., removing a light source that exists in the first frame from the second frame), or a change in one or more characteristics of the light source (such as luminosity, color, or position in the virtual environment). For example, the change in the light source may be detected from an engine that renders the content (e.g., a game engine for video game content).

[0078] Where step 306 includes detecting movement of only a dynamic element within at least a portion of the environment (e.g., within the field of view of the virtual camera of the second frame), step 312 may include detecting changes in the light source and movement of one or more other dynamic elements (such as dynamic elements that are outside at least a portion of the environment or outside the field of view of the virtual camera of the second frame). As discussed with respect to step 316, if no change in lighting is detected at step 312, the lighting of the environment determined for the first frame may be reused for the second frame. Because movement of a dynamic element may affect the lighting in the environment even if the element is outside at least a portion of the environment (e.g., if the dynamic element is reflective), detecting movement of one or more other dynamic elements outside at least a portion of the environment at step 312 helps ensure that lighting is not reused between frames when the lighting has actually changed.

[0079] In some cases, step 312 may include detecting changes in a subset of light sources and / or a subset of dynamic elements in the virtual environment. These subsets may be selected based on one or more predetermined conditions. A subset of light sources and / or dynamic elements whose changes are more likely to affect the lighting in the environment of the second frame may be selected based on these predetermined conditions. Example predetermined conditions for selecting a subset of dynamic elements may include: the distance between the dynamic elements and the virtual camera in the second frame (and / or between the dynamic elements and the volume of the environment falling within the viewpoint of the virtual camera), one or more optical properties of the dynamic elements (such as their reflectivity), and the shape and / or size of the dynamic elements. Example predetermined conditions for selecting a subset of light sources may include: the distance between the light source and the virtual camera in the second frame (and / or between the light source and the volume of the environment falling within the viewpoint of the virtual camera), the luminosity of the light source, the direction of the light source, the color of the light source, and the light cone of the light source. For example, a subset of light sources within a predetermined distance of the virtual camera and pointing toward the virtual camera may be selected; or a subset of dynamic elements with a reflectivity above a predetermined threshold may be selected. At step 312, changes to light sources and / or dynamic elements only within the selected subset may be tracked (i.e., detected), and only such changes may trigger a recalculation of the lighting at step 314. This allows reducing the number of lighting recalculations that may be required, and therefore allows improving the efficiency of the rendering method.

[0080] It will be appreciated that a light source may be a dynamic element (eg, the lights on a car, or the sun) or a static element (eg, a street lamp).

[0081] If a change in at least one light source between the first frame and the second frame is detected at step 312 , the method proceeds to step 314 .

[0082] At step 314, lighting for the second frame is determined based on the geometry of at least a portion of the virtual environment previously determined for the first frame. In other words, the geometry of the environment determined for the first frame is reused for the second frame, which allows for a reduction in the amount of computation required to render the second frame, since the results from the geometry stage for the first frame can be reused, and only the lighting stage is required to account for potential changes in lighting caused by changes in the light source detected at step 312. The lighting for the second frame can be determined based on the geometry of the first frame using the same techniques as described with respect to step 310.

[0083] If at step 312 , no change in the light source between the first frame and the second frame is detected, the method proceeds to step 316 .

[0084] At step 316, instead of re-determining the lighting for the second frame (as, for example, at steps 310 or 314), the lighting determined for the first frame is reused for the second frame. In effect, the first frame is thus reused for the second frame. In this way, the computationally expensive lighting determination is completely omitted for the second frame, and the efficiency of the rendering method is further improved. Since it has been previously determined that the dynamic element has not moved (at step 306) and the light source has not changed (at step 312), any resulting reduction in the quality or inaccuracy of the rendering is also reduced (or preferably avoided).

[0085] If it is determined at step 304 that the virtual camera position has changed between the first frame and the second frame (i.e., the second frame has a different second virtual camera position), the method proceeds to step 318. For example, the virtual camera position may change when a user moves a player avatar or when a developer repositions a light probe.

[0086] Step 318 includes determining and storing the geometry of the static elements in the virtual environment for the second frame. In other words, the geometry of the static elements is recalculated for the second frame to account for the movement of the virtual camera so that the geometry of the static elements of the second frame can be reused for subsequent frames, for example, in the same manner as described herein, the geometry of the static elements of the first frame can be reused for the second frame.

[0087] In some cases, step 318 may be performed according to one or more predetermined conditions. The predetermined condition may provide a measure of the likelihood that the second virtual camera position will be used in a subsequent frame, such that static element geometry is calculated and stored only for frames whose virtual camera position is likely to be used by another frame. The predetermined condition may, for example, be related to: a proportion of static elements in the second frame, a frequency of the second virtual camera position in previous frames, or a user-defined flag. For example, static element geometry may be determined for the second frame only if the second virtual camera position has been used in a predetermined number of previous frames (including at least a predetermined number of static elements), and / or has been flagged as likely to be reusable by a user (e.g., a game developer).

[0088] The static element geometry of the first frame may be retained in storage independent of whether the virtual camera has moved, since further future frames may have the first virtual camera position and therefore will benefit from the stored static element geometry of the first frame.

[0089] Alternatively, the method may proceed to step 320 after step 318. At step 320, the stored static element geometry for the first frame is removed from the storage device to reduce memory usage. The geometry information of the static elements in the frame may require a large amount of memory, so flushing the static element geometry cache in this way helps ensure that memory usage is not very high.

[0090] In some cases, the stored static element geometry of the first frame can be removed from the storage device according to one or more predetermined conditions. The predetermined condition can provide a measure of the likelihood that the first virtual camera position will be used in a subsequent frame. The predetermined condition can be related to, for example, the recency of the first frame (e.g., in number of frames, or real or content (e.g., in-game) time), the frequency of the first virtual camera position in previous frames, the game associated with the virtual environment, or a tag assigned by the user (e.g., to indicate a frequently reused virtual camera position). For example, if the first frame is an old frame, or if the first frame (or its virtual camera position) is not marked as frequently reusable, the static element geometry of the first frame can be removed from the storage device. Alternatively or additionally, if the first virtual camera position is used less than a predetermined number of times in previous frames, the static element geometry of the first frame can be removed from the storage device.

[0091] It should be understood that Figure 3 The rendering method allows the computational cost of rendering the environment to be further reduced by efficiently reusing geometry and / or lighting calculations between frames.

[0092] It should also be understood that Figure 3 Various steps of the method may be optional. For example, in some examples, step 304 (i.e., determining whether the virtual camera has moved between the first frame and the second frame) and associated steps 318 and 320 may be omitted, and assuming that the virtual camera has not moved (which may be suitable for a light probe, for example), the method may proceed directly from step 302 to step 306. Alternatively or additionally, in some examples, step 306 (i.e., determining whether the dynamic object has moved between the first frame and the second frame) and associated steps 312-316 may be omitted, and assuming that the dynamic object has moved, the method may proceed directly from step 304 to steps 308 and 310. Alternatively or additionally, in some examples, step 312 (i.e., determining whether the light source has changed between the first frame and the second frame) and associated step 316 may be omitted, and assuming that the light source has changed, the method may proceed directly from step 306 to step 314.

[0093] It should also be understood that Figure 2 and Figure 3 The various steps of the method may be performed in any suitable combination.

[0094] Figure 2 and / or Figure 3 The rendering method may include various additional steps. These additional steps may include one or more of the following:

[0095] - selecting a frame as a first frame according to one or more predetermined conditions; and

[0096] - Identify and reuse elements in the virtual environment that are fixed relative to the virtual camera (also known as

[0097] "Fixed Elements").

[0098] Considering selecting a first frame, a frame may be selected as a first frame among multiple frames of the content based on one or more predetermined conditions. The predetermined conditions may be similar to those described with reference to determining whether to calculate the static element geometry of the second frame at step 318. For example, the first frame may be selected based on the proportion of static elements in each frame and / or the distribution of virtual camera positions between multiple frames. For example, the first frame may be selected as the frame whose virtual camera position is most frequently used among multiple frames. When selecting the first frame, the proportion of static elements in the frame may also be considered, for example, to select a frame with a higher proportion of static elements (so that more geometric calculations can be reused between frames). Alternatively, frames with a lower proportion of static elements may be prioritized to reduce the amount of memory required to store the determined geometry of the static elements.

[0099] In some cases, the virtual camera position can be tracked over the course of multiple frames (e.g., using a counter) to determine how often each virtual camera position is repeated. Static element geometry can then be determined based on this tracking—e.g., the static element geometry can be determined and stored for any frame whose corresponding virtual camera position has been repeated at least a predetermined number of times. This allows caching of static element geometry for multiple frequently reused virtual camera positions, further reducing the computational cost of rendering multiple frames. In some cases, the selection of the first frame can be performed based on an analysis of virtual camera positions and user behavior in multiple previous sessions of the content, for example, to identify the most frequent virtual camera position.

[0100] With regard to fixed elements, the term "fixed element" may relate to an element that is fixed (i.e., has substantially the same position) relative to a virtual camera, regardless of the position of the virtual camera. Fixed elements may be static, but are typically dynamic, in other words, fixed elements typically move relative to the origin of the virtual environment, but are fixed relative to the virtual camera. Examples of such fixed elements may include: an avatar hand when the viewpoint corresponds to a first-person view, a portion of an avatar when the viewpoint corresponds to a third-person view, a portion of a helmet when the viewpoint corresponds to a first-person view, or one or more portions of a user's car in a driving game. For example, an element such as a helmet worn by a character in the game or a weapon held by a character in the game may be included in an image frame output for display, such that the position of the helmet or weapon in the image frame remains unchanged, regardless of changes in the position and / or orientation of the viewpoint of the virtual camera. For example, in some first-person shooter games, a virtual helmet or other similar structure may be included in an image frame to provide a viewer with the impression of viewing a scene as if they were wearing a helmet, and thus, even if the virtual camera moves relative to the virtual environment, the position and appearance of the helmet remains unchanged relative to a series of image frames generated for the virtual camera.

[0101] In some cases, Figure 2 and / or Figure 3 The rendering method may include identifying one or more fixed elements in the virtual environment; determining and storing the geometry of the fixed elements in the virtual environment for a first frame; and determining the geometry of the second frame based at least in part on the stored geometry of the fixed elements of the first frame.

[0102] Fixed elements in the virtual environment may be identified, for example, based on a user's pre-selection (e.g., a game developer may mark fixed elements in the environment), or based on content changes between multiple frames (e.g., where pixels that appear in the same position for a predetermined number of frames may be determined to be fixed). Figure 2 The geometry of the fixed elements of the first frame can then be determined and stored using techniques similar to those described in steps 220 and 230 of the method. Figure 2 The method of step 240 may be similar to the technique described above, using the stored fixed element geometry of the first frame to determine the geometry of the second frame. For example, the fixed element geometry of the first frame may be combined with the dynamic element geometry of the second frame and / or with the static element geometry of the first frame to more efficiently determine the environment geometry of the second frame. Fixed elements are often arranged near the virtual camera and occlude other elements, so reusing fixed element geometry in this manner may further reduce the computational cost of the rendering process.

[0103] In some cases, the fixed element geometry of the first frame may be used to determine the geometry of the second frame even when the second frame has a different virtual camera position than the first frame, because the fixed elements are static relative to all virtual camera positions.

[0104] As described above, the present disclosure may be particularly applicable to the generation and updating of optical probes.

[0105] It will be appreciated that a typical virtual environment, such as may be used in content such as video games or as a TV or movie set, may include ambient light (eg generated by a so-called skybox) and / or directional light (typically from light sources within the environment).

[0106] Another form of lighting comes from global illumination ("GI"), which takes into account secondary light sources, such as bounced or reflected light, or optionally emitted light from textured objects. Similarly, reflections can redirect light toward other surfaces and toward the virtual camera.

[0107] Calculation of this secondary light may be performed using global illumination probes ("GI probes"), and optionally also reflection probes, which may be more generally referred to as light probes.

[0108] Light Probes provide a way to capture information about light traveling through empty spaces in a virtual environment (e.g., in the spaces between buildings). In other words, Light Probes can be thought of as textures that capture lighting information about the virtual environment around them. This information can be stored in Light Probes to illuminate objects in the environment at runtime of content (e.g., a video game). For example, multiple Light Probes can be placed throughout a virtual environment, and the Light Probes can be used to provide lighting effects on moving objects in the virtual environment (e.g., cars between buildings)—for example, the lighting effects on a moving object can be approximated by interpolating between Light Probes adjacent to the object.

[0109] Lighting information for light probes can be captured once before runtime (e.g., baked) and used throughout the runtime of the content. Such light probes are sometimes called static or baked light probes.

[0110] Alternatively, the lighting information of a Light Probe can be updated to react to changes in the virtual environment around it in real time. Such Light Probes are sometimes referred to as dynamic, real-time, or non-baked Light Probes. Using dynamic Light Probes allows more realistic lighting effects to be provided because changes in the environment are incorporated into the information stored in the Light Probes. However, this comes with a high computational cost because the dynamic Light Probes need to be updated when the virtual environment changes, such as when an object moves or lighting conditions change (e.g., the time of day changes). Therefore, the process of generating and updating dynamic Light Probes can be computationally expensive.

[0111] Figure 4 A non-limiting example of a light probe 400 is schematically shown. The light probe 400 includes faces 402 (e.g., faces 402-a, 402-b, 402-c) and a center 404. The faces 402 represent the environment as viewed from the center 404 of the probe. Each face 402 is divided into a grid of pixels (not shown). It should be understood that the faces 402 of the probe can represent a texture. Therefore, the "pixels" on the faces can be considered "texels". In addition, although in Figure 4 In the example shown, the light probe 400 is shaped as a cube having six faces 402, but it should be understood that the light probe may have other shapes, such as a spherical shape.

[0112] Generating (or updating) a Light Probe may comprise an environment map using the faces of the Light Probe as a graph shape. The environment map may comprise projecting the virtual environment onto the faces of the Light Probe and storing the resulting textures - in other words, rendering the virtual environment onto the faces of the Light Probe. For the example cube Light Probe 400, the environment map may comprise a cube map, whereby the environment is projected onto the six faces 402 of the Light Probe 400. Thus, a cube map of the environment may be generated by rendering the virtual environment from the viewpoint of the center 404 of the Light Probe for each Light Probe face 402. For a cube Light Probe 400, six virtual environment rendering operations may therefore be required to generate a Light Probe 400 or to update a Light Probe 400 (e.g., to account for the movement of objects in the environment).

[0113] Typically a large number of light probes are provided in a virtual environment, and for each light probe, each face of the probe may require a separate rendering operation. Therefore, it should be understood that the process of generating and updating dynamic light probes in an environment may require a large number of rendering operations and may have a high associated computational cost.

[0114] The present method allows reducing this computational cost and improving the efficiency of updating dynamic Light Probes. For a given Light Probe, the virtual camera position used to render the environment to the Light Probe is defined by the center and face of the Light Probe. Therefore, the virtual camera position does not move between frames, and static element geometry can be reused for each subsequent frame (assuming the Light Probe itself has not moved). In addition, when rendering the environment to the Light Probe face from each viewing direction, the environment rendered to one or more faces of the probe may consist primarily of static elements (for example, at least the probe face facing the ground will typically consist primarily of static elements). Therefore, caching and reusing static element geometry can provide significant efficiency improvements when updating Light Probes, as it eliminates the need to repeatedly determine the geometry of the environment from scratch each time the Light Probe is updated.

[0115] Of course, the techniques described herein can be extended to generate and update multiple light probes in one or more virtual environments.

[0116] Return to reference Figure 2 In an embodiment of the present invention, a method for rendering a virtual environment includes the following steps.

[0117] Step 210 includes identifying one or more static elements in the virtual environment, as described elsewhere herein.

[0118] Step 220 includes determining the geometry of static elements in the virtual environment for a first frame having a first virtual camera position, as described elsewhere herein.

[0119] Step 230 includes storing the geometry of the static elements of the first frame, as described elsewhere herein.

[0120] Step 240 includes determining, for a second frame having a first virtual camera position, a geometry of at least a portion of the virtual environment based at least in part on the stored geometry of static elements of the first frame, as described elsewhere herein.

[0121] Step 250 includes determining lighting for at least a portion of the virtual environment for the second frame based at least in part on the geometry of at least a portion of the virtual environment determined for the second frame to render at least a portion of the virtual environment as described elsewhere herein.

[0122] It will be apparent to those skilled in the art that variations of the above methods corresponding to the operation of the various embodiments of the methods and / or apparatus described and claimed herein are considered to be within the scope of the present disclosure, including but not limited to:

[0123] - the method is for generating a light probe in a virtual environment, and at least a portion of the virtual environment is rendered to one or more faces of the light probe, as described elsewhere herein;

[0124] - the method further comprises identifying one or more dynamic elements in the virtual environment as described elsewhere herein;

[0125] - in this case, optionally, determining 240 the geometry of at least a portion of the virtual environment of the second frame comprises: determining the geometry of dynamic elements of the second frame and combining the stored geometry of static elements of the first frame and the determined geometry of dynamic elements of the second frame, as described elsewhere herein;

[0126] ■ wherein, optionally, the stored geometry of the static elements and the determined geometry of the dynamic elements are combined according to the relative depths of the static elements and the dynamic elements from the first virtual camera position, as described elsewhere herein;

[0127] - in this case, optionally, the step 240 of determining the geometry of the second frame is performed in dependence on detecting a movement of at least one of the dynamic elements between the first frame and the second frame, as described elsewhere herein;

[0128] ■ wherein, optionally, the method further comprises determining a geometry of at least a portion of the virtual environment of the first frame; wherein upon detecting that the dynamic element has not moved between the first frame and the second frame, determining 250 lighting for the second frame based at least in part on the determined geometry of at least a portion of the virtual environment of the first frame, as described elsewhere herein;

[0129] • wherein, optionally, the step 250 of determining the lighting for the second frame is performed in response to detecting a change in at least one light source in the virtual environment between the first frame and the second frame, as described elsewhere herein;

[0130] ● wherein, optionally, the method further comprises determining lighting for at least a portion of the virtual environment for the first frame; wherein upon detecting that the light source has not changed between the first frame and the second frame, determining 250 the lighting for the second frame comprises reusing the lighting determined for the first frame, as described elsewhere herein;

[0131] - determining 250 lighting for at least a portion of the virtual environment for a second frame based on the determined geometry of at least a portion of the virtual environment comprises: determining one or more output pixels for output to a display, and performing a lighting operation on each output pixel, as described elsewhere herein;

[0132] - the method further comprises determining a virtual camera position for a second frame, and upon detecting that the second frame has a second, different virtual camera position: determining the geometry of static elements in the virtual environment for the second frame, and storing the geometry of the static elements for the second frame, as described elsewhere herein;

[0133] - in this case, optionally, determining and storing the geometry of the static elements of the second frame is performed according to one or more predetermined conditions, the predetermined conditions relating to one or more selected from a list consisting of: the proportion of the static elements in the second frame, the frequency of the second virtual camera position in previous frames, or a tag assigned by a user to the second frame and / or the second virtual camera position, as described elsewhere herein;

[0134] - the method further comprises removing the geometry of the static elements of the first frame from the storage device, as described elsewhere herein;

[0135] - in this case, optionally, removing the geometry of the static elements of the first frame from the storage means is performed according to one or more predetermined conditions, the predetermined conditions relating to one or more selected from a list consisting of: recency of the first frame, a game associated with the virtual environment, a frequency of the first virtual camera position in previous frames, or a tag assigned by a user to the first frame and / or the first virtual camera position, as described elsewhere herein;

[0136] -The method further comprises: identifying one or more fixed elements in the virtual environment, the fixed elements being fixed relative to the virtual camera regardless of the position of the virtual camera; determining the geometry of the fixed elements in the virtual environment for the first frame; storing the geometry of the fixed elements for the first frame; and determining the geometry of the second frame based at least in part on the stored geometry of the fixed elements for the first frame, as described elsewhere herein;

[0137] - in this case, optionally, upon detecting that the second frame has a different second virtual camera position, determining the geometry of the second frame based at least in part on the stored geometry of fixed elements of the first frame, rather than the stored geometry of static elements of the first frame, such as in the present invention

[0138] As described elsewhere in this article;

[0139] - the method further comprises selecting a frame as a first frame among the plurality of frames according to one or more predetermined conditions, as described elsewhere herein; and

[0140] - In this case, optionally, the one or more predetermined conditions comprise one or more selected from a list consisting of: a proportion of static elements in a plurality of frames, or a distribution of virtual camera positions in a plurality of frames, as described elsewhere herein.

[0141] It should be understood that the above-described methods may be performed on conventional hardware appropriately adapted to the application by software instructions or by the inclusion or replacement of dedicated hardware.

[0142] Thus, the required adaptation of existing parts of conventional equivalent devices can be implemented in the form of a computer program product comprising processor-implementable instructions stored on a non-transitory machine-readable medium (such as a floppy disk, optical disk, hard disk, solid-state disk, PROM, RAM, flash memory, or any combination of these or other storage media), or implemented in hardware as an ASIC (application specific integrated circuit) or FPGA (field programmable gate array) or other configurable circuit suitable for adapting conventional equivalent devices. Separately, such a computer program can be sent via a data signal on a network such as Ethernet, a wireless network, the Internet, or any combination of these or other networks.

[0143] Reference Figure 5 In an embodiment of the present invention, a system for rendering a virtual environment may include the following:

[0144] The identification processor 510 is configured (eg, via suitable software instructions) to identify one or more static elements in the virtual environment, as described elsewhere herein.

[0145] A first geometry processor 520 is configured to determine (eg, via suitable software instructions) the geometry of static elements in the virtual environment for a first frame having a first virtual camera position, as described elsewhere herein.

[0146] Storage processor 530 is configured to store (eg, via suitable software instructions) the geometry of the static elements of the first frame, as described elsewhere herein.

[0147] A second geometry processor 540 is configured (e.g., via appropriate software instructions) to determine the geometry of at least a portion of the virtual environment for a second frame having a first virtual camera position based at least in part on the geometry of static elements of the stored first frame, as described elsewhere herein.

[0148] The lighting processor 550 is configured (e.g., via appropriate software instructions) to determine lighting for at least a portion of the virtual environment for a second frame based at least in part on the geometry of at least a portion of the virtual environment determined for the second frame to render at least a portion of the virtual environment, as described elsewhere herein.

[0149] These processors 510-550 may be implemented, for example, by the CPU 20 of the entertainment system 10. Of course, it should be understood that the functionality of these processors may be implemented by any suitable number of processors located at any suitable number of devices and any suitable number of devices as appropriate, without requiring a one-to-one mapping between functionality and devices or processors.

[0150] The foregoing discussion discloses and describes only exemplary embodiments of the present invention. As will be appreciated by those skilled in the art, the present invention may be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the disclosure of the present invention is intended to be illustrative, rather than limiting, the scope of the present invention and other claims. The present disclosure, including any readily discernible variations taught herein, defines in part the scope of the preceding claim terms so that no inventive subject matter is dedicated to the public.

Claims

1. A method for rendering a virtual environment, the method comprising: identifying one or more static elements in the virtual environment; determining, for a first frame having a first virtual camera position, a geometry of the static elements in the virtual environment; storing the geometry of the static element of the first frame; determining, for a second frame having the first virtual camera position, a geometry of at least a portion of the virtual environment based at least in part on the stored geometry of the static elements of the first frame; as well as For the second frame, lighting for the at least a portion of the virtual environment is determined based at least in part on the geometry of the at least a portion of the virtual environment determined for the second frame to render the at least a portion of the virtual environment.

2. The method according to claim 1, wherein: The method is for generating a light probe in a virtual environment, and wherein the at least a portion of the virtual environment is rendered to one or more faces of the light probe. 3 . The method of claim 1 , further comprising identifying one or more dynamic elements in the virtual environment.

4. The method according to claim 3, wherein: Determining the geometry of the at least a portion of the virtual environment of the second frame includes determining the geometry of the dynamic elements of the second frame, and combining the stored geometry of the static elements of the first frame and the determined geometry of the dynamic elements of the second frame.

5. The method according to claim 4, wherein: The stored geometry of the static elements and the determined geometry of the dynamic elements are combined according to the relative depths of the static elements and the dynamic elements from the first virtual camera position.

6. The method according to claim 3, wherein: The step of determining the geometry of the second frame is performed based on detecting movement of at least one of the dynamic elements between the first frame and the second frame.

7. The method of claim 6, further comprising determining a geometry of the at least a portion of the virtual environment of the first frame; wherein Upon detecting that the dynamic element has not moved between the first frame and the second frame, lighting for the second frame is determined based at least in part on the determined geometry of the at least a portion of the virtual environment of the first frame.

8. The method according to claim 7, wherein: The step of determining lighting for the second frame is performed in dependence on detecting a change in at least one light source in the virtual environment between the first frame and the second frame.

9. The method of claim 8, further comprising determining lighting for the at least a portion of the virtual environment for the first frame; wherein Upon detecting that the light source has not changed between the first frame and the second frame, determining lighting for the second frame includes reusing lighting determined for the first frame.

10. The method according to claim 1, wherein: Determining lighting for the at least a portion of the virtual environment for the second frame based on the determined geometry of the at least a portion of the virtual environment includes determining one or more output pixels for output to a display and performing a lighting operation on each output pixel.

11. The method according to claim 1 also includes determining a virtual camera position of the second frame, and when it is detected that the second frame has a different second virtual camera position: determining the geometry of the static elements in the virtual environment for the second frame, and storing the geometry of the static elements of the second frame.

12. The method of claim 11, further comprising removing the geometry of the static element of the first frame from a storage device.

13. The method according to claim 1, further comprising: identifying one or more fixed elements in the virtual environment, the fixed elements being fixed relative to a virtual camera and being independent of a position of the virtual camera; determining, for the first frame, a geometry of the fixed element in the virtual environment; storing the geometry of the fixed element of the first frame; as well as The geometry of the second frame is determined based at least in part on the stored geometry of the fixed elements of the first frame.

14. A computer program comprising computer executable instructions adapted to cause a computer system to perform the method of any preceding claim.

15. A system for rendering a virtual environment, the system comprising: an identification processor configured to identify one or more static elements in the virtual environment; a first geometry processor configured to determine, for a first frame having a first virtual camera position, a geometry of the static elements in the virtual environment; a storage processor configured to store the geometry of the static element of the first frame; a second geometry processor configured to determine, for a second frame having the first virtual camera position, a geometry of at least a portion of the virtual environment based at least in part on the stored geometry of the static elements of the first frame; as well as and a lighting processor configured to determine, for the second frame, lighting for the at least a portion of the virtual environment based at least in part on the geometry of the at least a portion of the virtual environment determined for the second frame to render the at least a portion of the virtual environment.

Images