Rendering method and system

Abstract

To provide a method, program, and system for rendering a virtual environment to provide more efficient rendering.SOLUTION: The present method includes: identifying one or more static elements in a virtual environment; determining, for a first frame having a first virtual camera position, a geometry of the static elements in the virtual environment; storing the geometry of the static elements for the first frame; determining, for a second frame having the first virtual camera position, a geometry of at least part of the virtual environment based at least in part on the stored geometry of the static elements for the first frame; and determining, for the second frame, lighting for the at least part of the virtual environment based at least in part on the geometry of the at least part of the virtual environment determined for the second frame, to render the at least part of the virtual environment.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a method and system for rendering a virtual environment. [Background technology]

[0002] Developers of content (e.g., video games) are constantly striving to provide more realistic virtual environments. The quality of lighting effects often plays a key role in how a user perceives such an environment and how real it appears to the user.

[0003] However, the process of rendering a virtual environment, and in particular the process of determining lighting, is often complex and computationally expensive. For example, while light probes can provide an effective way to obtain realistic lighting effects in a virtual environment, the process of generating or updating such light probes (which are typically provided in large numbers throughout the environment) can be prohibitively expensive and can require the use of computationally expensive, specialized hardware.

[0004] The present invention aims to alleviate or mitigate such problems and provide more efficient rendering. Summary of the Invention [Means for solving the problem]

[0005] Various aspects and features of the present invention are defined in the accompanying claims and detailed description, including at least the following:

[0006] In a first aspect, according to claim 1, a method for rendering a virtual environment is provided.

[0007] In another aspect, according to claim 15, there is provided a system for rendering a virtual environment. [Brief description of the drawings]

[0008] A more complete understanding of the present disclosure and many of the attendant advantages will be readily obtained as the same becomes better understood by reference to the following detailed description, when considered in connection with the accompanying drawings, in which: [Figure 1] FIG. 1 is a schematic diagram of an entertainment device; [Diagram 2] 1 is a flow chart that generally illustrates a method for rendering a virtual environment. [Diagram 3] 1 is a flow chart that generally illustrates a further method for rendering a virtual environment. [Figure 4] FIG. 2 is a schematic diagram of a light probe. [Diagram 5] FIG. 1 illustrates a schematic of a system for rendering a virtual environment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A method and system for rendering a virtual environment is disclosed. In the following description, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the present invention. However, it will be apparent to those skilled in the art that these specific details are not required to practice the present invention. Conversely, specific details known to those skilled in the art are omitted for the sake of clarity.

[0010] In an exemplary embodiment of the present invention, a suitable system and / or platform for implementing the methods and techniques herein may be an entertainment device such as a Sony® PlayStation5® video game console.

[0011] Referring now to the drawings, where like reference numerals indicate the same or corresponding parts, FIG. 1 shows an example of an entertainment device 10, which may be a computer or console, such as a Sony® PlayStation 5® (PS5).

[0012] The entertainment device 10 comprises a central processor 20. This may be a single-core processor or a multi-core processor, for example consisting of eight cores as in the PS5. The entertainment device also comprises a graphical processing unit (GPU) 30. The GPU may be physically separate from the CPU or may be integrated with the CPU as a system on chip (SoC) as in the PS5.

[0013] The entertainment device also includes RAM 40, which may be separate for each of the CPU and GPU, or a shared RAM as in the PS5, and may be physically separate or integrated as part of the SoC as in the PS5. Further storage is provided by disk 50, either as an external or internal hard drive, an external solid state drive, or an internal solid state drive as in the PS5.

[0014] The entertainment device may transmit and receive data via one or more data ports 60, as appropriate, such as a USB port, an Ethernet port, a Wi-Fi port, a Bluetooth port, etc. Optionally, the entertainment device may also receive data via an optical drive 70.

[0015] Audio / visual output from an entertainment device is typically provided through one or more A / V ports 90, or through one or more data ports 60, which may be wired or wireless.

[0016] One example of a device for displaying images output by an entertainment device is a head mounted display "HMD" 120, such as a PlayStation VR2 "PSVR2", worn by user 1. It will be appreciated that a variety of other devices may be used to display content, such as, for example, using a conventional television display connected to A / V port 90.

[0017] If the components are not integrated, they may be connected via dedicated data links or bus 100 as appropriate.

[0018] Interaction with the system is typically provided using one or more handheld controllers 130, 130A, such as a DualSense® controller 130 in the case of a PS5, or one or more VR controllers 130A-L,R in the case of an HMD. A user typically interacts with the system and the content displayed by the system, or the virtual environment rendered by the system, by providing input via the handheld controllers 130, 130A. For example, when playing a game, a user can navigate around the game virtual environment by providing input using the handheld controllers 130, 130A.

[0019] In an embodiment of the present disclosure, the entertainment device 10 generates one or more images of the virtual environment for display (eg, via a television or HMD 120).

[0020] Each of the multiple images generated by the entertainment device 10 is associated with a virtual camera position, in other words, the entertainment device 10 is configured to output an image for display depending on the position of the virtual camera.

[0021] It will be appreciated that the virtual camera has an associated viewpoint with respect to the virtual environment such that a portion of the virtual environment is included within the field of view of the virtual camera. This viewpoint is defined by a virtual camera position, which may include a location of the virtual camera within the virtual environment and an orientation of the virtual camera relative to the virtual environment. In some examples, the virtual camera position may further include a magnification (e.g., zoom) of the virtual camera.

[0022] The virtual camera may move relative to the virtual environment to follow a character in the virtual environment. For example, the virtual camera may follow a non-player character (NPC) in a video game, or may be controlled in response to user input to follow a player-controlled character. In some examples, the virtual camera may provide a first-person or third-person perspective of a given character in the game. Alternatively, the virtual camera may have a fixed location relative to the virtual environment and be controlled in response to input from a spectator (a non-playing user) to update the orientation of the virtual camera. Alternatively, the virtual camera may be controlled in response to input from a spectator such that a spectator can change the position of the virtual camera relative to the virtual environment without the spectator actively participating in the video game. Thus, the position (e.g., location and / or orientation) of the virtual camera may be updated, and the position of the virtual camera may optionally be controlled in response to user input.

[0023] In some examples, the position of the virtual camera can correspond to tracked movement of the HMD 120 worn by the user such that the viewpoint of the virtual camera is updated according to the movement of the user's head. The location and / or orientation of the HMD can be tracked using one or more image sensors and / or one or more inertial sensors, such as an accelerometer, a gyroscope and / or a magnetometer. For example, the location and orientation of the HMD can be tracked using known inside-out and / or outside-in tracking techniques that use image sensors attached to and / or that are attached to the HMD to capture images including the HMD. Thus, in some examples, the viewpoint of the virtual camera can be controlled by the user to move relative to the virtual environment based on the movement of the HMD.

[0024] An entertainment device or equivalent development kit, computer or server operating under appropriate software instructions can implement the methods for rendering virtual environments of the present disclosure.

[0025] The embodiments of the present disclosure relate to rendering a virtual environment in two stages, including a geometry stage where the geometry of the virtual environment is determined, and a lighting stage where the lighting of the virtual environment is then determined based on the geometry. The embodiments of the present disclosure allow for efficient reuse of rendering calculations between frames by determining the geometry of static elements in the virtual environment for a first frame, storing (e.g., caching) the results for future use, and then determining the geometry of the virtual environment in a second (e.g., next or other subsequent) frame having the same virtual camera position as the first frame based on the geometry of the static elements stored for the first frame. Because the virtual camera position does not change between frames, the static geometry of the previous frame can be effectively reused in determining the geometry of the subsequent frame. This approach can reduce the amount of calculations required to render the virtual environment over multiple frames, improving the efficiency of rendering the virtual environment. A virtual environment may be composed of many static elements, and in some cases, may be composed mostly of static elements (e.g., video games). Furthermore, virtual environments are typically re-rendered at a high frequency (i.e., frame rate). Thus, by reusing determined geometry of static elements across frames and improving the efficiency of rendering multiple frames, the present approach can provide significant computational cost savings. In some examples, by improving rendering efficiency, the present disclosure can enable an increase in the frame rate of the content, thus further improving the immersiveness of the content for the user.

[0026] As described in more detail below, the present approach is particularly applicable to generating and updating light probes. Light probes are often provided in large quantities in virtual environments and may require frequent updates (e.g., to reflect object movement and lighting changes in the virtual environment). Also, because light probes are stationary and viewpoint-fixed, the present approach can reuse stored stationary geometry each time a light probe is updated, reducing the computational cost of generating and updating the light probes.

[0027] The approach is also applicable to rendering any other environment where the same virtual camera position may be used for multiple frames. For example, in various video games, such as sports (e.g., tennis, soccer) games, the same virtual camera position may be used repeatedly over the course of the video game (e.g., a camera behind the court in tennis, a camera on the side of the pitch in soccer). Similarly, in various content, the virtual camera often remains stationary for a period of time. For example, in video games, the user often stops in the virtual environment for multiple frames to explore the surrounding environment in detail. In this way, by reusing geometry calculations between frames, the approach can reduce the computational cost of rendering such environments.

[0028] FIG. 2 illustrates an example method for rendering a virtual environment in accordance with one or more embodiments of the present disclosure.

[0029] Virtual environments for content such as video games typically include many individual elements (ie, graphical features).

[0030] It will be understood that the term "static element" as used herein relates to an element in a virtual environment that is static (i.e., does not move) relative to the origin or other reference point of the virtual environment. A static element may remain static during execution of a rendering method, such as during a game session or a portion of a game or other content (such as a level). Static elements may consist, for example, of the scenery of the virtual environment and / or other elements of the environment that do not move. Examples of static elements of a virtual environment include ground (such as pavement or roads), buildings, mountains, trees, or portions thereof.

[0031] In contrast, the term "dynamic element" will be understood to refer to an element in a virtual environment that may move relative to the virtual environment's origin or other reference point. Examples of dynamic elements in a virtual environment include a player character, a vehicle, or a non-player character (NPC).

[0032] Returning to FIG. 2, step 210 involves identifying one or more static elements within the virtual environment.

[0033] Static elements may be identified based on predefined properties associated with elements in the virtual environment. For example, each element in the virtual environment may be assigned a predefined flag, such as by a developer or other user, to indicate whether the element is static or dynamic. Static elements may then be identified based on the predefined flag for the element in the virtual environment.

[0034] Alternatively, or in addition, static elements may be identified by analyzing the movement of the element in the virtual environment across multiple frames of the content. For example, an element may be classified as a static element if it is determined that the element has not moved for a predefined number of frames. The analysis of the element's movement may be based, for example, on a velocity buffer of pixels for each frame. For example, an element with a velocity buffer of zero for a predefined number of frames may be considered static. The analyzed frames may be from the same session (e.g., a gaming session) of the rendered environment, and / or from one or more past sessions. In some examples, the element may be assigned a predefined flag based on the analysis of the movement, and in subsequent sessions, the flag may be used to identify the static element as described above.

[0035] It will be appreciated that elements in the virtual environment (whether static or dynamic) may vary in size and each may comprise one or more pixels.

[0036] It will also be appreciated that step 210 may include identifying all or only a subset of the static elements present in the virtual environment, the geometries of which are stored in step 230, as described in more detail below. Thus, depending on the number (and / or geometric complexity) of static elements identified, the amount of storage required may increase, and in some cases, it may be appropriate to identify (and store geometries for) only a subset of the static elements in the virtual environment in order to reduce memory usage.

[0037] Step 220 includes determining the geometry of static elements in the virtual environment for a first frame having a first virtual camera position.

[0038] Determining the geometry of the static element may include rendering the static element to determine geometry information of the static element. The geometry information may be determined for each fragment of the static element. The geometry information may include one or more of a position vector, a color vector, a normal vector, and / or a specular value.

[0039] The geometry information may include depth information for the static elements from the perspective of the first virtual camera position. For example, the geometry information may include depth information that corresponds to depth information typically stored in a Z-buffer. As described below, storing the depth information for the static elements allows for efficient reuse of the determined geometry of the static elements in subsequent frames.

[0040] In some examples, the geometry information may include information determined in a geometry stage of a deferred shading process for storage in a G-buffer. In some examples, determining the geometry of a static element may include determining data necessary for pixel shading calculations of the static element. For example, the static element may be rendered by rasterizing it into a texture map (also called a texture buffer).

[0041] The first frame may relate to any frame of the content, with the term "first" merely indicating a distinction from a subsequent "second" frame. In some cases, the first frame may relate to a frame that is not visible to any user, such as a loading frame, in other words, the geometry of static elements may be determined (e.g., baked) prior to displaying the content to a user or rendering the content for display to a user. For example, the geometry of static elements may be determined at level load or asset bake time for a video game.

[0042] It will be appreciated that in step 220, lighting of static elements is typically not calculated and only geometry information is collected. Typically, omitting the calculation of lighting, which is computationally expensive, may reduce the usage of computational resources. Furthermore, since only static elements are rendered in step 220 and dynamic elements are omitted, the lighting calculated only for static elements may not be accurate for the entire virtual environment.

[0043] Step 230 involves storing the geometry of the static elements in the first frame, in this manner the geometry of the static elements may be cached for future use in subsequent frames.

[0044] The geometry of the static elements may be stored in any suitable type and / or number of storage devices. For example, when the method is performed by entertainment device 10, the static element geometry may be stored in RAM 40 and / or disk 50. Alternatively, or in addition, the geometry of the static elements may be stored in one or more external devices, such as a cloud server.

[0045] In some examples, the type of storage device used to store static element geometry may be selected depending on one or more predetermined conditions, such as the frame rate of the content or the amount of memory required to store the geometry. For example, if the frame rate is high (and therefore requires faster lookup of the stored static element geometry) and / or a smaller amount of memory is required, a faster access storage device such as SRAM may be used, and if the frame rate is low or a larger amount of memory is required, a cheaper but slower storage device such as a disk may be used.

[0046] In some examples, the geometry of static elements may be initially stored in a buffer (sometimes called a G-buffer) for determining lighting for an initial frame, and then the geometry of static elements may be transferred from the buffer to storage for use in subsequent frames.

[0047] Step 240 includes determining a geometry of at least a portion of the virtual environment for a second frame having the first virtual camera position based at least in part on the stored geometry of the static elements for the first frame. This reuse of the stored geometry of the static elements for the first frame can improve the efficiency of determining the geometry for the second frame.

[0048] Determining a geometry of the virtual environment (or at least a portion thereof) for the second frame may include determining a geometry of dynamic elements in the virtual environment for the second frame and combining the stored geometry of static elements for the first frame with the determined geometry of the dynamic elements for the second frame. By determining only the geometry of the dynamic elements for the second frame and combining this with the stored geometry of the static elements for the first frame, the need to repeatedly redetermine the geometry of the entire environment may be eliminated, reducing the computational cost of determining geometry for multiple frames.

[0049] One or more dynamic elements in the virtual environment may be identified using the techniques described in step 210 with respect to identifying static elements, e.g., may be applied in reverse to identify dynamic elements. For example, dynamic elements may be identified based on predefined flags assigned to dynamic elements in the environment and / or by analyzing the movement of elements in the virtual environment over the course of multiple frames of content to identify elements that have moved at least a predetermined degree.

[0050] Dynamic elements and static elements may be considered complementary sets of elements in a virtual environment. In other words, elements that are not identified as static elements may be considered dynamic elements, and vice versa. Thus, in some examples, dynamic elements may be identified as all elements that were not identified as static in step 210. Similarly, it will be appreciated that static elements may be identified in step 210 based on a previous identification of dynamic elements, and static elements are elements that were not identified as dynamic.

[0051] The geometry of the dynamic elements for the second frame may then be determined using the same techniques as described in determining the geometry of the static elements in step 220 .

[0052] Once determined, the geometry of the dynamic elements for the second frame may be combined with the stored geometry of the static elements for the first frame. The geometries of the static and dynamic elements may be combined depending on the relative depths of the static and dynamic elements from the first virtual camera position. The relative depths of the elements may be determined based on depth information (such as a depth buffer) determined as part of determining the geometry of each of the elements. Based on the relative depths of the static and dynamic elements, portions of the elements that are occluded from the viewpoint of the virtual camera may be discarded to determine a combined geometry of the virtual environment.

[0053] Combining the geometry of the static and dynamic elements may involve creating a copy of the stored static geometry and writing the dynamic geometry on top of the copy of the static geometry depending on the relative depths of the static and dynamic elements, thus overwriting the copy with the determined geometry of the environment including both static and dynamic elements, while avoiding modifying the stored static geometry so that it can be used for further frames.

[0054] It will be appreciated that step 240 may include determining a geometry of some or all of the virtual environment for the second frame. Typically, the geometry of a portion of the virtual environment within the viewpoint of the virtual camera for the second frame is determined. In some examples, further portions of the geometry may be determined for use in future frames, e.g., portions of the environment outside the viewpoint of the virtual camera may be pre-rendered for quick access in future frames.

[0055] Step 250 includes determining lighting for at least a portion of the virtual environment based at least in part on the geometry of at least a portion of the virtual environment for the second frame determined in step 240 so as to render at least a portion of the virtual environment for the second frame.

[0056] In an embodiment of the present disclosure, the virtual environment (or a portion thereof) is rendered in two stages including a first "geometry" stage and a second "lighting" stage. Step 240 relates to the first geometry stage and step 250 relates to the second lighting stage.

[0057] Determining the lighting for the second frame based on the geometry of the virtual environment determined in step 240 may include determining one or more output pixels (i.e., pixels for output to a display) and performing a lighting operation for each output pixel. The lighting operations may each include a shading operation. The lighting operations may be performed by a pixel shader that determines the lighting (e.g., direct lighting and / or indirect lighting) for each output pixel based on the geometry of the determined environment.

[0058] By separating the rendering process into a geometry stage and a lighting stage, the efficiency of the rendering of the environment can be improved since the geometry determined in step 240 provides information related to the fragments provided as output pixels for display (e.g., determined based on the fragment's depth information), and thus lighting only needs to be determined once for each output pixel. Thus, the number of lighting operations can be reduced compared to forward rendering approaches where the geometry and lighting stages are not separated.

[0059] Of course, it will be appreciated that the techniques described herein can be extended to any number of frames, for example, the static element geometry of an initial frame can be reused to determine the geometry of multiple subsequent frames having the same virtual camera position, thus providing further efficiency improvements.

[0060] FIG. 3 illustrates a further exemplary method for rendering a virtual environment in accordance with one or more embodiments of the present disclosure.

[0061] Step 302 includes determining and storing the geometry of one or more static elements in the virtual environment for a first frame having a first virtual camera position. The static elements may be identified and their geometry determined using the techniques described above with reference to steps 210-230 of the method of FIG.

[0062] Step 304 includes determining a virtual camera position for the second frame and detecting whether the virtual camera position has changed between the first and second frames (i.e., whether the virtual camera has moved).

[0063] If, in step 304 , it is determined that the virtual camera position is the same in the first and second frames (ie, the second frame has an identical first virtual camera position), the method proceeds to step 306 .

[0064] In step 306, movement of one or more dynamic elements in the virtual environment is detected between the first and second frames. The dynamic elements and their movement may be detected using techniques described above with reference to the method of FIG. 2, for example, based on a velocity buffer of pixels associated with the elements of the virtual environment.

[0065] In some examples, step 306 may include detecting motion of dynamic elements within at least a portion of the environment to be rendered. For example, motion of dynamic elements within the field of view of the virtual camera for the second frame may be detected, and motion of dynamic elements outside the field of view may be ignored. As described in connection with step 314, if motion of dynamic elements is not detected in step 306, the geometry of the environment determined for the first frame may be reused to determine the lighting for the second frame. Detecting motion of only dynamic elements that are inside at least a portion of the environment (e.g., within the field of view of the virtual camera) allows for efficient reuse of geometry from the first frame, further reducing the computational cost of rendering the environment, since dynamic elements that are outside at least a portion of the environment do not affect the geometry of that portion of the environment for the second frame.

[0066] It will be appreciated that detecting movement of dynamic elements in step 306 may include detecting other changes to dynamic elements, such as the addition of new dynamic elements, the removal of previous dynamic elements, and / or changes to the shape of a dynamic element (e.g., damage to a car element in the virtual environment).

[0067] If, in step 306 , movement of at least one dynamic element between the first and second frames is detected, the method proceeds to step 308 .

[0068] In step 308, a geometry of at least a portion of the virtual environment is determined for the second frame based at least in part on the geometry of the static elements for the first frame determined and stored in step 302. The geometry of the environment for the second frame may be determined using the techniques described above with reference to step 240 of the method of FIG.

[0069] At step 310, lighting for at least a portion of the virtual environment is determined for a second frame based at least in part on the geometry for the second frame determined in step 308 to render at least a portion of the virtual environment. The lighting of the environment for the second frame may be determined using the techniques described above with reference to step 250 of the method of FIG.

[0070] It will therefore be appreciated that steps 302-310 of the method of Figure 3 may correspond to steps 210-250 of the method of Figure 2. Various further steps of the method of Figure 3 relate to further enhancements of the rendering method that allow for even greater efficiency in rendering the environment.

[0071] If, in step 306 , no movement of the dynamic element is detected between the first and second frames, the method proceeds to step 312 .

[0072] In step 312, changes to one or more light sources in the virtual environment between the first and second frames are detected. The changes to the light sources include one or more of adding one or more light sources (i.e., adding a light source to the second frame that was not present in the first frame), removing one or more light sources (i.e., removing a light source from the second frame that was present in the first frame), or changing one or more characteristics of the light sources, such as brightness, color, position within the virtual environment, etc. The changes to the light sources may be detected, for example, from an engine that renders the content (e.g., a game engine for video game content).

[0073] If step 306 includes detecting movement of only dynamic elements within at least a portion of the environment (e.g., within the field of view of the virtual camera for the second frame), then step 312 may include detecting both a change in the light source and movement of one or more other dynamic elements, such as dynamic elements outside at least a portion of the environment or outside the field of view of the virtual camera for the second frame. As described in connection with step 316, if no change in lighting is detected in step 312, the lighting of the environment determined for the first frame may be reused for the second frame. Because movement of a dynamic element may affect the lighting of the environment even if that element is outside at least a portion of the environment (e.g., if the dynamic element is reflective), detecting movement of one or more other dynamic elements outside at least a portion of the environment in step 312 helps ensure that lighting is not reused between frames if the lighting has actually changed.

[0074] In some examples, step 312 may include detecting changes to a subset of light sources and / or a subset of dynamic elements in the virtual environment. These subsets may be selected based on one or more predefined conditions. Of the subset of light sources and / or dynamic elements, changes that are likely to affect the lighting of the environment for the second frame may be selected based on these predefined conditions. Exemplary predefined conditions for selecting the subset of dynamic elements may include the distance between the dynamic elements and the virtual camera in the second frame (and / or the distance between the dynamic elements and the volume of the environment that falls within the viewpoint of the virtual camera), one or more optical properties of the dynamic elements, such as reflectance, and the shape and / or size of the dynamic elements. Exemplary predefined conditions for selecting the subset of light sources may include the distance between the light sources and the virtual camera in the second frame (and / or the distance between the light sources and the volume of the environment that falls within the viewpoint of the virtual camera), the brightness of the light source, the direction of the light source, the color of the light source, and the light cone of the light source. For example, a subset of light sources that are within a predefined distance from the virtual camera and facing towards the virtual camera may be selected, and a subset of dynamic elements that have a reflectance above a predefined threshold may be selected. In step 312, changes to only the light sources and / or dynamic elements in the selected subset may be tracked (i.e., detected), and only such changes may trigger a lighting recalculation in step 314. This may reduce the number of lighting recalculations that may be required, thus improving the efficiency of the rendering method.

[0075] It will be appreciated that the light source may be a dynamic element (such as a car light, the sun, etc.) or a static element (such as a street lamp, etc.).

[0076] If, in step 312 , a change in at least one light source between the first and second frames is detected, the method proceeds to step 314 .

[0077] In step 314, lighting for the second frame is determined based on at least a portion of the geometry of the virtual environment previously determined for the first frame. In other words, the geometry of the environment determined for the first frame is reused for the second frame, which may reduce the amount of computation required to render the second frame since the results of the geometry stage from the first frame may be reused and only the lighting stage is required to account for potential changes to the lighting caused by changes in the light source as detected in step 312. Lighting for the second frame may be determined based on the geometry of the first frame using the same techniques as described in relation to step 310.

[0078] If, in step 312 , no change in the light source is detected between the first and second frames, the method proceeds to step 316 .

[0079] In step 316, instead of determining new lighting for the second frame (e.g., as in steps 310 or 314), the lighting determined for the first frame is reused for the second frame. In effect, the first frame is reused for the second frame. In this way, the computationally expensive lighting determination is omitted entirely for the second frame, further improving the efficiency of the rendering method. Also, because it has already been determined that the dynamic elements are not moving (in step 306) and that the light sources are not changing (in step 312), any resulting degradation or inaccuracy in the rendering is also reduced (or at best avoided).

[0080] If, in step 304, it is determined that the virtual camera position has changed between the first and second frames (i.e., the second frame has a different second virtual camera position), the method proceeds to step 318. For example, the virtual camera position may change due to a user moving a player avatar or a developer repositioning a light probe.

[0081] Step 318 includes determining and storing geometry of static elements in the virtual environment for the second frame. In other words, the geometry of the static elements is recalculated for the second frame to account for the movement of the virtual camera, and the geometry of the static elements for the second frame may be reused for subsequent frames, e.g., in the same way that the geometry of the static elements for the first frame may be reused for the second frame, as described herein.

[0082] In some examples, step 318 may be performed dependent on one or more predefined conditions. The predefined conditions may provide a measure of how likely the second virtual camera position is to be used in a subsequent frame, such that static element geometry is calculated and stored only for frames for which the virtual camera position is likely to be used in further frames. The predefined conditions may relate, for example, to the percentage of static elements in the second frame, the frequency of the second virtual camera position in previous frames, or a user-defined flag. For example, static element geometry may be determined for the second frame only if the second virtual camera position has been used in a predefined number of previous frames, is composed of at least a predefined number of static elements, and / or has been flagged by a user (e.g., a game developer) as likely to be reusable.

[0083] The static element geometry for the first frame may be kept in storage regardless of whether the virtual camera has moved, and future frames may have the first virtual camera position and therefore will benefit from the stored static element geometry for the first frame.

[0084] Alternatively, the method may proceed after step 318 to step 320, where the geometry of static elements saved for the first frame is deleted from storage to reduce memory usage. Since the geometry information of static elements in a frame may require a large amount of memory, flushing the geometry cache of static elements in this manner can prevent memory usage from becoming too high.

[0085] In some examples, the saved static element geometry for the first frame may be deleted from storage in response to one or more predefined conditions. The predefined conditions may provide a measure of how likely the first virtual camera position is to be used in a subsequent frame. The predefined conditions may relate, for example, to the recency of the first frame (in number of frames, or in real time or in-content (e.g., in-game) time), the frequency of the first virtual camera position in previous frames, the game associated with the virtual environment, or a flag assigned by the user (e.g., a flag indicating a frequently reused virtual camera position). For example, the static element geometry for the first frame may be deleted from storage if the first frame is an old frame or if the first frame (or its virtual camera position) has not been flagged as frequently reusable. Alternatively or additionally, the static element geometry for the first frame may be deleted from storage if the first virtual camera position has been used less than a predefined number of times in previous frames.

[0086] It will be appreciated that the rendering method of FIG. 3 can further reduce the computational cost of rendering an environment by efficiently reusing geometry and / or lighting calculations between frames.

[0087] It will also be appreciated that various steps of the method of FIG. 3 may be optional. For example, in some examples, step 304 (i.e., determining whether the virtual camera has moved between the first and second frames) and associated steps 318 and 320 may be omitted, and the method may proceed directly from step 302 to step 306 under the assumption that the virtual camera has not moved (e.g., as may be appropriate for a light probe). Alternatively, or in addition, in some examples, step 306 (i.e., determining whether the dynamic object has moved between the first and second frames) and associated steps 312-316 may be omitted, and the method may proceed directly from step 304 to steps 308 and 310 under the assumption that the dynamic object has moved. Alternatively, or in addition, in some examples, step 312 (i.e., determining whether the light source has changed between the first and second frames) and associated step 316 may be omitted, and the method may proceed directly from step 306 to step 314 under the assumption that the light source has changed.

[0088] It will also be appreciated that the various steps of the methods of Figures 2 and 3 may be performed in any suitable combination.

[0089] The rendering methods of Figures 2 and / or 3 may include various further steps, which may include one or more of the following: Selecting a frame as the first frame in response to one or more predetermined conditions. Identifying and reusing the geometry of elements in the virtual environment that are fixed relative to the virtual camera (also called "fixed elements").

[0090] Considering the selection of the first frame, one frame may be selected as the first frame among the plurality of frames of the content depending on one or more predetermined conditions. The predetermined conditions may be similar to those described with reference to determining whether to calculate the geometry of the static elements for the second frame in step 318. For example, the first frame may be selected depending on the proportion of static elements in each frame and / or the distribution of the virtual camera positions among the plurality of frames. The first frame may be selected, for example, as the frame in which the virtual camera position is most frequently used among the plurality of frames. The proportion of static elements in the frame may also be taken into account when selecting the first frame, for example, selecting a frame with a high proportion of static elements (thereby allowing a higher reuse of geometry calculations between frames). Alternatively, a frame with a low proportion of static elements may be prioritized to reduce the amount of memory required to store the determined geometry of the static elements.

[0091] In some examples, the virtual camera positions may be tracked (e.g., using a counter) across multiple frames to determine how often each virtual camera position is repeated. For example, the geometry of static elements may be determined and stored for frames in which the corresponding virtual camera position has been repeated at least a predetermined number of times. This may allow the geometry of static elements to be cached for multiple frequently reused virtual camera positions, further reducing computational costs when rendering multiple frames. In some examples, the selection of the first frame may be performed in response to an analysis of user behavior and virtual camera positions in multiple previous sessions of the content, such as to identify the most frequently used virtual camera positions.

[0092] Considering fixed elements, the term "fixed element" may relate to an element that is fixed (i.e., has substantially the same position) relative to the virtual camera regardless of the position of the virtual camera. A fixed element may be static, but is typically dynamic. In other words, a fixed element typically moves relative to the origin of the virtual environment, but is fixed relative to the virtual camera. Examples of such fixed elements include an avatar's hand when the viewpoint corresponds to a first-person viewpoint, a part of an avatar when the viewpoint corresponds to a third-person viewpoint, a part of a helmet when the viewpoint corresponds to a first-person viewpoint, or one or more parts of a user's car in a driving game. For example, elements such as a helmet worn by an in-game character and a weapon held by an in-game character may be included in an image frame output for display such that the position of the helmet or weapon in the image frame does not change even if the position and / or orientation of the viewpoint of the virtual camera changes. For example, in some first-person shooter games, including a virtual helmet or other similar structure in the image frame can give the viewer the impression of viewing the scene as if the helmet were worn, so that as the virtual camera moves relative to the virtual environment, the position and appearance of the helmet remains unchanged for a series of image frames generated for the virtual camera.

[0093] In some examples, the rendering method of Figures 2 and / or 3 may include identifying one or more fixed elements within the virtual environment, determining and storing a geometry of the fixed elements within the virtual environment for a first frame, and determining a geometry for a second frame based at least in part on the geometry of the fixed elements stored for the first frame.

[0094] Fixed elements in the virtual environment may be identified, for example, based on a preselection by a user (e.g., a game developer may mark fixed elements in the environment) or based on changes in content between frames (e.g., pixels that appear in the same location for a predefined number of frames may be determined as fixed). The geometry of the fixed elements for the first frame may then be determined and stored using techniques similar to those described with reference to steps 220, 230 of the method of FIG. 2. The geometry of the fixed elements stored for the first frame may then be used to determine the geometry for the second frame using techniques similar to those described with reference to step 240 of the method of FIG. 2. For example, the geometry of the fixed elements for the first frame may be combined with the geometry of the dynamic elements for the second frame and / or the geometry of the static elements for the first frame to more efficiently determine the environment geometry for the second frame. Reusing the geometry of the fixed elements in this manner may further reduce the computational cost of the rendering process, since fixed elements are typically located in close proximity to the virtual camera and occlude other elements.

[0095] In some examples, the fixed elements are static for all virtual camera positions, so that the geometry of the fixed elements for a first frame can be used to determine the geometry for a second frame, even if the second frame has a different virtual camera position than the first frame.

[0096] As mentioned above, this disclosure is particularly applicable to generating and updating Light Probes.

[0097] It will be appreciated that a typical virtual environment, whether for use in content such as video games or as a television or movie set, may be configured with ambient lighting (e.g., generated by a so-called skybox) and / or directional lighting (typically from light sources within the environment).

[0098] Additionally, there is Global Illumination ("GI") lighting, which takes into account secondary light sources such as bounce and reflection light, as well as light emitted from textured objects. Similarly, reflection light can be directed towards other surfaces and the virtual camera.

[0099] Such secondary light calculations can be performed using Global Illumination Probes ("GI Probes"), and optionally Reflection Probes, sometimes collectively referred to as Light Probes.

[0100] Light probes provide a way to obtain information about light passing through empty spaces (e.g., spaces between buildings) in a virtual environment. In other words, a light probe can be thought of as a texture that captures lighting information about the surrounding virtual environment. This information may be stored in the light probe to illuminate objects in the environment during execution of the content (e.g., a video game). For example, multiple light probes may be placed throughout the virtual environment and the light probes may be used to provide lighting effects to moving objects (e.g., a car between buildings) in the virtual environment. For example, the lighting effect on the moving object may be approximated by interpolating between light probes adjacent to the object.

[0101] Lighting information for Light Probes is captured (e.g. baked) once before runtime and used throughout the content's runtime. Such Light Probes are sometimes called Static Light Probes or Baked Light Probes.

[0102] Alternatively, the lighting information of a Light Probe may be updated to react in real-time to changes in the surrounding virtual environment. Such Light Probes are sometimes called Dynamic Light Probes, Real-time Light Probes, or Non-Baked Light Probes. Dynamic Light Probes can provide more realistic lighting effects because changes in the environment are incorporated into the information stored in the Light Probe. However, when the virtual environment changes due to object movement or changing lighting conditions (such as a change in time of day), the Dynamic Light Probes must be updated, which is computationally expensive. Thus, the process of generating and updating Dynamic Light Probes can be computationally expensive.

[0103] FIG. 4 illustrates a schematic of a non-limiting example of a light probe 400. The light probe 400 includes faces 402 (e.g., faces 402-a, 402-b, 402-c) and a center 404. The faces 402 represent the environment as seen from the center 404 of the probe. Each face 402 is divided into a grid of pixels (not shown). The faces 402 of the probe may represent a texture. Thus, a "pixel" on a face may be thought of as a "texel." Additionally, in the example illustrated in FIG. 4, the light probe 400 is in the shape of a cube having six faces 402, although it will be appreciated that the light probe may have other shapes, such as a spherical shape.

[0104] Generating (or updating) a light probe may include environment mapping using the faces of the light probe as map geometry. Environment mapping may include projecting the virtual environment onto the faces of the light probe and saving the resulting texture, in other words, rendering the virtual environment onto the faces of the light probe. In the example of a cubic light probe 400, environment mapping may include cubic mapping, where the environment is projected onto the six faces 402 of the light probe 400. Thus, a cubic map of the environment may be generated by rendering the virtual environment from the perspective of the center 404 of the light probe for each light probe face 402. Thus, for a cubic light probe 400, six virtual environment rendering operations may be required to generate the light probe 400 or to update the light probe 400 (e.g., to account for the movement of objects in the environment).

[0105] A virtual environment is typically provided with a large number of light probes, and for each light probe, a separate rendering operation may be required for each face of the probe. It will therefore be appreciated that the process of generating and updating dynamic light probes in an environment may require a large number of rendering operations and associated high computational costs.

[0106] Our approach can reduce this computational cost and improve the efficiency of dynamic light probe updates. For a given light probe, the virtual camera position for rendering the environment to the light probe is defined by the center and face of the light probe. Thus, the virtual camera position does not move between frames, and the geometry of static elements can be reused in each subsequent frame (assuming the light probe itself is not moving). Furthermore, because the environment is rendered to the face of the light probe from each viewing direction, the environment rendered to one or more faces of the probe may consist mostly of static elements (e.g., at least the face of the probe that faces the ground is typically composed mostly of static elements). Thus, caching and reusing the geometry of static elements can achieve significant efficiency improvements in light probe updates, since it is not necessary to repeatedly determine the environment geometry from scratch every time the light probe is updated.

[0107] Of course, the techniques described herein can be extended to generating and updating multiple Light Probes in one or more virtual environments.

[0108] Returning to FIG. 2, in an overview of an embodiment of the present invention, a method for rendering a virtual environment includes the following steps.

[0109] Step 210 involves identifying one or more static elements within the virtual environment, as described elsewhere herein.

[0110] Step 220 involves determining the geometry of static elements in the virtual environment for a first frame having a first virtual camera position, as described elsewhere herein.

[0111] Step 230 involves storing the geometry of the static elements for the first frame, as described elsewhere herein.

[0112] Step 240 includes determining, for a second frame having a first virtual camera position, a geometry of at least a portion of the virtual environment based at least in part on the geometry of static elements stored for the first frame, as described elsewhere herein.

[0113] Step 250 includes determining lighting of at least a portion of the virtual environment for the second frame based at least in part on the geometry of at least a portion of the virtual environment determined for the second frame to render at least a portion of the virtual environment, as described elsewhere herein.

[0114] It will be apparent to one of ordinary skill in the art that variations of the above methods that correspond to the operation of various embodiments of the methods and / or apparatus as described and claimed herein are considered within the scope of the present disclosure, including, but not limited to, those described herein. The method is for generating a Light Probe in a virtual environment, where at least a portion of the virtual environment is rendered relative to one or more faces of the Light Probe, as described elsewhere herein. The method further includes identifying one or more dynamic elements in the virtual environment, as described elsewhere herein. In this case, optionally, determining 240 a geometry of at least a portion of the virtual environment for the second frame includes determining a geometry of dynamic elements for the second frame, as described elsewhere herein, and combining the stored geometry of static elements for the first frame with the determined geometry of dynamic elements for the second frame. Here, optionally, combining the stored geometry of the static elements with the determined geometry of the dynamic elements depends on the relative depths of the static and dynamic elements from the first virtual camera position, as described elsewhere in this specification. In this case, optionally, step 240 of determining the geometry for the second frame is performed in response to detection of movement of at least one of the dynamic elements between the first and second frames, as described elsewhere herein. Here, optionally, the method further includes determining a geometry of at least a portion of the virtual environment for the first frame, and wherein upon detecting that the dynamic element has not moved between the first frame and the second frame, determining 250 lighting for the second frame is based at least in part on the determined geometry of at least a portion of the virtual environment for the first frame, as described elsewhere herein. Here, optionally, step 250 of determining the lighting for the second frame is performed in response to detecting a change to at least one light source in the virtual environment between the first and second frames, as described elsewhere in this specification. Here, optionally, the method further includes determining lighting of at least a portion of the virtual environment for the first frame, where upon detecting that the light source has not changed between the first and second frames, determining 250 the lighting for the second frame includes reusing the lighting determined for the first frame, as described elsewhere herein. Determining 250 lighting of at least a portion of the virtual environment for a second frame based on the determined geometry of at least a portion of the virtual environment includes determining one or more output pixels for output to the display and performing a lighting operation for each output pixel, as described elsewhere herein. The method further includes determining a virtual camera position for the second frame, as described elsewhere herein, and upon detecting that the second frame has a second, different virtual camera position, determining a geometry of static elements in the virtual environment for the second frame, and storing the geometry of the static elements for the second frame. In this case, optionally, determining and storing the geometry of the static elements for the second frame is performed in response to one or more predefined conditions, the predefined conditions relating to one or more selected from the list of: the proportion of static elements for the second frame, the frequency of the second virtual camera position among previous frames, or a flag assigned by a user to the second frame and / or the second virtual camera position, as described elsewhere in this specification. The method further includes deleting the geometry of the static elements for the first frame from storage, as described elsewhere herein. In this case, optionally, deleting the geometry of the static elements for the first frame from the storage is performed in response to one or more predetermined conditions, the predetermined conditions being related to one or more selected from the list of: the freshness of the first frame, a game associated with the virtual environment, a frequency of the first virtual camera position among previous frames, or a flag assigned by a user to the first frame and / or the first virtual camera position, as described elsewhere herein. The method further includes identifying one or more fixed elements in the virtual environment, as described elsewhere in this specification, where the fixed elements are fixed relative to the virtual camera regardless of the position of the virtual camera, determining a geometry of the fixed elements in the virtual environment for a first frame, storing the geometry of the fixed elements for the first frame, and determining a geometry for a second frame based at least in part on the stored geometry of the fixed elements for the first frame. In this case, optionally, upon detecting that the second frame has a second, different virtual camera position, determining a geometry for the second frame is based at least in part on the stored geometry of the fixed elements for the first frame instead of the stored geometry of the static elements for the first frame, as described elsewhere herein. The method further includes selecting one frame of the plurality of frames as the first frame depending on one or more predetermined conditions, as described elsewhere herein. In this case, optionally, the one or more predetermined conditions include one or more selected from the list consisting of: the proportion of static elements in the plurality of frames, or the distribution of virtual camera positions in the plurality of frames, as described elsewhere in this specification.

[0115] It will be appreciated that the above methods may be implemented on suitably adapted conventional hardware, either by means of software instructions, or by the inclusion or substitution of dedicated hardware.

[0116] Thus, the necessary adaptation to existing portions of the conventional equivalent device may be implemented in the form of a computer program product including processor-implementable instructions stored on a non-transitory machine-readable medium, such as a floppy disk, optical disk, hard disk, solid state disk, PROM, RAM, flash memory, or any combination of these or other storage media, or may be realized in hardware as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array) or other configurable circuitry suitable for use in adapting the conventional equivalent device. Alternatively, such a computer program may be transmitted via data signals over a network, such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks.

[0117] Referring to FIG. 5, in an overview of an embodiment of the present invention, a system for rendering a virtual environment may be configured as follows.

[0118] An identification processor 510 configured (eg, by suitable software instructions) to identify one or more static elements within the virtual environment, as described elsewhere herein.

[0119] A first geometry processor 520 configured (e.g., by suitable software instructions) to determine the geometry of static elements in the virtual environment for a first frame having a first virtual camera position, as described elsewhere in this specification.

[0120] A storage processor 530 configured (eg, by suitable software instructions) to store the geometry of the static elements for the first frame, as described elsewhere herein.

[0121] A second geometry processor 540 configured (e.g., by suitable software instructions) to determine, for a second frame having a first virtual camera position, a geometry of at least a portion of the virtual environment based at least in part on the stored geometry of static elements for the first frame, as described elsewhere in this specification.

[0122] A lighting processor 550 configured (e.g., by suitable software instructions) to determine, for the second frame, lighting for at least a portion of the virtual environment based at least in part on the geometry of at least a portion of the virtual environment determined for the second frame, and to render at least a portion of the virtual environment, as described elsewhere herein.

[0123] These processors 510-550 may be implemented, for example, by CPU 20 of entertainment system 10. Of course, it will be appreciated that the functionality of these processors may be suitably realized by any suitable number of processors located on any suitable number of devices, and any suitable number of devices, rather than requiring a one-to-one mapping between functionality and devices or processors.

[0124] The foregoing discussion discloses and describes merely exemplary embodiments of the present invention. As will be understood by those skilled in the art, the present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. Accordingly, the disclosure of the present invention, as well as the other claims, are intended to be illustrative and not limiting of the scope of the present invention. The present disclosure, including readily identifiable variations of the teachings herein, defines in part the scope of the preceding claim terms, so that the inventive subject matter is not dedicated to the public.

Claims

1. 1. A method of rendering a virtual environment, comprising: Identifying one or more static elements within the virtual environment; determining a geometry of the static elements in the virtual environment for a first frame having a first virtual camera position; storing the geometry of the static elements for the first frame; determining a geometry of at least a portion of the virtual environment for a second frame having the first virtual camera position based at least in part on the stored geometry of the static elements for the first frame; determining, for the second frame, lighting for at least a portion of the virtual environment based at least in part on a geometry of the at least a portion of the virtual environment determined for the second frame to render the at least a portion of the virtual environment; The method includes:

2. The method of claim 1 , wherein the method is for generating a light probe in a virtual environment, at least a portion of the virtual environment being rendered relative to one or more faces of the light probe.

3. The method of claim 1 or 2, further comprising identifying one or more dynamic elements within the virtual environment.

4. 4. The method of claim 3, wherein determining a geometry of at least a portion of the virtual environment for the second frame includes determining a geometry of dynamic elements for the second frame and combining the stored geometry of static elements for the first frame with the determined geometry of dynamic elements for the second frame.

5. The method of claim 4 , wherein combining the stored geometries of the static elements and the determined geometries of the dynamic elements depends on relative depths of the static elements and the dynamic elements from the first virtual camera position.

6. 6. The method of claim 3, wherein determining the geometry for the second frame is performed in response to detecting a movement of at least one of the dynamic elements between the first frame and the second frame.

7. 7. The method of claim 6, further comprising determining a geometry of at least a portion of the virtual environment for the first frame, and wherein upon detecting that the dynamic element has not moved between the first frame and the second frame, determining lighting for the second frame is based at least in part on the determined geometry of at least a portion of the virtual environment for the first frame.

8. 8. The method of claim 7, wherein determining the lighting for the second frame is performed in response to detecting a change to at least one light source in the virtual environment between the first frame and the second frame.

9. 10. The method of claim 8, further comprising determining lighting of at least a portion of the virtual environment for the first frame, and upon detecting that the light source has not changed between the first frame and the second frame, determining lighting for the second frame comprises reusing the lighting determined for the first frame.

10. 10. The method of claim 1, wherein determining lighting of at least a portion of the virtual environment for the second frame based on a determined geometry of at least a portion of the virtual environment comprises determining one or more output pixels for output to a display and performing a lighting operation for each output pixel.

11. 11. The method of claim 1, further comprising: determining a virtual camera position for the second frame; and upon detecting that the second frame has a second, different virtual camera position, determining a geometry of static elements in the virtual environment for the second frame; and storing the geometry of the static elements for the second frame.

12. The method of claim 11 , further comprising deleting the static element geometry for the first frame from storage.

13. identifying one or more fixed elements within the virtual environment, the fixed elements being fixed relative to the virtual camera independent of a position of the virtual camera; determining a geometry of the fixed elements in the virtual environment for the first frame; storing the geometry of the fixation elements for the first frame; determining a geometry for the second frame based at least in part on the stored geometry of the fixed elements for the first frame; 13. The method of claim 1, further comprising:

14. A computer program comprising computer executable instructions adapted to cause a computer system to carry out a method according to any one of claims 1 to 13.

15. 1. A system for rendering a virtual environment, comprising: an identification processor configured to identify one or more static elements within the virtual environment; a first geometry processor configured to determine a geometry of the static elements in the virtual environment for a first frame having a first virtual camera position; a storage processor configured to store the geometry of the static elements for the first frame; a second geometry processor configured to determine a geometry of at least a portion of the virtual environment for a second frame having the first virtual camera position based at least in part on the stored geometry of the static elements for the first frame; a lighting processor configured to determine, for the second frame, lighting for at least a portion of the virtual environment based at least in part on a geometry of the at least a portion of the virtual environment determined for the second frame to render the at least a portion of the virtual environment; A system comprising:

Images