Method and system for optical probe generation

Abstract

A method of generating an optical probe in a virtual environment is provided. The method includes determining a first region of the light probe based on one or more predetermined conditions; collecting light information from the virtual environment, wherein the light information relates to light passing through the location of the light probe in the virtual environment; and storing the optical information in the optical probe, wherein the optical information is stored in the first region of the optical probe at a higher resolution than in a different second region of the optical probe.

Description

Technical Field

[0001] The present invention relates to a method and system for generating light probes in a virtual environment. Background Art

[0002] Developers of content (such as video games) are constantly striving to provide more realistic virtual environments. The quality of lighting effects often plays a key role in the user's perception of such environments and how realistic they appear to the user.

[0003] Light Probes can provide an efficient way to achieve realistic lighting effects in virtual environments. However, the process of generating Light Probes is often complex and computationally expensive. This computational cost increases further as the quality of the Light Probes increases. Virtual environments often include a large number of Light Probes distributed throughout the environment - therefore, the computational cost of generating high-quality Light Probes for an environment can be prohibitive and may require the use of computationally expensive and specialized hardware.

[0004] The present invention seeks to alleviate or mitigate these problems and provide for more efficient generation of optical probes. Summary of the invention

[0005] Various aspects and features of the present invention are defined within the text of the accompanying description, and include at least:

[0006] - In a first aspect, there is provided a method of generating a light probe in a virtual environment according to embodiments herein.

[0007] - In another aspect, according to embodiments herein there is provided a system for generating light probes in a virtual environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] A more complete understanding of the present disclosure and many of its attendant advantages will be readily obtained, while becoming better appreciated, by referring to the following detailed description when considered in conjunction with the accompanying drawings, in which:

[0009] Figure 1 The entertainment equipment is schematically shown;

[0010] Figure 2 is a schematic flow chart illustrating a method of generating an optical probe;

[0011] Figure 3 schematically illustrates an optical probe; and

[0012] Figure 4 A system for generating optical probes is schematically shown. DETAILED DESCRIPTION

[0013] A method and system for generating light probes in a virtual environment are disclosed. In the following description, many specific details are presented in order to provide a thorough understanding of the embodiments of the present invention. However, it is apparent to those skilled in the art that these specific details need not be employed to practice the present invention. Instead, for the sake of clarity, specific details known to those skilled in the art have been omitted where appropriate.

[0014] In an example embodiment of the invention, a suitable system and / or platform for implementing the methods and techniques herein may be an entertainment device (such as video game consoles).

[0015] Referring now to the drawings, wherein like reference numerals designate like or corresponding parts, Figure 1 An example of an entertainment device 10 is shown, which is a computer or a computer such as (PS5) console.

[0016] The entertainment device 10 includes a central processor 20. This can be a single core or multi-core processor, for example, including eight cores as in the PS5. The entertainment device also includes a graphics processing unit or GPU 30. The GPU can be physically separate from the CPU, or integrated with the CPU as a system on a chip (SoC), as in the PS5.

[0017] The entertainment device also includes RAM 40, and may have separate RAM for each of the CPU and GPU, or shared RAM as in the PS5. The or each RAM may be physically separate, or integrated as part of the SoC (as in the PS5). Further storage is provided by a disk 50, as an external or internal hard drive, or as an external solid state drive, or an internal solid state drive (as in the PS5).

[0018] The entertainment device may optionally be connected via one or more data ports 60 (such as a USB port, port, port, It can also optionally receive data via an optical drive 70.

[0019] Audio / video output from the entertainment device is typically provided through one or more A / V ports 90 or through one or more of the wired or wireless data ports 60 .

[0020] An example of a device for displaying images output by an entertainment device is a head mounted display "HMD" 120, such as a PlayStation VR 2 "PSVR2", worn by user 1. It should be understood that various other devices may be used to display content (e.g., using a conventional television display connected to A / V port 90).

[0021] Where components are not integrated, they may be connected by dedicated data links or via bus 100 as appropriate.

[0022] Typically one or more handheld controllers 130, 130A (such as in the case of the PS5 The controller 130 and / or one or more VR controllers 130A-L, R in the case of an HMD provide interaction with the system. The user typically interacts with the system and any content displayed by the system or the virtual environment rendered by the system by providing input via the handheld controller 130, 130A. For example, when playing a game, the user can navigate in the game virtual environment by providing input using the handheld controller 130, 130A.

[0023] Operating under appropriate software instructions, the entertainment device or equivalent development kit, computer or server may then implement the method of generating light probes in a virtual environment as described herein.

[0024] It will be appreciated that a typical virtual environment, such as may be used in content such as video games or as a TV or movie set, may include ambient light (eg generated by a so-called skybox) and / or directional light (typically from light sources within the environment).

[0025] However, another form of lighting comes from global illumination ("GI"), which takes into account secondary light sources, such as bounced or reflected light, or optionally emitted light from textured objects. Similarly, reflections can redirect light toward other surfaces and toward the virtual camera.

[0026] Calculation of this secondary light may be performed using global illumination probes ("GI probes"), and optionally also reflection probes, which may be more generally referred to as light probes.

[0027] Lighting interactions are often with static components within a scene, but not necessarily.

[0028] In a typical example, a red cube on a white surface will result in a pink / red tint to the white surfaces surrounding the cube due to GI. At the same time, depending on the surface properties of the cube, the surface, or another object, sharp or diffuse reflections may be seen on the surface of one or more of them.

[0029] Light Probes provide a way to capture information about light traveling through empty spaces in a virtual environment (e.g., in the spaces between buildings). This information can be stored in Light Probes for use at runtime in rendering content (e.g., a video game). For example, several Light Probes can be placed throughout a virtual environment, and the Light Probes can be used to provide lighting effects on moving objects in the virtual environment (e.g., cars between buildings)—for example, the lighting effects on a moving object can be approximated by interpolating between Light Probes adjacent to the object.

[0030] As mentioned above, the process of generating light probes and calculating global illumination and optionally reflections can be computationally intensive. This computational cost can be reduced by reducing the light probe quality - however, lighting effects often play a key role in providing a realistic virtual environment to the user, and this reduction in quality is often noticeable to the user and reduces the immersion of the content.

[0031] Embodiments of the present disclosure seek to address or mitigate this problem and provide a method for more efficiently generating light probes without sacrificing the quality of the lighting effects.

[0032] One or more embodiments of the present disclosure relate to methods and systems for generating a light probe in a virtual environment. This includes determining a first region of the light probe (e.g., a region of a face of the probe) based on (one or more) predetermined conditions (e.g., based on the brightness of objects facing different regions of the probe, where the first region can be selected to be a region facing a bright object (such as the sun)), collecting light information related to light passing through the position of the light probe in the virtual environment, and storing the light information in the light probe, wherein the light information is stored in the first region of the light probe at a higher resolution (e.g., x2 or x4 higher) than in a different second region of the light probe (e.g., the remainder of the face of the probe including the first region).

[0033] In this way, the area(s) of Light Probes that are most important (i.e., salient) for lighting objects in the virtual environment at runtime can be determined, and the light information for these area(s) can be stored at a higher resolution. This allows for an improved balance between the computational cost of generating (i.e., constructing) Light Probes and the quality of the lighting effects provided by the Light Probes at runtime - in other words, achieving good quality lighting effects more efficiently. Figure 2 An example of a method of generating a light probe in a virtual environment according to one or more embodiments of the present disclosure is shown.

[0034] Step 210 includes determining a first region of the light probe based on one or more predetermined conditions.

[0035] For the purpose of explanation, Figure 3Schematically illustrated is a non-limiting example of a light probe 300. The light probe 300 includes faces 302 (e.g., faces 302-a, 302-b, 302-c) and a center 304. The faces 302 represent the environment as viewed from the center 304 of the probe. Each face 302 is divided into a pixel grid (not shown).

[0036] Each pixel on the face of the Light Probe (i.e., each pixel of the Light Probe) can be considered an "outgoing" viewing direction from the center 304 of the Probe. In other words, each pixel of the Light Probe can be considered a separate light source used to render content at runtime. During rendering of content, shading (e.g., pixel shading) of the virtual environment can include sampling from the face 302 of the Light Probe 300 to determine the GI that affects objects in the virtual environment near the Light Probe 300, thereby allowing the lighting effects of objects in the environment to be determined.

[0037] It will be appreciated that the face 302 of the probe may represent a texture. Therefore, unless otherwise stated, a "pixel" on a face may be considered a "texel", and the two expressions may be used interchangeably below.

[0038] It will also be appreciated that the term "resolution" as used herein preferably relates to the number of probe pixels / texels per unit surface area of ​​the face of the probe. The resolution in a given area of ​​an optical probe (e.g., over the face of the probe) may be the same in all (e.g., vertical and horizontal) directions, or different in different directions (e.g., it may be greater in the vertical direction than in the horizontal direction).

[0039] exist Figure 3 In the example shown, the optical probe 300 is shaped as a cube having six faces 302, - however, it will be appreciated that the optical probe may have other shapes, such as a sphere.

[0040] At step 210, the light probe 300 is divided into two or more regions, including at least a first region (whose light information is stored at a higher resolution) and a second region (whose light information is stored at a lower resolution). The first region and the second region each include one or more probe pixels. The first region and the second region may include different regions of the face of the light probe—for example, the first region may include regions 306-a and / or 306-b, and the second region may include the remaining portion of the face 302-a and / or 302-b. Alternatively, the first region and the second region may include different faces of the light probe—for example, the first region may include face 302-a, and the second region may include face 302-b and / or 302-c.

[0041] The first region of the Light Probes (i.e., how the Probes are divided into regions) is determined based on one or more predetermined conditions. The predetermined condition(s) may be selected to identify the region(s) of the Light Probes that are most important (i.e., salient) for lighting objects in the virtual environment during runtime rendering of the content. The predetermined condition(s) are dynamically evaluated when the Light Probes are generated - this provides an automatic way to identify the first (i.e., salient) regions of the Light Probes without requiring the developer to manually select these regions in advance.

[0042] As described above, pixels on the face of a Light Probe represent an "outgoing" viewing direction from the center of the Light Probe. The importance of light passing through the location of the Probe (for later lighting objects in the content) can vary between the directions of the light—for example, the incident light may come from a particular direction, while other directions are less relevant for lighting. Therefore, as described in further detail with respect to step 230, storing light information at a higher resolution in certain areas of the probe allows the corresponding directions to be prioritized when generating the Light Probe. In other words, the present method allows the use of dynamically determined important directions as a criterion for locally increasing the resolution of the Light Probe. This can allow higher resolution lighting information to be stored for directions that a user is more likely to notice. Storing light information in this manner can allow more efficient use of computing resources to obtain high-quality lighting at runtime (e.g., compared to assigning different resolutions to different probes without considering the direction).

[0043] The one or more predetermined conditions for determining the first area based on the one or more predetermined conditions may include one or more of the following:

[0044] - one or more conditions related to an object in the virtual environment in which the light probe is generated; and / or

[0045] - One or more conditions related to light probes in one or more previous frames of the virtual environment (ie, light probes generated for one or more previous frames of the content).

[0046] Taking into account the conditions related to the objects in the virtual environment, the predetermined conditions may include one or more conditions related to one or more objects and / or one or more characteristics of the objects in the virtual environment. Then, determining the first area may include determining that an area of ​​the light probe (i.e., an area of ​​the light probe whose corresponding output viewing direction faces the object) faces the object in the one or more objects. In this way, for probe pixels facing the one or more objects, light information may be stored at a higher resolution, which may be more relevant to the runtime lighting of the environment.

[0047] Determining whether a region of a Light Probe is facing an object in the virtual environment may be based on a threshold cone angle (i.e., beam angle) around the output viewing direction corresponding to that region. For example, objects within a 5 (or 10 or 20) degree cone angle from the output viewing direction may be determined to be facing the corresponding Light Probe region. In some cases, the threshold cone angle may vary depending on the characteristics of the object in question, e.g., a larger threshold cone angle may be used for objects with rougher surfaces (and therefore more likely to scatter light such that it reaches the relevant region of the Light Probe).

[0048] One or more objects may be selected based on their various characteristics, such as one or more of: brightness, luminosity, color, surface roughness, shape, level of detail (LOD), or proximity to other objects. For example, a first region may be determined as a region of a probe facing an object (such as the sun in a virtual environment) having a brightness above a predetermined threshold (e.g., as measured at the location of the light probe) and / or a luminosity above a predetermined threshold.

[0049] Alternatively or additionally, the first region may be determined as a region of the probe facing an object having a surface roughness (and / or frequency variation) above a predetermined threshold. In some cases, such an object may be more likely to scatter light and produce more complex lighting effects, and storing light information at a higher resolution in a region facing such an object may provide improved lighting effects.

[0050] Alternatively or additionally, the first region may be determined as the region of the probe facing objects of a color (e.g., red) and / or shape (e.g., polyhedron) within a predetermined list. For example, light reflections of red objects may be more noticeable to the end user than light reflections of white objects.

[0051] Alternatively or additionally, the first region may be determined as a region of the probe facing an object having a level of detail above a predetermined threshold and / or an object within a predetermined threshold distance of one or more other objects in the virtual environment (optionally having specific characteristics, e.g., similar to those described above). Similar to objects with high surface roughness, objects with high LOD (e.g., many small geometric shapes) or close to other objects may be more likely to scatter light and / or produce more complex lighting effects, and it may be advantageous to store the corresponding lighting information at a higher resolution.

[0052] The various predetermined thresholds described herein may be absolute (i.e., predefined) or relative to other objects in the current virtual environment. For example, one or more objects may include one or more objects having a LOD above a predetermined level and / or having the highest LOD among the objects in the current virtual environment.

[0053] It should be understood that the above merely represents exemplary conditions related to objects and their characteristics. Furthermore, these conditions may be used in any appropriate combination. For example, a first region may be determined as a region of probes facing objects having both brightness and LOD above a predetermined threshold. In some cases, one or more conditions may be jointly evaluated using a mathematical expression—for example, a polynomial function with weights for various object characteristics that outputs an overall "importance" score for different regions and / or pixels of the Light Probe. An appropriate importance score function may be determined empirically based on historical data related to the Light Probes and the virtual environment.

[0054] It should also be understood that in some cases one or more of the objects may be pre-selected by a developer. The orientation of these pre-selected objects relative to the light probe may then be tracked at step 210 to determine whether the area of ​​the light probe is facing any of the pre-selected objects.

[0055] Taking into account conditions associated with light probes in one or more previous frames, it should be understood that multiple frames depicting a virtual environment are typically rendered for displayed content (e.g., a video game), and light probes may be generated for one or more of these frames. For example, light probes may be regenerated for each frame (e.g., light information may be recollected for the probes) to reflect changes in the virtual environment over time (e.g., time of day or weather changes). Data associated with light probes from previous frames (and in particular which areas of the light probes were prominent in the previous frames) may provide an indicator of which areas of the probes are expected to be prominent in the current frame. Use of this data allows for more accurate prediction of the prominent areas of the probes in the current frame, so that light information for these areas may be stored at a higher resolution.

[0056] It should be understood that data associated with light probes of one or more previous frames may be used. Data from multiple frames may be further processed as appropriate. For example, data may be aggregated (e.g., averaged), and / or a recency factor may be applied to assign a higher weight to more recent data (i.e., data from more recent frames).

[0057] In some cases, determining the first region may include determining a region of light probes having one or more characteristics satisfying a predetermined condition in one or more previous frames. The relevant characteristics of the light probes in the previous frame may, for example, include luminosity and / or color. For example, the first region may be determined as a region of light probes having a luminosity above a predetermined threshold and / or having a color within a predetermined list (e.g., various red colorings) in the previous frame. The lighting effect provided by such a region of light probes may be more noticeable to the user, and thus improved lighting effect quality may be obtained by assuming that the characteristics of the regions in the previous frames will be at least partially maintained in the current frame, and by storing light information for those regions at a higher resolution.

[0058] In some cases, determining the first region may include determining regions of Light Probes used to illuminate one or more objects in the virtual environment in one or more previous frames while rendering the content. For example, it may be determined which regions of Light Probes were closest to moving objects in the environment in previous frames, and therefore used to illuminate the moving objects—based on this, it may be anticipated that the same (and / or adjacent) regions of Light Probes may be used in the current frame, and higher resolution light information may be stored for those regions.

[0059] The first region can be determined based on the extent to which the region was used to illuminate objects in the virtual environment in one or more previous frames. For example, the first region can be determined as the region that was most frequently used to illuminate objects in one or more previous frames. The extent to which a region is used to illuminate an object can be quantified in various suitable ways. For example, a counter can be provided for each region of the light probe (e.g., each pixel or each 2×2, 4×4, or 8×8 pixel block), and the counter is incremented each time the region is used to illuminate an object in a frame. The first region can then be determined based on the counters for each region—for example, the first region can be determined as the region with the highest counter. In some cases, the extent to which a region is used to illuminate an object can also include a recency weighting, so that regions used to illuminate objects in more recent frames are more likely to be selected for the first region—for example, in a counter implementation, at each frame, the current counter can be reduced by a predetermined factor (e.g., a factor of 2) so that older instances of using a region gradually reduce the impact on the overall counter.

[0060] In some cases, the first region may be determined based on the virtual camera position and / or orientation in the virtual environment in one or more previous frames. For example, the first region may be determined based on whether the region of the probe was facing the virtual camera in a previous frame. In this way, higher resolution light information may be stored for light probe regions that are facing the virtual camera (and may be facing the virtual camera in the current frame), so the quality of their light information is more likely to be apparent to the user.

[0061] In some cases, the motion of an object in the virtual environment relative to the light probe may be predicted based on one or more previous frames of the virtual environment. For example, the motion of the object between previous frames may be extrapolated to predict the future motion of the object. The first region may then be determined based on the predicted motion of the object, and the one or more predetermined conditions (for determining the first object) may include a condition related to the predicted motion of the object. For example, the first region of the light probe may be determined to be the region to which the object is determined to be most likely closest in the current frame (and therefore most likely to be used to illuminate the object) based on the predicted motion of the object.

[0062] It should be understood that the first area can be determined based on both conditions related to the object in the virtual environment and the previous frame of the virtual environment. For example, the first area can be determined based on whether the area is facing an object with a specific characteristic and based on whether the object exists in the previous frame. Objects that do not exist in the previous frame (or exist in fewer previous frames) can be prioritized for determining the first area because the user is more likely to notice the lighting effects associated with objects that are new to the environment. Alternatively or additionally, the first area can be determined based on whether the area is facing an object with a specific characteristic and based on whether the area is used to illuminate the object in the virtual environment in the previous frame.

[0063] In some cases, the first area may be determined based on gaze information associated with one or more users of the content (i.e., the rendered virtual environment). Such gaze information may be obtained, for example, using HMD 120. The gaze information may include historical data associated with user gaze of the content. The first area may then be determined based on the historical user gaze data. For example, the first area may be determined as an area of ​​the light probe facing a point on the display where the user gaze is directed, on average, toward a corresponding point in the content (e.g., a corresponding point of gameplay in a video game (such as a checkpoint)).

[0064] In some cases, the first area may be determined based on historical data related to the content. Observing historical trends in the content allows for an estimate of which areas of the probe may be prominent when the content is rendered in the future, and therefore which areas should store higher resolution light information. For example, multiple previous instances of rendered content may be analyzed to determine the average position of moving objects in the virtual environment, and the first area may be determined as the area of ​​the light probe that is closest to that determined average position. The historical data may relate to one or more users of the content. In some cases, the content may first be displayed to multiple test users to collect data related to the content, based on which the prominent areas of the light probe may be estimated as described above.

[0065] Step 220 includes collecting light information from the virtual environment. The light information is related to light passing through the location of the light probe in the virtual environment.

[0066] Collecting light information may include multiple light collection operations for different areas of the probe. Each light collection operation may be implemented as a shading operation for a given probe area. The light collection operation for the probe area may include sampling incident light from multiple directions of the virtual environment and aggregating the incident light into light information corresponding to an output viewing direction associated with the probe area. Therefore, each light collection operation may correspond to collecting light for different viewing directions from a light probe. For example, a cubemap convolution and a BRDF (bidirectional reflectance distribution function) may be used to aggregate the incident light.

[0067] In existing arrangements, a separate light collection operation is performed for each probe pixel, and the incident light is uniformly sampled from all possible incident directions from the virtual environment (which may include hundreds or thousands of directions). It will be appreciated that these arrangements therefore have a high associated computational cost for generating light probes.

[0068] In one or more examples of the present disclosure, collecting light information includes importance sampling the light information. Importance sampling can be applied to one or more light collection operations. For a given light collection operation, importance sampling can include samples taken along a specular lobe aligned with incident light based on data related to a corresponding surface of the virtual environment. This allows reducing the total number of samples that need to be taken in a light collection operation by prioritizing samples that will have the greatest impact on the probe area where light is collected, thereby reducing the computational cost of generating light probes without significantly reducing their quality.

[0069] Step 230 includes storing the light information collected at step 220 in the light probe. At step 230, the light information is stored in a first region of the light probe at a higher resolution than in a second region of the light probe. The light information may be stored in the light probe in any suitable format. For example, the light information may be stored as spherical harmonic basis functions (such as L2 spherical harmonics).

[0070] By storing more detailed light information for more prominent areas of Light Probes in this way, Light Probes can provide higher quality lighting effects at runtime without increasing (or in some cases actually reducing) the computational cost of generating Light Probes.

[0071] As discussed above, collecting light information may include multiple light collection operations for different regions of the light probe.Storing light information may include storing results of a light collection operation for a first region at a higher resolution than a light collection operation for a second region.

[0072] Storing the results of light collection operations at different resolutions for different regions of the probe can be achieved by changing the number of probe pixels that each light collection operation may affect. For example, a face of a light probe (e.g., face 302-a) can be divided into regions of a target light information resolution (e.g., region 306-a is assigned a higher target light information resolution than the rest of face 302-a), which defines the number and arrangement of probe pixels affected by light operations in a given region (e.g., 2x2, 4x2, 2x4, etc.). The results of each light collection operation can then be written to the number and arrangement of probe pixels according to the target light information resolution for that region.

[0073] Each light operation may include a shading operation. In order to vary the number of probe pixels that each light gathering operation may affect, variable rate shading may be applied during the environment map creation (light gathering) or probe filtering (light filtering) stages of light probe construction.

[0074] Alternatively or additionally, storing the results of the light collection operation at different resolutions for different regions of the probe may be achieved by presetting the resolution of the probe pixels in different regions of the probe before collecting light information at step 220. For example, the probe pixels may be preset to a first resolution in a first region and preset to a second resolution in a second region. Light information may then be collected for each probe pixel (which will have different sizes in different regions) at step 220 and stored in each probe pixel at step 230.

[0075] Storing the results of the light collection operations at different resolutions for different regions of the light probe actually allows the number of light collection operations performed for different regions of the light probe to be varied. In this way, more light collection operations may be performed for probe pixels in a first region of the light probe (which were determined at step 210 to be more prominent for the lighting effect at runtime) than for probe pixels in a second region of the light probe.

[0076] Respective resolutions for storing light information in the first region and the second region may be predetermined. For example, appropriate resolutions for storing light information for more or less prominent regions may be determined empirically based on user feedback on a rendering environment including light probes. In some cases, the resolution may depend on the hardware used to generate the light probes (e.g., its static properties such as processing power, or dynamic properties such as current load), e.g., a higher resolution (for the first region and / or the second region) may be used when more processing power is available.

[0077] Alternatively or additionally, the respective resolutions for storing light information in the first and second regions may be determined based on criteria similar to the criteria used to determine the first region. For example, the resolution of a region may be determined based on one or more of whether the region faces a predefined object, a characteristic of the object (e.g., brightness), and / or whether the region was used to illuminate an object in a previous frame and / or the degree to which the region was used to illuminate an object in a previous frame. The resolution may be determined using an empirically determined function that includes variables of one or more of these parameters.

[0078] The resolution of the first and / or second region may be different in different directions. For example, different resolutions may be provided in the horizontal and vertical directions, for example, the result of the light collection operation in a given region may be stored in a 1×2 or 2×1 probe pixel grid.

[0079] In some cases, amplification techniques may be applied to the optical probe. Temporal and / or spatial amplification techniques may be used. Various amplification techniques may be used, such as FidelityFX super-resolution (FSR), point amplification, or bilinear amplification.

[0080] Upscaling may be applied to the light information stored in the first region and / or the second region of the light probe to increase the resolution of the stored light information. Applying upscaling in this manner allows the resolution of the light information stored in the light probe to be increased while being computationally cheaper than rendering the light probe at a higher resolution.

[0081] A magnification may be applied to the first region of the light probe to further increase its resolution, thereby obtaining higher resolution information for the most prominent output viewing direction from the light probe.

[0082] Alternatively or additionally, upscaling may be applied to the second region of the Light Probe to increase the resolution of the light information in the second region. For example, upscaling may be applied to the light information stored in the second region to increase its resolution to the resolution of the first region. This allows Light Probes with uniform resolution to be obtained at reduced computational cost, where the first region is prioritized and rendered locally at a higher resolution, and the second region is simply upscaled to a higher resolution (and therefore has a higher probability of artifacts).

[0083] Alternatively or additionally, upscaling may be applied to the optical probe as a whole, thereby more generally improving the resolution of the optical probe at reduced computational cost.

[0084] It should be understood that the above reference Figure 2 The described approach offers several advantages over existing arrangements.

[0085] In existing arrangements, a separate light collection operation is performed for each probe pixel, and the probe pixel resolution is constant over the surface of the light probe. This results in an inherent trade-off between the quality of the light information and the computational cost of generating the light probe, as increasing one parameter leads to an increase in the other.

[0086] Instead, by varying the resolution at which light information is written to the Light Probes between different regions of the Probes at step 230, the present method allows the regions of the Probes that are most important for lighting effects at runtime (and therefore the output viewing directions) to be prioritized, and more detailed light information for these regions to be stored. This allows an improved balance between the quality of lighting effects at runtime and the computational cost of generating Light Probes. In particular, as described above, each probe pixel / region corresponds to an output viewing direction from a probe, and therefore the present method allows the most important directions of light to be prioritized for storage in the probes. For lighting effects, the directionality of light is often crucial, and the present method allows ensuring that light information is stored at a suitably high resolution for salient directions, while using a lower resolution for less relevant directions to save computational resources.

[0087] It should be appreciated that, alternatively or additionally, to determining the first "higher resolution" region, step 210 may include determining a second "lower resolution" region of the light probe. In other words, step 210 may include determining a second region of the light probe that is less relevant to runtime lighting (e.g., a region facing objects in the virtual environment whose brightness is below a predetermined threshold), and step 230 may include storing light information at a lower resolution in this second region to save computational resources. The same techniques described herein (e.g., applied in reverse) may be used to determine this second region.

[0088] It should also be understood that step 210 may include determining multiple regions of the probe (i.e., the light probe 300 may be divided into more than two regions) and / or that the light information may be stored in these regions at more than two different resolutions in step 230. For example, the light probe 300 may be divided into a first region such as region 306-a (having an associated first resolution of stored light information), a second region such as region 306-b (having an associated second resolution of stored light information), and a third region such as facet 302-c (having an associated third resolution of stored light information).

[0089] Of course, it should be understood that the techniques described herein may also be extended to generate multiple light probes in one or more virtual environments.

[0090] Return to reference Figure 2 In an embodiment of the present invention, a method for generating a light probe in a virtual environment includes the following steps.

[0091] Step 210 includes determining a first region of the light probe based on one or more predetermined conditions, as described elsewhere herein.

[0092] Step 220 includes collecting light information from the virtual environment, the light information relating to light passing through the location of the light probe in the virtual environment, as described elsewhere herein.

[0093] Step 230 includes storing the optical information in the optical probe, wherein the optical information is stored at a higher resolution in a first region of the optical probe than in a different, second region of the optical probe, as described elsewhere herein.

[0094] It will be apparent to those skilled in the art that variations of the above methods corresponding to the operation of the various embodiments of the methods and / or apparatus described and claimed herein are considered to be within the scope of the present disclosure, including but not limited to:

[0095] - the step 220 of collecting light information comprises a plurality of light collection operations for different regions of the light probe, and the step 230 of storing light information comprises storing the results of the light collection operation for a first region at a higher resolution than the light collection operation for a second region, as described elsewhere herein;

[0096] - in this case, optionally storing the results of the light collection operations for the first area at a higher resolution than the resolution of the light collection operations for the second area, including storing the results of each light collection operation for the first area in fewer pixels of the light probe than the pixels of the light collection operations for the second area, as described elsewhere herein;

[0097] - the one or more predetermined conditions (used to determine the first area at step 210) include at least one condition related to one or more objects in the virtual environment, as described elsewhere herein;

[0098] - In this case, optionally, depending on whether the area of ​​the light probe is facing one or more pairs

[0099] determining a first region based on at least one object in the image, as described elsewhere herein;

[0100] - in this case, optionally, based on one or more properties of one or more objects

[0101] identifying a first region, as described elsewhere herein;

[0102] ■ wherein, optionally, the one or more characteristics of the object include one or more selected from a list consisting of: brightness, surface roughness, shape, level of detail, or proximity to other objects, as described elsewhere herein;

[0103] - the one or more predetermined conditions (used to determine the first area in step 210) include at least one condition related to a light probe in one or more previous frames of the virtual environment, as described elsewhere herein;

[0104] - in this case, optionally, the first region is determined based on whether the region of the light probe has one or more characteristics that satisfy a predetermined condition in one or more previous frames, as described elsewhere herein;

[0105] ■ wherein, optionally, the one or more characteristics of the light probe include one or more selected from the list consisting of: luminosity, or color, as described elsewhere herein;

[0106] - in this case, optionally, determining the first region based on whether the region of the light probe was used to illuminate one or more objects in the virtual environment in one or more previous frames, as described elsewhere herein;

[0107] ■ wherein, optionally, the first region is determined based on the extent to which the region was used in one or more previous frames to illuminate one or more objects in the virtual environment, as described elsewhere herein;

[0108] - in this case, optionally, the first region is determined based on whether the region was facing a virtual camera in the virtual environment in one or more previous frames, as described elsewhere herein;

[0109] - the method further comprises predicting the motion of an object in the virtual environment relative to the light probe based on one or more previous frames of the virtual environment, as described elsewhere herein;

[0110] - in this case, optionally, the first area is determined according to a predicted movement of an object in the virtual environment, as described elsewhere herein;

[0111] - determining a first region based on gaze information associated with one or more users of the content (corresponding to the virtual environment), as described elsewhere herein;

[0112] - determining the first region based on historical data related to the content, as described elsewhere herein;

[0113] - the first region and the second region comprise different areas of the face of the light probe, as described elsewhere herein;

[0114] - the first region and the second region comprise different faces of the light probe, as described elsewhere herein;

[0115] - the step 220 of collecting light information comprises importance sampling the light information as described elsewhere herein;

[0116] - the method further comprises applying amplification to the light information stored in the first region and / or the second region of the light probe to increase the resolution of the stored light information as described elsewhere herein; and

[0117] - In this case, optionally, the upscaling is a temporal or spatial upscaling.

[0118] It should be understood that the above-described methods may be performed on conventional hardware appropriately adapted to the application by software instructions or by the inclusion or replacement of dedicated hardware.

[0119] Thus, the required adaptation of existing parts of conventional equivalent devices can be implemented in the form of a computer program product comprising processor-implementable instructions stored on a non-transitory machine-readable medium (such as a floppy disk, optical disk, hard disk, solid-state disk, PROM, RAM, flash memory, or any combination of these or other storage media), or implemented in hardware as an ASIC (application-specific integrated circuit) or FPGA (field programmable gate array) or other configurable circuit suitable for adapting conventional equivalent devices. Separately, such a computer program can be sent via a data signal on a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks.

[0120] refer to Figure 4 In an embodiment of the present invention, a system for generating a light probe in a virtual environment may include the following:

[0121] A determination processor 410 is configured (eg, via suitable software instructions) to determine a first region of the optical probe based on one or more predetermined conditions, as described elsewhere herein.

[0122] A collection processor 420 is configured (eg, via suitable software instructions) to collect light information from the virtual environment, the light information relating to light passing through the location of the light probe in the virtual environment, as described elsewhere herein.

[0123] A storage processor 430 configured (e.g., via suitable software instructions) to store optical information in the optical probe, wherein the optical information is stored at a higher resolution in a first region of the optical probe than in a different second region of the optical probe, as described elsewhere herein.

[0124] These processors 410, 420, 430 may be implemented, for example, by the CPU 20 of the entertainment system 10. Of course, it should be understood that the functionality of these processors may be implemented by any suitable number of processors located at any suitable number of devices and any suitable number of devices as appropriate, without requiring a one-to-one mapping between functionality and devices or processors.

[0125] The foregoing discussion discloses and describes only exemplary embodiments of the present invention. As will be appreciated by those skilled in the art, the present invention may be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the disclosure of the present invention is intended to be illustrative, rather than limiting, the scope of the present invention and other claims. The present disclosure, including any readily discernible variations taught herein, defines in part the scope of the preceding claim terms so that no inventive subject matter is dedicated to the public.

Claims

1. A method for generating a light probe in a virtual environment, the method comprising: determining a first region of the optical probe based on one or more predetermined conditions; collecting light information from the virtual environment, the light information relating to light passing through a location of the light probe in the virtual environment; as well as The light information is stored in the light probe, wherein the light information is stored at a higher resolution in the first region of the light probe than in a different, second region of the light probe.

2. The method of claim 1 , wherein collecting light information comprises a plurality of light collection operations for different regions of the light probe, and wherein storing the light information comprises storing results of the light collection operation for the first region at a higher resolution than the light collection operation for the second region.

3. The method according to claim 2, wherein: Storing results of light collection operations for the first region at a higher resolution than a resolution of light collection operations for the second region includes storing the result of each light collection operation for the first region in fewer pixels of the light probe than pixels of the light collection operation for the second region.

4. The method according to claim 1, wherein: The one or more predetermined conditions include at least one condition related to one or more objects in the virtual environment.

5. The method according to claim 4, wherein: The first area is determined according to whether an area of ​​the light probe faces at least one of the one or more objects.

6. The method according to claim 4, wherein: The first region is determined based on one or more characteristics of the one or more objects, wherein the one or more characteristics of the objects include one or more selected from a list consisting of: brightness, surface roughness, shape, level of detail, or proximity to other objects.

7. The method according to claim 1, wherein: The one or more predetermined conditions include at least one condition related to the light probe in one or more previous frames of the virtual environment.

8. The method according to claim 7, wherein: The first region is determined based on whether a region of the light probe has one or more characteristics that satisfy a predetermined condition in the one or more previous frames, wherein the one or more characteristics of the light probe include one or more selected from a list consisting of: luminosity, or color.

9. The method according to claim 7, wherein: The first area is determined based on whether an area of ​​the light probe was used to illuminate one or more objects in the virtual environment in the one or more previous frames; and / or whether an area faced a virtual camera in the virtual environment in one or more previous frames.

10. The method of claim 1 , further comprising predicting the motion of an object in the virtual environment relative to the light probe based on one or more previous frames of the virtual environment; wherein The first area is determined according to predicted movement of the object in the virtual environment.

11. The method according to claim 1, wherein: The first region and the second region comprise different regions of a face of the light probe, or the first region and the second region comprise different faces of the light probe.

12. The method according to claim 1, wherein: Collecting light information includes importance sampling the light information.

13. The method according to claim 1, further comprising: Amplification is applied to the stored light information in the first region and / or the second region of the light probe to increase the resolution of the stored light information.

14. A computer program comprising computer executable instructions adapted to cause a computer system to perform the method according to claim 1.

15. A system for generating a light probe in a virtual environment, the system comprising: a determination processor configured to determine a first region of the light probe based on one or more predetermined conditions; a collection processor configured to collect light information from the virtual environment, the light information relating to light passing through a location of the light probe in the virtual environment; as well as A storage processor is configured to store the optical information in the optical probe, wherein the optical information is stored at a higher resolution in the first region of the optical probe than in a different second region of the optical probe.

Images