System and method for rendering graphics data

Abstract

A computer-implemented method for rendering graphical data representing a 3D scene on a video gaming system, the 3D scene comprising one or more objects and one or more light sources. The method comprises the following steps: obtaining a set of rendering parameter values for performing ray tracing on the scene; changing the set of rendering parameter values across a plurality of pixels of a frame of the scene; and performing spectral rendering using the changed rendering parameter values to generate frames of the scene.

Description

Technical Field

[0001] This invention belongs to the field of video game systems and relates to systems and methods for rendering graphics data. Background Technology

[0002] Ray-traced spectral rendering is a technique in computer graphics used to simulate the interaction between the spectrum of light wavelengths (e.g., the spectrum of visible light wavelengths) and surfaces and materials in a scene. Ray-traced spectral rendering uses algorithms that simulate real-world physics to accurately model how light is absorbed, transmitted, and / or scattered by different materials at different wavelengths.

[0003] By simulating the interactions of light across wavelengths of the spectrum, ray-traced spectral rendering can offer improvements over traditional rendering techniques, which often simplify the behavior of light, treating it as a single entity or having only a few discrete colors (e.g., red, blue, and green). This is because light in the real world comprises a continuous wavelength spectrum, and different materials interact with light in complex ways depending on the wavelength.

[0004] Ray-traced spectral rendering allows for more realistic rendering effects such as dispersion (separating light into its component colors), subsurface scattering (the diffusion of light through translucent materials), and spectral reflection and refraction (the bending of light as it passes through different media). Therefore, ray-traced spectral rendering can produce more lifelike images compared to traditional rendering techniques.

[0005] However, ray-traced spectral rendering is computationally intensive and requires significant processing power. This is partly due to the fact that ray-traced spectral rendering involves tracing every ray at multiple wavelengths, which significantly increases the computational workload compared to traditional rendering techniques that only consider a single wavelength.

[0006] Some algorithms incorporate wavelength multiplexing techniques, allowing each ray to comprise multiple wavelengths of light. These multiple wavelengths are typically chosen randomly from a set of wavelengths. However, wavelength multiplexing still requires simulating a large number of rays to gather the necessary information, and is therefore computationally expensive.

[0007] Therefore, a method is needed to overcome these problems and facilitate ray-traced spectral rendering with reduced computational intensity. Summary of the Invention

[0008] In a first aspect of the invention, a computer-implemented method is provided for rendering graphical data representing a 3D scene comprising one or more objects and one or more light sources on a video game system, the method comprising: obtaining a set of rendering parameter values ​​for ray tracing of the scene; changing a set of rendering parameter values ​​for multiple pixels across frames of the scene; and performing spectral rendering using the changed rendering parameter values ​​to generate frames of the scene.

[0009] A “scene” can be represented by a data structure that includes information required to render a frame, such as data describing one or more objects (e.g., position, vertices, material properties, and other data known in the art) and data describing one or more light sources (e.g., position, color, intensity, orientation, and other data known in the art). A “frame” can refer to an image rendered based on information stored in the scene data structure. “Multiple pixels” can refer to an array of pixels (e.g., a two-dimensional array) that forms a frame.

[0010] The set of rendering parameter values ​​can be understood as a set of values ​​for a specific rendering parameter. For example, a rendering parameter can be a wavelength (e.g., a wavelength used for ray tracing), and the set of rendering parameter values ​​can be a set of wavelengths used for ray tracing of a scene (i.e., a set of wavelength values).

[0011] The set of rendering parameter values ​​that change across multiple pixels in a frame can be understood as the rendering parameter value used for ray tracing associated with a first pixel being different from the rendering parameter value used for ray tracing associated with a second pixel. For example, when the rendering parameter is the wavelength of the ray to be traced, the wavelength sampled for the first pixel may be different from the wavelength sampled for the second pixel. Therefore, variations in rendering parameter values ​​across multiple pixels are possible.

[0012] As described above, ray tracing spectral rendering techniques typically involve tracing each ray at multiple wavelengths, resulting in a significant amount of computational effort required to render a frame. This invention can advantageously reduce the number of rendering parameter values ​​sampled for each pixel, thereby reducing the computational effort required to render a frame. Specifically, a set of rendering parameter values ​​that changes across multiple pixels results in fewer rendering parameter values ​​sampled for each pixel during ray tracing. For example, when the rendering parameter is wavelength, fewer wavelengths can be used to sample each ray for a given pixel than existing ray tracing spectral rendering techniques. In the context of this invention, the term "sampling" can refer to performing ray tracing iterations.

[0013] In some embodiments, changing the set of rendering parameter values ​​across multiple pixels includes: selecting a first group of pixels from the multiple pixels; and assigning a corresponding subset of rendering parameter values ​​to each pixel in the first group of pixels; wherein each pixel in the first pixel subset is assigned a different subset of rendering parameter values. A “subset of rendering parameter values” can be understood as a proper subset, such that there exists fewer elements than the set of rendering parameter values. In this way, the subset of rendering parameter values ​​assigned to each pixel is smaller than the resulting set of rendering parameter values. Advantageously, fewer rendering parameter values ​​can be sampled for each pixel because all rendering parameter values ​​in the set less than the set of rendering parameter values ​​are used for ray tracing, thereby reducing the computational effort required to render a frame. Preferably, each rendering parameter value in the subset of rendering parameter values ​​exists in at least one subset of rendering parameter values, such that all rendering parameter values ​​are used in the ray tracing process of the first group of pixels.

[0014] There may be no common rendering parameter values ​​between each subset of rendering parameter values. That is, there may not be rendering parameter values ​​for more than one pixel. For example, when the rendering parameter is wavelength, a first subset of the wavelength can be assigned to the first pixel, and a second subset of the wavelength can be assigned to the second pixel, where the first subset of the wavelength is different from the second subset. This can be applied to all pixels in the selected first set of pixels. In this way, the number of ray tracing iterations can be greatly reduced.

[0015] Selecting a first set of pixels from a plurality of pixels can include selecting a pixel array from the plurality of pixels. For example, the pixel array can be a two-dimensional pixel array. Each pixel in the array can be close to at least one other pixel in the array. The pixel array is preferably a subset of the plurality of pixels in a frame. Multiple pixel arrays can exist in a frame, and each pixel in the pixel array can be assigned a different subset of rendering parameter values. Advantageously, the proximity of pixels in the first set of pixels mitigates any loss of frame image quality because all rendering parameter values ​​are sampled at least once in the first set of pixels.

[0016] The number of pixels in the first group of pixels can be equal to the number of rendering parameter values ​​in the set of rendering parameter values. In this way, each rendering parameter value in the set of rendering parameter values ​​is applied to the first group of pixels, where different rendering parameter values ​​are assigned to each pixel. For example, if the rendering parameter is wavelength, the number of pixels in the first group of pixels can be equal to the number of wavelengths in the set of wavelengths, and a different wavelength can be assigned to each pixel for ray tracing. In this way, the same number of wavelengths can be sampled as in conventional methods, but each pixel can use only one wavelength, thus significantly reducing the number of rays required for the ray tracing process.

[0017] The method may further include: assigning a corresponding second subset of rendering parameter values ​​to each pixel in the first set of pixels; and performing spectral rendering using the second subset of rendering parameter values ​​to generate a second frame of the scene; wherein each pixel in the first pixel subset is assigned a different second subset of rendering parameter values. In this way, the rendering parameter values ​​used to render a particular pixel can change from the first frame to the second frame. For example, when the rendering parameter is wavelength, the wavelength used to render a particular pixel changes between the first and second frames. At typical frame rates, the user may not perceive the change in pixel color from one frame to the next, but this significantly reduces the number of ray tracing steps required per frame, thereby reducing the computational workload required to render a sequence of frames.

[0018] In some embodiments, the set of rendering parameter values ​​that change for multiple pixels across frames in a scene includes: selecting a first group of pixels from the multiple pixels; selecting a second group of pixels from the multiple pixels; assigning a first set of rendering parameter values ​​to each pixel in the first group of pixels; and assigning a second set of rendering parameter values ​​to each pixel in the second group of pixels. In this way, the set of rendering parameter values ​​that change for multiple pixels across frames can be achieved by using rendering parameter values ​​that are different for the first group of pixels than those used for the second group of pixels.

[0019] In some embodiments, selecting a first group of pixels from a plurality of pixels includes: identifying a user's point of interest in the 3D scene; determining the point of interest pixels, wherein the point of interest pixels are pixels in a frame corresponding to the point of interest in the 3D scene; and selecting one or more pixels within the boundary of the point of interest pixels as the first group of pixels. Therefore, the first group of pixels may be a group of pixels associated with the user's central visual region (i.e., the fovea region). The user may be more sensitive to the color of this first group of pixels. Identifying the first group of pixels in this manner advantageously provides a means for the system to assign a large number of rendering parameter values ​​to pixels that have a significant impact on the user's perceived quality.

[0020] The boundary can be a shape centered on the pixel of interest. The shape can be circular, and the boundary can be defined by a threshold number of pixels. In this way, the system can advantageously select the first group of pixels in a computationally inexpensive manner.

[0021] In some embodiments, selecting a second group of pixels includes selecting one or more peripheral pixels as the second group of pixels; wherein, peripheral pixels are pixels located outside the boundaries of the frame. Therefore, this second group of pixels can be associated with a peripheral region of the user's vision, which may be less sensitive to color. Identifying the second group of pixels in this way can advantageously provide the system with a means of identifying the second group of pixels in a computationally inefficient manner.

[0022] The first set of rendering parameter values ​​can include a larger number of rendering parameter values ​​than the second set. In this way, fewer rendering parameter values ​​can be used to render the second set of pixels. The second set of pixels can be associated with the peripheral region of the user's vision, and the computational load required to render the second set of pixels can be advantageously reduced, while with minimal perceptible loss of detail.

[0023] Selecting a first group of pixels from a plurality of pixels may include selecting a plurality of neighboring pixels that have corresponding spectral characteristics. That is, each pixel may be associated with an object or surface in the scene that has corresponding spectral characteristics. In the context of this invention, "corresponding spectral characteristics" may refer to spectral characteristics that have threshold similarity. Therefore, the first group of pixels may be pixels that can be sampled using similar or identical rendering parameter values.

[0024] Selecting multiple neighboring pixels with corresponding spectral characteristics can include selecting multiple pixels associated with a first object in the 3D scene. This can be achieved by comparing pixel position data with geometric data describing the boundary of the first object. In this way, the system can identify neighboring pixels with corresponding spectral characteristics in a computationally inefficient manner.

[0025] The set of rendering parameter values ​​may include a set of wavelengths, and the first set of rendering parameter values ​​assigned to each pixel in the first group of pixels may include: identifying one or more wavelengths that provide a threshold contribution to each pixel; and selecting multiple wavelengths that provide a threshold contribution to each pixel as a subset of the rendering parameter values. "One or more wavelengths that provide a threshold contribution to each pixel" may refer to wavelengths that substantially affect the "appearance" of the pixel. Advantageously, computational resources can be saved because the pixel can be sampled without using wavelengths that do not satisfy the threshold contribution to the pixel.

[0026] Identifying one or more wavelengths that contribute a threshold contribution to each pixel may include: identifying one or more wavelengths associated with a first object and at least one of one or more light sources. "One or more wavelengths associated with the first object" may refer to wavelengths reflected by the first object. "One or more wavelengths associated with at least one light source" may refer to wavelengths emitted by at least one light source. This wavelength information may be stored in texture data and lighting data describing the 3D scene. Advantageously, the system can identify these one or more wavelengths in a computationally inefficient manner.

[0027] This invention provides an alternative to Montecarlo sampling. Montecarlo sampling is a process in which uniformly random samples from a domain are taken to compute an average. Montecarlo sampling is known to introduce noise into the rendering process and has slow convergence. This invention provides an importance sampling process in which samples are distributed in more important regions (i.e., regions that contribute more to the convolution of each pixel). The samples can be weighted accordingly, for example, by employing known importance sampling techniques to account for the non-uniform distribution. Advantageously, this invention can reduce noise and requires fewer samples to generate the same image quality. Therefore, this invention can be faster and / or computationally cheaper than known spectral rendering algorithms.

[0028] In some embodiments, the object spectrum includes wavelengths that satisfy a threshold reflectance intensity, and the light source spectrum includes wavelengths that satisfy a threshold emission intensity. In this way, the number of wavelengths sampled can be further reduced, since only wavelengths satisfying the threshold reflectance intensity in the object spectrum are compared with wavelengths satisfying the threshold emission intensity in the light source spectrum. Advantageously, the computational load can be further reduced.

[0029] In another aspect of the invention, a non-transitory storage medium is provided, comprising instructions that, when executed by a processor, cause the processor to perform the method of the first aspect. The processor may include a central processing unit (CPU) and a graphics processing unit (GPU).

[0030] In another aspect of the invention, a computer program including instructions is provided that, when executed by a processor, causes the processor to perform the method of the first aspect. The processor may include a central processing unit (CPU) and a graphics processing unit (GPU).

[0031] In another aspect of the invention, a video game system for rendering graphical data of a 3D scene is provided, the system including a processor and a GPU and configured to perform the method of the first aspect.

[0032] In another aspect of the invention, a computer-implemented method is provided for rendering graphical data representing a 3D scene on a video game system, the 3D scene including one or more objects and one or more light sources, the method comprising: obtaining a set of rendering parameter values ​​for ray tracing of the scene; identifying one or more rendering parameter values ​​that contribute a threshold to the scene; and performing spectral rendering using the rendering parameter values ​​that contribute a threshold to the scene to generate frames of the scene. Attached Figure Description

[0033] Figure 1 A video game system according to the present invention is shown;

[0034] Figure 2It shows how to use Figure 1 The flowchart of the system's method for rendering graphics data;

[0035] Figure 3 A method flowchart of a first embodiment is shown, illustrating the steps of changing a set of rendering parameter values ​​for multiple pixels across frames in a scene;

[0036] Figure 4 An example scene including a 10×10 pixel array is shown;

[0037] Figure 5A A flowchart illustrating a second embodiment of the method is provided, showing the steps of changing a set of rendering parameter values ​​for multiple pixels across frames in a scene.

[0038] Figure 5B A method flowchart illustrating an implementation of the step of selecting a first group of pixels from a plurality of pixels; and

[0039] Figure 6 It shows how to use Figure 1 An alternative implementation of the system's method for rendering graphics data. Detailed Implementation

[0040] Figure 1 A video game system 100 configured to implement the methods disclosed herein is shown.

[0041] System 100 includes a central processing unit (CPU) 102; memory 104; a graphics processing unit (GPU) 106; graphics data storage 108; a frame buffer 110; a display interface 112; and a display 114. System 100 also includes a system bus 118 arranged to communicatively couple the CPU 102, memory 104, GPU 106, and frame buffer 110. System 100 may also include other components typical of video game systems, such as a user input interface 116.

[0042] CPU 102 is configured to control the operation of system 100. For example, CPU 102 is configured to load instructions stored on memory 104 to cause CPU 102 to perform the methods disclosed herein. Memory 104 may include one or more volatile or non-volatile storage devices, such as random access memory (RAM), dynamic RAM (DRAM), and / or any other suitable storage medium including instructions that, when executed by CPU 102, cause CPU 102 to perform the steps of the methods disclosed herein. Memory 104 includes one or more graphics rendering instructions that instruct CPU 102 to render graphics data to display 114.

[0043] GPU 106 is arranged to be controlled by CPU 102. For example, CPU 102 includes a GPU driver (not shown) for controlling the operation of GPU 106 to facilitate the execution of the methods disclosed herein. To process graphics rendering instructions, CPU 102 controls the GPU to perform the rendering of graphics data. These graphics rendering instructions conform to a graphics application programming interface (API), enabling the GPU to efficiently implement graphics rendering steps. GPU 106 includes GPU memory (not shown), which is arranged to store visual data such as textures, geometry, and other graphics assets. GPU memory includes, for example, video random access memory (VRAM).

[0044] CPU 102 and GPU 106 are coupled to graphics data storage 108 (e.g., via a bus (not shown)) so that CPU 102 and GPU 106 can read data and write data to graphics data storage 108. Graphics data storage 108 may include one or more volatile or non-volatile storage devices, such as random access memory (RAM), dynamic RAM (DRAM), and / or any other suitable storage medium.

[0045] Graphics data storage 108 is arranged to store graphics data representing a virtual 3D scene comprising one or more objects and one or more light sources. The graphics data includes geometric data, viewpoint data, texture data, lighting data, shading data, and other data describing the virtual scene. Geometric data includes a list of vertices defining the shape of one or more objects; a list of faces or polygons used to form the surfaces of one or more objects; and other optional data. Viewpoint data defines the position and orientation of the viewer or camera within the scene. Texture data includes spectral texture data describing the spectral behavior of a particular texture (e.g., reflection and / or absorption at a specific wavelength); and mapping coordinates that map the spectral texture data to the geometric surfaces of one or more objects. Lighting data defines the characteristics (e.g., position) and behavior (e.g., spectral properties) of one or more light sources within the scene. Shading data includes information about shading for different materials and provides instructions on how to combine spectral data during rendering.

[0046] The graphics data storage 108 is also arranged to store a set of rendering parameter values ​​used for ray tracing of the virtual scene. This set of rendering parameter values ​​typically includes a set of wavelengths used for ray tracing of the virtual scene. This set of wavelength values ​​is a group of wavelength values, and a predefined wavelength difference may exist between each wavelength value in the group. For example, the set of wavelength values ​​may be a discrete set of wavelength values ​​ranging from 380 nm to 700 nm, where the wavelength difference between adjacent wavelength values ​​in the set is 1 nm.

[0047] Frame buffer 110 is configured to receive rendered image data (e.g., in the form of processed pixels) from CPU 102 and / or GPU 106, the rendered image data representing an image (or frame) to be displayed on display 114. Display interface 112 is configured to retrieve the rendered image data from the frame buffer and cause display 114 to display the image represented by the rendered image data. Display 114 can be any device suitable for displaying the image represented by the rendered image data. Display 114 can be, for example, a liquid crystal display (LCD), a plasma display panel, or any other suitable display device.

[0048] In some embodiments, system 100 includes or communicates with a head-mounted display (HMD) (not shown), the HMD including display 114. The HMD is arranged to display a virtual reality (VR) environment representing a virtual scene. The HMD includes one or more light sources (e.g., infrared light sources) arranged to illuminate the user's eyes, and one or more optical sensors arranged to capture light reflected from the user's eyes. The HMD may also include a processing unit. System 100 is adapted to determine the user's point of interest or gaze. This can be achieved by CPU 102 or the processing unit of the HMD.

[0049] Figure 2 A flowchart of the method according to the present invention is shown. More specifically, Figure 2 A computer-implemented method 200 is shown for rendering graphics data of a 3D scene including one or more objects and one or more light sources on a video game system 100. One or more steps of method 200 are implemented by CPU 102 and GPU 106.

[0050] Step 202 of method 200 includes obtaining a set of rendering parameter values ​​for ray tracing the scene.

[0051] As described above, the set of rendering parameter values ​​includes a set of wavelength values ​​used for ray tracing of the virtual scene. The CPU 102 obtains the set of 202 wavelength values ​​by accessing the graphics data storage 108.

[0052] Step 204 includes a set of multiple pixel-level changes to rendering parameter values ​​across frames in the scene.

[0053] Figure 3 A first implementation of the steps is shown, which involves changing a set of 204 rendering parameter values ​​across multiple pixels in frames of a scene.

[0054] In the following example, the set of rendering parameter values ​​includes a set of wavelength values. The set of wavelength values ​​includes 9 different wavelength values.

[0055] CPU 102 selects a first group of 302 pixels from a plurality of pixels. For example, CPU 102 selects a pixel array (i.e., a two-dimensional pixel array) from a plurality of pixels. CPU 102 can select this first group of 302 pixels by identifying pixels associated with an object. Alternatively, CPU 102 can select any pixel array.

[0056] Figure 4 An example frame 400 comprising multiple pixels is shown. Frame 400 comprises a 10×10 pixel array 400. It will be understood that this example frame 400 is for illustration only, and the multiple pixels are not limited to a 10×10 pixel array. In this example, CPU 102 selects a 3×3 array of 302 pixels 402 by identifying pixels associated with object 404. For example, CPU 102 can determine which pixel locations fall within the boundaries of the object, which can be stored in geometric data on graphics data storage 108. Pixel array 402 can be considered as pixels of interest because they are associated with a specific object. It should be understood that, although... Figure 4 Only one pixel array 402 is shown, but multiple pixel arrays 402 can exist across frame 400. Each of the multiple pixel arrays 402 may not share common pixels. The change step can be applied separately to each array.

[0057] CPU 102 assigns a corresponding subset of 304 rendering parameter values ​​(e.g., wavelength values) to each pixel in the first group of pixels. Each of the rendering parameter values ​​exists in at least one subset of the subset of rendering parameter values. Therefore, all rendering parameter values ​​are used in the first group of pixels, and the same number of wavelengths are sampled as in conventional methods. For example, if each pixel is sampled at 10 wavelengths per frame per pixel in a conventional spectral rendering algorithm, in this invention, each pixel in the first group of pixels is sampled at fewer than 10 wavelengths, but all 10 wavelengths are sampled across the first group of pixels.

[0058] A subset of rendering parameter values ​​is a proper subset of the set of rendering parameter values, such that there are fewer elements in the subset than in the set of rendering parameter values. Continuing this example, each pixel in the first group of pixels 402 is assigned 8 or fewer distinct wavelength values ​​(i.e., fewer than 9 distinct wavelength values ​​in the set of wavelength values). Therefore, the reduced set of rendering parameter values ​​is sampled in subsequent rendering steps, thus reducing the computational load required to render each pixel.

[0059] Each pixel in the first pixel subset is assigned a different subset of 304 wavelength values. Therefore, a set of 204 rendering parameter values ​​is implemented, since each pixel in the first set of pixels is assigned a different subset of 404 rendering parameter values, making the rendering parameter values ​​different for each pixel in the first set of pixels.

[0060] In some embodiments, there are no common rendering parameter values ​​between each subset of the rendering parameter values. Therefore, each pixel is assigned a completely different subset of wavelengths. Continuing with this example, since there are 9 wavelength values ​​and 9 pixels in pixel array 402, each pixel in the first set of pixels 402 is assigned only one wavelength value.

[0061] Therefore, if each pixel is sampled at 9 wavelengths per frame in a conventional spectral rendering algorithm, in this implementation, 9 wavelengths will be sampled across the first group of pixels, but each pixel will be rendered using fewer than 9 wavelengths.

[0062] CPU 102 also assigns a corresponding second subset of 306 rendering parameter values ​​to each pixel in the first group of pixels. This second subset of rendering parameter values ​​is used in subsequent rendering steps, as discussed further below. Each pixel in the first group of pixels is assigned a different (e.g., corresponding) second subset of 306 rendering parameter values. For a particular pixel in the first group of pixels, the subset of second rendering parameter values ​​differs from the subset of first rendering parameter values. For example, for the first frame, pixel 410 is assigned a first subset of 306 wavelengths, such as wavelength 1. For the second frame, pixel 410 is assigned a second subset of 306 wavelengths, such as wavelength 2.

[0063] In conventional methods of spectral rendering, each pixel is sampled using a selection of wavelength values ​​for each frame. The same set of wavelength values ​​is sampled for each pixel in each frame. In this embodiment of the invention, each pixel is sampled with a reduced number of wavelength values, and the wavelength values ​​sampled for a pixel in the first frame differ from the wavelength values ​​sampled for a pixel in the second (subsequent) frames. At typical frame rates, the user will not perceive a change in pixel color from one frame to the next, but the number of ray tracing iterations per frame is significantly reduced, thereby reducing the computational resources required to render frames.

[0064] Figure 5A A second implementation of the steps is shown, which involves changing a set of 204 rendering parameter values ​​for multiple pixels across frames in a scene.

[0065] In the following example, the set of rendering parameter values ​​includes a set of wavelength values. The set of wavelength values ​​includes nine different wavelength values. In this second embodiment, step 204 occurs at runtime.

[0066] CPU 102 selects the first group of pixels 502 from multiple pixels.

[0067] Figure 5B An implementation of the step of selecting the first group of pixels 502 from a plurality of pixels is shown.

[0068] In this example, an HMD is used, but it should be understood that this can also be achieved using a conventional display with components for tracking the user's gaze.

[0069] CPU 102 identifies the user's point of interest, or gaze, in the 502A 3D scene. The user's "point of interest" refers to a specific location in the 3D virtual environment or scene where the user's gaze is focused at any given point. One or more infrared light sources illuminate the user's eyes with infrared light, and one or more sensors capture the reflected infrared light. CPU 102 detects the position of the pupil and the position of the corneal reflection (e.g., based on sensor data) and uses this data to determine a gaze vector representing the line of sight from the user's eyes. CPU 102 uses the gaze vector to determine the intersection point with virtual objects or surfaces in the virtual environment. This intersection point is the user's point of interest in the 3D scene.

[0070] CPU 102 determines 502B focus pixels. Focus pixels are pixels in a frame that correspond to a point of interest in the 3D scene. For example, CPU 102 maps gaze vectors to specific pixels on the HMD display. (Return to Reference) Figure 4 The focus pixel was determined to be pixel 406 in frame 400.

[0071] CPU 102 selects one or more pixels within the boundary of the 502C pixel of interest as the first group of pixels. The boundary can be defined by a shape centered on the pixel of interest. For example, the shape can be circular, and the boundary can be defined by a threshold number of pixels. (Return to Reference) Figure 4 The shape is a circle 408 centered on the point of interest pixel 406, and its boundary is defined by a threshold number of pixels. In this example, the threshold number of pixels is 1.5 pixels, but this can be any number of pixels. Optionally, the shape can be a square or any other shape. Any pixels that are at least partially within the boundary are selected as the first group of pixels.

[0072] Return to reference Figure 5A In the second embodiment shown, the CPU 102 selects 504 second group pixels from a plurality of pixels.

[0073] For example, CPU 102 selects one or more peripheral pixels as the second group of pixels. Peripheral pixels are pixels of the frame located outside the boundary. (Return to reference) Figure 4 The outermost pixels are all pixels that are at least partially outside the boundary 408.

[0074] The human eye is typically most sensitive to color in the central region of vision (called the fovea), and color sensitivity decreases in the peripheral regions. The first group of pixels is associated with this central visual region, while the second group is associated with the peripheral regions.

[0075] CPU 102 assigns a first set of 506 wavelength values ​​to each pixel in the first group of pixels.

[0076] CPU 102 assigns a second set of 508 wavelength values ​​to each pixel in the second set of pixels.

[0077] The first set of wavelength values ​​includes a greater number of wavelength values ​​than the second set of wavelength values. For example, the first set of wavelength values ​​includes all nine wavelength values ​​in the set of wavelength values. In some embodiments, fewer than all nine wavelength values ​​are used in the first set of wavelength values. In some embodiments, the second set of wavelength values ​​does not include wavelength values, such that no spectral rendering occurs at the second group of pixels.

[0078] As mentioned above, the first group of pixels is associated with the fovea, while the second group is associated with the peripheral region. Therefore, more wavelengths are allocated to the most color-sensitive visual regions compared to less sensitive visual regions. Sampling fewer wavelengths in the peripheral regions reduces the computational load required to render the scene without causing noticeable loss of detail.

[0079] In this second embodiment, a set of changing rendering parameter values ​​(i.e., a set of wavelength values) is implemented because a different set of rendering parameter values ​​is assigned to the first group of pixels compared to those assigned to the second group of pixels.

[0080] In an alternative implementation of the step 502 of selecting a first group of pixels from multiple pixels, the CPU 102 selects the first group of pixels by selecting multiple neighboring pixels that have corresponding spectral characteristics. That is, the CPU 102 selects multiple neighboring pixels that have spectral characteristics within a threshold similarity to each other. This threshold similarity can be determined using any known means, such as clustering algorithms.

[0081] For example, return to reference Figure 4 The CPU 102 determines that the first pixel 410 and the second pixel 412 (i.e., pixels among the plurality of pixels 402) are both associated with the object 404 (e.g., because their pixel positions fall within the object's boundary), and that pixels 410 and 412 therefore both have corresponding spectral characteristics. It should be understood that multiple adjacent pixels may not be associated with a specific object and can be associated with any feature of the scene (such as the sky), as long as these adjacent pixels have corresponding spectral characteristics.

[0082] Continuing with this alternative implementation, the set of rendering parameter values ​​includes a set of wavelengths. Assigning 506 rendering parameter values ​​to each pixel in the first group of pixels includes: identifying one or more wavelengths that provide a threshold contribution to each pixel; and selecting one or more of the wavelengths that provide a threshold contribution to each pixel in the first group of pixels as the first set of rendering parameter values.

[0083] Identifying one or more wavelengths that provide a threshold contribution to each pixel includes: identifying one or more wavelengths that reach at least one of the one or more light sources.

[0084] For example, using the first pixel 410 as an example, the GPU 106 performs initial sampling by simulating the paths of light rays of different wavelengths passing through the first pixel 410. These rays propagate through the scene and interact with the object 404 based on the spectral characteristics of the object 404. Based on these spectral characteristics, different wavelengths will be reflected or absorbed. The GPU 106 determines which rays (e.g., which secondary rays) reach at least one light source and identifies the wavelengths associated with these rays as those that provide a threshold contribution. These rays will be those not absorbed by the object 404 (e.g., because they are reflected). By performing this process, the GPU 106 identifies which wavelengths will actually contribute to the “appearance” of the scene for the first set of pixels 402 associated with the object 404.

[0085] Since the first group of pixels has corresponding spectral characteristics (i.e., because they are associated with the first object), the GPU can perform initial sampling on only one pixel in the first group (e.g., only the first pixel 410). In this way, the initial sampling step is computationally less expensive compared to initial sampling of multiple pixels.

[0086] Through this initial sampling, the GPU 106 can determine that the first group of pixels does not require certain wavelengths or wavelength ranges because they do not contribute to the "appearance" of the frame or scene, for example, because they are absorbed or scattered by objects, rendering them ineffective. Based on this information, the wavelengths of the first group of pixels can be selected, thereby removing other unnecessary wavelengths. Therefore, a reduced number of rays and wavelengths are sampled in subsequent rendering of the scene's pixels. Thus, fewer wavelengths are sampled to render the pixels in the first group of pixels with minimal loss of visual quality.

[0087] Optionally, identifying one or more wavelengths that provide a threshold contribution to each pixel includes: obtaining an object spectrum associated with a first object; obtaining a light source spectrum associated with each of one or more light sources; and identifying one or more wavelengths present in both the object spectrum and at least one light source spectrum.

[0088] CPU 102 can obtain the object spectrum from the spectral texture data stored in the graphics data storage 108. The object spectrum describes one or more wavelengths reflected (i.e., not absorbed) by the object. CPU 102 can obtain the light source spectrum from the lighting data stored in the graphics data storage 108. The light source spectrum describes the spectrum of wavelengths emitted by the light source.

[0089] CPU 102 identifies one or more wavelengths present in both the object spectrum and the spectrum of at least one light source by comparing the wavelengths reflected by the object with the spectrum of the wavelengths emitted by each light source.

[0090] In some embodiments, the object spectrum includes wavelengths that satisfy a threshold reflection intensity, and the light source spectrum includes wavelengths that satisfy a threshold emission intensity. Specifically, the CPU 102 compares wavelengths that satisfy the threshold reflection intensity reflected by the object with wavelengths that satisfy the threshold emission intensity emitted by each light source. The threshold reflection intensity and the threshold emission intensity can be the same or different values. Some light sources and objects contain only a very small number of certain wavelengths. For example, the emission spectrum of a red light source will have a negligible intensity toward the blue end of the visible spectrum. In this embodiment, the threshold intensity can be selected such that blue wavelengths do not satisfy the threshold intensity because blue wavelengths do not have sufficiently high intensity. An object with a monochromatic red surface will reflect only a negligible portion of the blue light incident upon it. Similarly, in this embodiment, the threshold intensity can be selected such that reflected blue wavelengths do not satisfy the threshold intensity because blue wavelengths do not reflect with sufficiently high intensity. Therefore, in this implementation, the CPU 102 identifies one or more wavelengths present in both the object spectrum and at least one light source spectrum by comparing the wavelengths reflected by the object (i.e., red wavelengths satisfying the threshold intensity) with the spectrum of wavelengths emitted by each light source (i.e., red wavelengths satisfying the threshold intensity). Therefore, fewer wavelengths are sampled, thus reducing the computational load required to render the scene.

[0091] This implementation limits sampling to wavelengths where there is overlap between the light source spectrum and the object spectrum. For example, in a scene involving a white light source emitting light toward a red object, sampling in the blue portion of the spectrum would have no perceptible effect on the appearance of the surface because it reflects the least amount of blue light. Therefore, limiting the range of wavelengths where sampling occurs (and accordingly using fewer samples) achieves substantially similar visual effects while rendering fewer samples, thus saving resources.

[0092] This allows for changing rendering parameter values ​​because different sets of pixels will have different wavelengths that contribute to the threshold (e.g., because they are associated with objects that have different spectral properties).

[0093] One or more of the foregoing embodiments can be combined. For example, the second embodiment can be combined with the first embodiment, such that each pixel in the first group of pixels is assigned a corresponding first subset of rendering parameter values ​​and a corresponding second subset of rendering parameter values.

[0094] Step 206 includes performing spectral rendering using the modified rendering parameter values ​​to generate frames of the scene.

[0095] For example, performing spectral rendering 206 to generate frames of the scene using modified rendering parameter values ​​includes the following steps. CPU 102 defines one or more objects (based on geometric and texture data), light sources (based on lighting data), and camera positions (based on viewpoint data) based on graphics data stored on graphics data storage 108. CPU 102 generates a master ray from the camera position for each pixel. The master ray includes rendering parameter values ​​(e.g., wavelength values) assigned to each pixel, as determined in step 204. GPU 106 determines which rays intersect with the scene geometry and which points intersect. GPU 106 uses texture data stored in graphics data storage 108 to determine the spectral radiance values ​​at the intersection points, for example, using memory access operations. The spectral radiance values ​​describe how the object material interacts with the wavelengths of the intersecting rays. GPU 106 determines the direction and intensity of reflected and refracted rays (master and secondary rays) based on the spectral radiance values. GPU 106 integrates the contributions from all scene wavelengths and generates a color value for each pixel as rendered image data. The GPU 106 stores the pixel color values ​​in the frame buffer 110 and then transmits the frame buffer data to the display 114, so that the display 114 displays the scene.

[0096] Regarding the first implementation, performing 206-spectrum rendering to generate scene frames using the modified rendering parameter values ​​includes performing 206-spectrum rendering to generate a first frame and performing 206-spectrum rendering to generate a second frame. Specifically, GPU 106 samples each pixel in a first group of pixels using a corresponding subset of the rendering parameter values ​​to generate the first frame, and samples each pixel in the first group of pixels using a corresponding second subset of the rendering parameter values ​​to generate the second frame.

[0097] Figure 6 An alternative implementation 600 of the computer-implemented method 200 of the present invention is shown.

[0098] The method 600 includes obtaining 602 a set of rendering parameter values ​​for ray tracing of the scene. Step 602 is essentially similar to step 202. For the sake of brevity, it will not be repeated here.

[0099] In method 600, the step of changing the set of rendering parameter values ​​across multiple pixels 204 is replaced by a step of identifying one or more rendering parameter values ​​that provide a threshold contribution to the scene 604.

[0100] Identifying one or more wavelengths that contribute a threshold to the scene includes: identifying one or more wavelengths that reach at least one of the one or more light sources.

[0101] For example, using first pixel 410 as an example, GPU 106 performs initial sampling by simulating the paths of light rays of different wavelengths passing through first pixel 410. These rays propagate through the scene and interact with object 404 based on the spectral characteristics of object 404. Based on these spectral characteristics, different wavelengths will be reflected or absorbed. GPU 106 determines which rays (e.g., which secondary rays) reach at least one light source and identifies the wavelengths associated with these rays as those that provide a threshold contribution. These rays will be those not absorbed by object 404 (e.g., because they are reflected). By performing this process, GPU 106 identifies which wavelengths will actually contribute to the “look” of the scene. GPU 106 can then limit this initial sampling to pixels associated with objects. In this way, initial sampling is performed only on pixels of “interest.”

[0102] Through this initial sampling, the GPU 106 can determine that certain wavelengths or wavelength ranges are unnecessary for the scene because they do not contribute to the scene, for example, because they are absorbed or scattered by objects, rendering them ineffective. Based on this information, the scene's wavelengths can be selected, thereby removing other unwanted wavelengths. Therefore, a reduced number of rays and wavelengths are sampled in subsequent rendering of the frame's pixels. Thus, fewer wavelengths are sampled to render the frame with minimal loss of visual quality.

[0103] Optionally, identifying one or more wavelengths that provide a threshold contribution to each pixel includes: obtaining an object spectrum associated with each of the one or more objects; obtaining a light source spectrum associated with each of the one or more light sources; and identifying one or more wavelengths present in at least one object spectrum and at least one light source spectrum.

[0104] CPU 102 can obtain the object spectrum from the spectral texture data stored in the graphics data storage 108. The object spectrum includes one or more wavelengths reflected by the object (i.e., not absorbed). CPU 102 can obtain the light source spectrum from the lighting data stored in the graphics data storage 108. The light source spectrum describes the spectrum of wavelengths emitted by the light source.

[0105] CPU 102 identifies one or more wavelengths present in at least one object spectrum and at least one light source spectrum by comparing the wavelengths reflected by the object with the spectrum of wavelengths emitted by each light source.

[0106] Some light sources and objects contain only very small amounts of certain wavelengths. For example, the emission spectrum of a red light source will have a negligible intensity toward the blue end of the visible spectrum. An object with a monochromatic red surface will reflect only a negligible portion of the blue light incident upon it.

[0107] This implementation limits sampling to wavelengths where there is overlap between the light source and object spectra. For example, in a scene involving a white light source emitting light towards multiple red objects, sampling in the blue portion of the spectrum would have no perceptible effect on the appearance of the surface because it reflects the least amount of blue light. Therefore, limiting the range of wavelengths where sampling occurs (and accordingly using fewer samples) achieves essentially the same visual effect while rendering fewer samples, thus saving resources.

[0108] Method 600 finally includes performing 606 spectral rendering to generate frames of the scene using rendering parameter values ​​that contribute a threshold to the scene.

[0109] Step 606 is essentially similar to step 206, except that the master ray generated from the camera position of each pixel is the wavelength that provides a threshold contribution to the scene.

[0110] The difference between Method 600 and Method 200 is that all pixels are sampled using the same set of wavelengths. Since this set of wavelengths is the wavelength that contributes to the threshold of the scene, fewer wavelengths are sampled compared to using the full set of rendering parameter values.

Claims

1. A computer-implemented method for rendering graphics data representing a 3D scene on a video game system, the 3D scene comprising one or more objects and one or more light sources, the method comprising: obtaining a set of rendering parameter values for ray tracing the scene; varying the set of rendering parameter values across a plurality of pixels of a frame of the scene; and performing spectral rendering using the varied rendering parameter values to generate the frame of the scene. The set of rendering parameter values is a set of wavelengths for ray tracing the scene.

2. The computer-implemented method of claim 1, wherein, Varying the set of rendering parameter values across the plurality of pixels comprises:

3. The computer-implemented method of claim 1 or claim 2, wherein, selecting a first group of pixels from the plurality of pixels; and assigning each pixel in the first group of pixels a respective subset of rendering parameter values; wherein each pixel in the first group of pixels is assigned a different subset of rendering parameter values. There are no common rendering parameter values between each subset of rendering parameter values.

4. The computer-implemented method of claim 3, wherein, Selecting the first group of pixels from the plurality of pixels comprises:

5. The computer-implemented method of claim 3 or claim 4, wherein, selecting an array of pixels from the plurality of pixels. The number of pixels in the first group of pixels is equal to the number of rendering parameter values in the set of rendering parameter values.

6. The computer-implemented method of any one of claims 3 to 5, wherein, 7. The computer-implemented method of any one of claims 3 to 6, further comprising: assigning each pixel in the first group of pixels a respective second subset of rendering parameter values; and performing spectral rendering using the second subset of rendering parameter values to generate a second frame of the scene; wherein each pixel in the first group of pixels is assigned a different second subset of rendering parameter values. Varying the set of rendering parameter values across the plurality of pixels of the frame of the scene comprises: selecting a first group of pixels from the plurality of pixels; 8. The computer-implemented method of claim 1 or claim 2, wherein, selecting a second group of pixels from the plurality of pixels; assigning each pixel in the first group of pixels a first set of rendering parameter values; and assigning each pixel in the second group of pixels a second set of rendering parameter values. Selecting the first group of pixels from the plurality of pixels comprises: identifying a point of interest of a user in the 3D scene; 9. The computer-implemented method of claim 8, wherein, determining a point of interest pixel, wherein the point of interest pixel is a pixel of a frame corresponding to the point of interest in the 3D scene; and selecting one or more pixels within a boundary of the point of interest pixel as the first group of pixels. The boundary is a shape centered on the point of interest pixel. The shape is circular and the boundary is defined by a threshold number of pixels.

10. The computer-implemented method of claim 9, wherein, Selecting the second group of pixels comprises:

11. The computer-implemented method of claim 10, wherein, selecting one or more peripheral pixels as the second group of pixels; 12. The computer-implemented method of any one of claims 9 to 11, wherein, wherein the peripheral pixels are pixels of the frame that lie outside the boundary. The first set of rendering parameter values comprises a greater number of rendering parameter values than the second set of rendering parameter values. Selecting the first group of pixels from the plurality of pixels comprises:

13. The computer-implemented method of any one of claims 9 to 12, wherein, selecting a plurality of adjacent pixels having a corresponding spectral characteristic.

14. The computer-implemented method of claim 8, wherein, Selecting the plurality of adjacent pixels having a corresponding spectral characteristic comprises: selecting a plurality of pixels associated with a first object of the 3D scene.

15. The computer-implemented method of claim 14, wherein, ​ ​ 16. The computer-implemented method of claim 15, wherein, The set of rendering parameter values includes a set of wavelengths, and assigning the first set of rendering parameter values to each pixel in the first group of pixels includes: identifying one or more wavelengths that provide a threshold contribution to the first group of pixels; and selecting the one or more wavelengths that provide the threshold contribution to the first group of pixels as the first set of rendering parameter values.

17. The computer-implemented method of claim 16, wherein, Identifying the one or more wavelengths that provide the threshold contribution to the first group of pixels includes: identifying one or more wavelengths that reach at least one of the one or more light sources.

18. The computer-implemented method of claim 16, wherein, Identifying the one or more wavelengths that provide the threshold contribution to the first group of pixels includes: obtaining an object spectrum associated with the first object; obtaining a light source spectrum associated with each of the one or more light sources; and identifying one or more wavelengths that exist in both the object spectrum and at least one light source spectrum.

19. The computer-implemented method of claim 18, wherein, The object spectrum includes wavelengths that satisfy a threshold reflectance intensity, and the light source spectrum includes wavelengths that satisfy a threshold emission intensity.

20. A non-transitory storage medium comprising instructions that, when executed by a processor, cause the processor to perform the method of any preceding claim.

21. A computer program comprising instructions that, when executed by a processor, cause the processor to perform the method of any of claims 1 to 19.

22. A video game system for rendering graphical data of a 3D scene, the system comprising a processor and a GPU, and configured to perform the method of any of claims 1 to 19.

Images