Pixel filter, method for filtering a sequence of pixels

Abstract

Pixel filter, method for filtering a sequence of pixels. A pixel filter includes: an input configured to receive a sequence of pixels, each pixel having an associated pixel value; a filter module configured to perform a first recursive filtering operation through the sequence of pixels in a first direction to form a first filtered pixel value for each pixel, and to perform a second recursive filtering operation through the sequence of pixels in a second direction to form a second filtered pixel value for each pixel, wherein the first and second recursive filtering operations form the respective filtered pixel values for a pixel based on the pixel value at the given pixel and the filtered pixel values of the pixels preceding it in their respective operation directions, and the filtered pixel values of the preceding pixels are adjusted by a similarity measure between the data associated with the pixel and its preceding pixels; and filter logic configured to, for each pixel in the sequence, combine the first and second filtered pixel values formed by the first and second recursive filtering operations with respect to the pixel to generate a filter output for the pixel.

Description

[0001] This application is a divisional application of the invention patent application with the application number 201610621289.8, the application date of August 1, 2016, and the invention title of "Pixel Filter, Method for Filtering a Sequence of Pixels". Technical Field

[0002] The present invention relates to filters and methods for filtering pixels of a frame in a graphics processing system, and more particularly, to filters and methods for filtering pixels in a frame rendered by path or ray tracing. Background Art

[0003] Several techniques can be used to render a three-dimensional (3D) scene for display on a computer system. These techniques include scanline rendering of a three-dimensionally described scene in a computer system into two dimensions, path tracing, and ray tracing. Path tracing forms an image by determining the net colour of each pixel of a frame defined at a particular viewpoint in a 3D scene. The net colour is calculated by integrating the contributions of all the light received by the pixel, including light reflected (potentially multiple times) by surfaces in the scene. By defining a physically accurate model of the surfaces and light sources in the scene, including the reflection characteristics of the surfaces, very realistic images can be rendered. Ray tracing forms an image in a similar way, but typically for each point in the scene that contributes to the rendered image, the colour of that point is determined only based on the light received from the light sources in the scene at that point (i.e., only direct light). Many variations of path and ray tracing are possible, including hybrid techniques that use path or ray tracing to determine the lighting of a 3D scene to be applied to traditional texture mapping.

[0004] To significantly reduce the computational amount required to render an image using path or ray tracing techniques, rays are typically traced backward from the viewpoint towards the light source. This avoids the overhead of performing calculations for rays that cannot be seen at the viewpoint where the output image is captured. The number of rays traced through the scene for each pixel in the output image is commonly referred to as the number of samples per pixel or SPP.

[0005] An illustration of a frame 12 rendered by path tracing is shown in Figure 1 as a floor plan of a 3D scene 100. In the scene, a nominal camera 101 looks down a corridor 102, which includes a window 106 through which light from a light source (in this case the sun 103) enters. The viewpoint of the camera is to be rendered as a frame 112, and the colour of each pixel of this frame is determined by the totality of the light rays passing through that pixel. The corridor additionally includes columns 104 along the wall 117 and a picture 105 on the wall 118.

[0006] The scene can be a 3D scene defined in a computer game, and the camera 101 represents the protagonist, playing the game from the protagonist's viewpoint. In this example, the scene is defined at the computer system by a 3D model (a set of polygons that define the scene geometry) and texture information that can be applied to the 3D model. The 3D model defines the position and contour of the surfaces in the scene, and each texture defines the characteristics of the surface to which it is applied. For example, a texture will typically represent the characteristics of the appearance of the surface in terms of surface color and composition, and can also determine other characteristics (such as the reflectivity of the surface) and the degree to which the reflection from the surface is specular or diffuse.

[0007] Figure 1 Three exemplary rays 108 - 110 are shown, each ray originating from a light source that passes through the window 106 and reflects off one or more surfaces of the scene before reaching the camera 101. However, each ray 108 - 110 is identified by tracing the ray backward from the camera 101 through the pixels of the frame 112 where the image of the scene is to be formed. Ray 108 passes through pixel 114 and reflects off the painting 105 that gives the ray color and causes a diffuse reflection 127. Then, ray 108 encounters the wall 117 with a low reflectivity (resulting in a loss of luminance) and gray color, and causes a diffuse reflection 128. The diffuse nature of the reflections 127 and 128 can be modeled by randomly (usually according to the Lambert cosine law) picking the reflection angles 121 and 122. In the case of reflection 128, this causes the ray to be directed out of the window 106. By considering the amount of light traveling along the path from the light source to the camera, the contribution from ray 108 to pixel 114 can be determined: the substantial white light of a given intensity passing through the window 106 becomes dark gray as a result of reflection 128 and is then reflected by the painting 105 with a color change.

[0008] Rays 109 and 110 each undergo a single reflection. Ray 109 passes through pixel 115 and is traced back to encounter the column 104 of polished pink marble with a higher reflectivity, giving the ray a pink color and causing a reflection 125. In contrast, ray 110 passes through pixel 116 and is traced back to encounter the wall 117, so it is largely absorbed, given a gray color, and undergoes a diffuse reflection. By reversing the paths of rays 109 and 110 and applying the characteristics of reflections 125 and 126 to the rays entering through the window 106, the contributions of these rays to pixels 115 and 116 can be determined. Typically, rays will be traced back from the camera only up to some predetermined number of reflections; if the light source is not encountered within that number of reflections, the ray can be discarded.

[0009] Through a path tracing process that repeats multiple rays through a pixel (each ray passing through the pixel with a slightly different (possibly randomly selected) orientation), the color of the pixel can be established by calculating the average of the results from all samples (e.g., according to Monte Carlo integration). By performing this process for each pixel of frame 112, a complete frame can be generated. After being generated in this way, the frame includes lighting information generated by the scene geometry, the positions and characteristics of light sources in the scene, and texture information and surface characteristics of the surfaces present in the scene.

[0010] When a smaller number of rays contribute to each pixel and the samples per pixel (SPP) is low, the noise level in the rendered frame may be high. Increasing the number of samples per pixel improves the accuracy of the rendered frame and reduces the noise level in the frame. However, increasing the number of samples per pixel significantly increases the time required to render the frame. Summary of the Invention

[0011] According to a first aspect of the present invention, there is provided a pixel filter, the pixel filter comprising:

[0012] an input configured to receive a sequence of pixels, each pixel having an associated pixel value;

[0013] a filter module configured to perform a first recursive filtering operation along a first direction through the sequence of pixels to form a first filtered pixel value for each pixel, and to perform a second recursive filtering operation along a second direction through the sequence of pixels to form a second filtered pixel value for each pixel, wherein the first recursive filtering operation and the second recursive filtering operation form the respective filtered pixel values for the pixel based on the pixel value at a given pixel and the filtered pixel values of the pixels before it along each operation direction, and the filtered pixel values of the previous pixels are adjusted by a similarity measure between the data associated with the pixel and its previous pixels; and

[0014] filter logic configured to, for each pixel in the sequence, combine the first filtered pixel value and the second filtered pixel value formed by the first recursive filtering operation and the second recursive filtering operation with respect to the pixel to generate a filter output for the pixel.

[0015] The second direction may be opposite to the first direction.

[0016] The first recursive filtering operation and the second recursive filtering operation may be configured to adjust the filtered pixel values of the previous pixels such that the contribution of the pixel value to the filtered pixel value is greater when the similarity measure indicates high similarity than when the similarity measure indicates low similarity.

[0017] The first recursive filtering operation and the second recursive filtering operation can be configured to form, for each pixel, respective first filtered pixel values and second filtered pixel values for each pixel by determining the difference between the following:

[0018] A first intermediate value representing the output of each recursive filtering operation when adjusting the respective filtered pixel values of the pixels before the first parameter adjustment sequence; and

[0019] A second intermediate value representing the output of each recursive filtering operation when adjusting the respective filtered pixel values of the pixels before the second parameter adjustment sequence.

[0020] The difference between the first parameter and the second parameter can be at least one order of magnitude smaller than at least one of the sizes of the first parameter and the second parameter.

[0021] The input can also be set to receive an additional pixel sequence, the additional pixel sequence including at least one pixel value from the output of the filter, and the filter module is also set to perform the first recursive filtering operation and the second recursive filtering operation on the additional pixel sequence.

[0022] The pixel sequence can represent a row of pixels selected from a rectangular pixel array, and the additional pixel sequence represents a column of pixels selected from the rectangular pixel array.

[0023] The pixel sequence can represent a column of pixels selected from a rectangular pixel array, and the additional pixel sequence represents a row of pixels selected from the rectangular pixel array.

[0024] The filter module can be configured to weight the pixel values by a confidence measure of each pixel value when forming the filtered pixel values of the pixels in the sequence.

[0025] Each pixel value and the measure of confidence in that pixel value can represent a set of coordinates, and the filter module can be configured to operate on the components of each set of coordinates simultaneously to generate a filtered pixel value and a filtered confidence measure.

[0026] For each pixel of the sequence, the first recursive filtering operation and the second recursive filtering operation can also form respective first filtered confidence measures and second filtered confidence measures of the pixel according to the confidence measure of the pixel value and according to the filtered confidence measure formed by the previous pixel with respect to the operation direction of the recursive filter for the sequence, the filtered confidence measure of the previous pixel being adjusted by a similarity measure; wherein, the first recursive filtering operation is adapted to form the first filtered pixel value of each pixel according to the first filtered confidence measure of the pixel, and the second recursive filtering operation is adapted to form the second filtered pixel value of each pixel according to the second filtered confidence measure of the pixel.

[0027] The filter logic can be configured to combine the first filtered pixel value and the second filtered pixel value of each pixel by calculating a weighted average of the first filtered pixel value and the second filtered pixel value, and the weights of the corresponding filtered pixel values are formed according to the confidence measure of the pixel value contributing to the filtered pixel value.

[0028] The pixel value can be generated by path or ray tracing, and the confidence measure of each pixel value before filtering represents the number of rays contributing to the pixel value.

[0029] The filter module can include a recursive filter block configured to perform both the first recursive filtering operation and the second recursive filtering operation, such that the first recursive filtering operation is performed when passing through the recursive filter block for the first time, and the second recursive filtering operation is performed when passing through the recursive filter block for the second time.

[0030] The filter module can further include a row storage configured to store the first filtered pixel value of each pixel, and wherein the filter logic is further configured to combine the second filtered pixel value output from the recursive filter block with the first filtered pixel value stored in the row storage.

[0031] The filter module can include a first recursive filter block and a second recursive filter block configured to operate simultaneously, and wherein the first recursive filter block is configured to perform the first recursive filtering operation, and the second recursive filter block is configured to perform the second recursive filtering operation.

[0032] The pixel filter can further include a pre-filter adapted to filter the pixel value to form a pre-filtered pixel value of each pixel, and wherein the data associated with the pixel and the pixel before it respectively includes the pre-filtered pixel value at the pixel and the pre-filtered pixel value at the previous pixel regarding the operation direction of each recursive filtering operation.

[0033] The pre-filter can have a filtering radius that is at least one order of magnitude smaller than the sizes of the first recursive filtering operation and the second recursive filtering operation.

[0034] The pre-filter can be adapted to form the pre-filtered pixel value at the pixel by forming a weighted average of the pixel value and a predetermined group of adjacent pixel values, and the pixel value has a higher weight than the adjacent pixel value group.

[0035] The recursive filter can have an infinite filtering radius.

[0036] The recursive filter can be an exponential filter.

[0037] Each recursive filter can include a recursive filter of a pre-filter, which is configured to form pre-filtered pixel values used at the recursive filter.

[0038] The pixel can represent a raster image rendered to represent a three-dimensional scene, and wherein the data associated with the pixel and the pixel before it includes one or more of the following: pixel characteristics; and scene characteristics of the three-dimensional scene at the point represented by the pixel and the previous pixel in the image.

[0039] The raster image can represent the lighting information of the scene, but not the texture information.

[0040] The scene characteristics can be one or more of the following: a vector describing the scene geometry, a normal vector describing the orientation of the surface in the scene, and a scene depth representing the distance measure of the surface in the scene from a defined reference point, line, or plane.

[0041] The similarity measure can include at least one of the following: a measure of the angle between the normal vectors of the surfaces in the scene represented by the pixel and the previous pixel; and a measure of the difference in scene depth between the surfaces in the scene represented by the pixel and the previous pixel.

[0042] The pixel characteristics can be one or more of color, color component, chromaticity, color saturation, luminance, lightness, brightness, and transparency.

[0043] Each pixel value can represent luminance or color, and the similarity measure between the data associated with the pixel and its previous pixel can be a distance measure between the luminance or color in the color space represented by the pixel and the previous pixel.

[0044] The recursive filter can be configured such that each pixel and its previous pixel are adjacent in the pixel sequence.

[0045] Each pixel sequence can represent a sequence of consecutive pixels.

[0046] One or more recursive filtering operations can be one or more of the following: an instance of a software recursive filter; different filters implemented in hardware; and appropriately configured computing units of a GPU.

[0047] Each pixel value can include one or more color components and / or luminance measures.

[0048] According to a second aspect of the present invention, there is provided a method for filtering a pixel sequence, each pixel having an associated pixel value, the method comprising the steps of:

[0049] For each pixel, recursively filter the pixel sequence along a first direction and a second direction to form a first filtered pixel value and a second filtered pixel value for each pixel by:

[0050] Read out the filtered pixel value of the previous pixel of the sequence with respect to the operation direction of the recursive filter;

[0051] Adjust the filtered pixel value of the previous pixel of the sequence by a similarity measure between the data associated with the pixel and the previous pixel; and

[0052] Form the filtered pixel value of the pixel according to the pixel value of the pixel and according to the adjusted filtered pixel value of the previous pixel;

[0053] And

[0054] Combine the first filtered pixel value and the second filtered pixel value formed for each pixel of each sequence to generate the filter output of the pixel.

[0055] The second direction may be opposite to the first direction.

[0056] Adjusting the filtered pixel value of the previous pixel may cause the contribution of the pixel value to the filtered pixel value of the pixel to be greater when the similarity measure indicates high similarity than when the similarity measure indicates low similarity.

[0057] Adjusting the filtered pixel value of the previous pixel may include:

[0058] Adjust the previous pixel by a first parameter; and

[0059] Adjust the previous pixel by a second parameter;

[0060] And, forming the filtered pixel value of the pixel includes:

[0061] Form a first intermediate value using the previous pixel adjusted by the first parameter;

[0062] Form a second intermediate value using the previous pixel adjusted by the second parameter; and

[0063] Form the filtered pixel value of the pixel by determining the difference between the first intermediate value and the second intermediate value.

[0064] The difference between the first parameter and the second parameter may be at least one order of magnitude smaller than the size of either the first parameter or the second parameter.

[0065] The method may further include the following steps:

[0066] Receiving an additional pixel sequence, the additional pixel sequence including at least one filter output; and

[0067] Filtering the additional pixel sequence according to the method described herein to form a filter output for each pixel of the additional sequence.

[0068] The pixel sequence may represent a row of pixels selected from a rectangular pixel array, and the additional pixel sequence represents a column of pixels selected from the rectangular pixel array.

[0069] The pixel sequence may represent a column of pixels selected from a rectangular pixel array, and the additional pixel sequence represents a row of pixels selected from the rectangular pixel array.

[0070] Forming a filtered pixel value for each pixel of the sequence may include weighting the pixel value by a confidence measure of the pixel value.

[0071] Each pixel value and the measure of confidence in that pixel value may represent a set of coordinates, and the filter module may be configured to operate on the components of each set of coordinates simultaneously to generate a filtered pixel value and a filtered confidence measure.

[0072] Recursive filtering may include: for each pixel of the sequence, forming a first filtered confidence measure and a second filtered confidence measure for the pixel according to the confidence measure of the pixel value and according to the filtered confidence measure formed for a previous pixel in the sequence with respect to the direction of operation of the recursive filter, the filtered confidence measure of the previous pixel being adjusted by a similarity measure; wherein, a first filtered pixel value and a second filtered pixel value for each pixel are formed according to the first filtered confidence measure and the second filtered confidence measure of the pixel, respectively.

[0073] The method may further include the following steps: forming a weight for the filtered pixel value according to the confidence measure of the pixel value contributing to the corresponding filtered pixel value, and combining the first filtered pixel value and the second filtered pixel value may include calculating a weighted average of the first filtered pixel value and the second filtered pixel value.

[0074] The pixel values may be generated by path or ray tracing, and the confidence measure of each pixel value before filtering represents the number of rays contributing to that pixel value.

[0075] The method may further include the steps of pre-filtering the pixels to form pre-filtered pixel values for each pixel, and wherein the data associated with the pixel and the pixel preceding the pixel respectively includes the pre-filtered pixel value at the pixel and the pre-filtered pixel value at the preceding pixel regarding the operation direction of each recursive filtering operation.

[0076] The pre-filtering of the pixels may be performed with a filter radius that is at least one order of magnitude smaller than the size of the filter radius used for recursive filtering.

[0077] The pre-filtering of the pixels may be performed by forming a weighted average of the pixel value and a set of predetermined pixel values adjacent to the pixel to form the pre-filtered pixel value at the pixel, and the pixel value has a higher weight than the set of adjacent pixel values.

[0078] The recursive filtering may be performed with an infinite filter radius.

[0079] Computer program code may be provided for defining a filter as described herein, whereby a filter may be manufactured. A permanent computer-readable storage medium may be provided, on which computer-readable instructions are stored, which, when executed at a computer system for generating a display of an integrated circuit, cause the computer system to generate a display of a filter as described herein.

[0080] Computer program code may be provided for performing a method for filtering pixels of a frame in a graphics processing system. A permanent computer-readable storage medium may be provided, on which computer-readable instructions are stored, which, when executed at a processor, cause the processor to perform a method for filtering pixels of a frame in a graphics processing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0082] Figure 1 A plan view illustrating a three-dimensional scene 100 and the path of light passing through the scene is shown.

[0083] Figure 2 is a raster illumination map of the scene 100 as viewed from the viewpoint of the camera 101.

[0084] Figures 3a - 3g Shows an illustration of Figure 2 a graph of the filtering of pixel values between points 202 and 203 in

[0085] Figure 4a is a schematic diagram of a filter 400 according to a first example.

[0086] Figure 4b Is a schematic diagram of a filter 400 according to a second example.

[0087] Figure 5 Is a schematic diagram of a graphics system including the filter 400.

[0088] Figure 6 Illustrates filtering of pixels of a frame.

[0089] Figure 7 Illustrates the filtering performed by the filter 400.

[0090] Figure 8 Shows an exemplary structure of data and functional elements of the filter 400 operating at a graphics system.

[0091] Figure 9 Is a flowchart illustrating the operation of the filter 400. Detailed Description

[0092] The following description is presented by way of examples that enable any person skilled in the art to make and use the invention. The invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be readily apparent to those skilled in the art.

[0093] Filters and methods for filtering pixels of a frame in a graphics processing system are provided, specifically filters and methods for filtering out noise gradually generated by path / ray tracing techniques while maintaining an acceptable image quality. The methods described herein are particularly suitable for filtering noise in raster illumination maps.

[0094] Noise in the rendered image can be removed by using conventional filters such as bilateral filters. Such filters can provide a good quality image but are computationally expensive. Path and ray tracing themselves are computationally complex, and even in the case of a hybrid approach where path / ray tracing is used to provide illumination information for a rasterized scene, real-time implementation can be challenging. Using complex filters to achieve an acceptable image quality exacerbates the problem and makes it difficult to provide a real-time path tracing solution, especially on mobile platforms with limited processing resources.

[0095] Now the filter will be described with reference to Figure 5 the exemplary graphics system 500 shown in. The system renders an output frame 509. The system can include, for example, a graphics processing unit (GPU) suitable for supporting a rendering engine for computer games. More generally, the filter can be used in any kind of image processing system and is configured to perform filtering of any kind of raster graphics.

[0096] Graphics system

[0097] The graphics system 500 is configured to independently generate lighting information for a 3D scene ( Figure 5 data path 507 in), and form a raster image of the scene without any lighting information (data path 508). The system includes a memory 506 that holds a data set representing the scene geometry 511; light sources in the scene 510 (e.g., the positions and characteristics of the light sources such as the color of the light sources); the texture of the surfaces 512 defined by the scene geometry; and a camera 513 (e.g., the position of the nominal camera in the scene, from whose viewpoint the output frame 509 is generated). The memory 506 is Figure 5 schematically shown in as in fact being able to include one or more memories that may be supported at the graphics system and / or external to the system (e.g., at the main system in which the system 500 is set up). For example, the system 500 may be a suitably programmed GPU supported at a main computer system, the texture data 512 being held in the memory of the main system and partially held at the texture buffer of the GPU.

[0098] The texture data path 508 includes a texture mapper 503 and a rasterizer 504. The texture mapper performs the mapping of the texture 512 onto the surfaces defined by the scene geometry 511. This generates the mapped texture data of the scene that describes the texture at each sampling point on at least the visible surfaces of the scene. The rasterizer 504 generates a raster image of the texture-mapped scene from the viewpoint of the camera 513. In other words, the rasterizer projects the surfaces of the scene together with the texture applied to the surfaces onto a raster image corresponding to the output frame 509 to generate a representation of the scene from the viewpoint of the camera 513. The raster image will typically have a predefined pixel resolution that is equal to or greater than the pixel resolution of the desired output frame 509.

[0099] The lighting data path 507 includes a path tracer 501 and a filter 400, and the operations of the path tracer 501 and the filter 400 are described below. The path tracer may be configured to determine the lighting of the scene by tracing the paths of light rays from the camera 513 through the scene towards the light sources 510 present in the scene. More generally, the path tracer may be configured to trace light N steps from the light source and / or M steps from the camera, each step being a segment of the optical path extending between points of interaction in the scene (e.g., reflections from the surfaces defined in the scene) from the source or the camera. If the path tracer traces in both directions (from the camera and the source), the path tracer may continue until the paths can be joined to create a complete optical path from the source to the camera. The path tracing performed in different directions may be filtered after combination, or may be filtered independently and then combined.

[0100] Path tracing uses the scene geometry 511 and texture information determined for each reflection point in the scene from the mapped texture data generated by the texture mapper 503. The path tracer 501 is configured to compute a path through the scene from the camera to the light source in the same manner as described for the conventional path tracing method described above Figure 1 However, the path tracer 501 does not texture the light ray at the point of the last reflection before reaching the camera. Instead, the path tracer 501 generates a light map for the output frame based on the light ray before the last reflection, such that each pixel of the light map represents a color to be fused with the raster image generated by the rasterizer 504. The light map can include information representing both the intensity and color of light in any suitable format (e.g., represented in RGB or YUV), or can include information representing only the intensity of light (e.g., represented by the Y value). The spatial resolution and / or the number of bits used to represent each value can vary between channels. For example, RGB 5:6:5 means using five bits for each of the red and blue channels and six bits for the green channel. In another example, the YUV representation can use a higher spatial resolution for the Y channel than for the U and V channels.

[0101] The operation of the path tracer 501 will now be described in more detail with reference to Figure 1 We first consider the light ray 110. To determine the contribution to the pixel 116 in the frame 112, the path tracer traces the light ray 110 through the pixel 116 from the camera 101 and identifies the surface of the wall 117 where the light ray 110 meets at the point 126. By looking up the material information of the point 126 in the information defining the characteristics of the surface in the scene, the path tracer determines that the light ray undergoes diffuse reflection.

[0102] In traditional graphics rendering, it is common for textures to represent the appearance characteristics of a surface in terms of surface color and composition. The texture data 512 can be extended to provide additional material information (e.g., the reflectivity or reflection characteristics of the surface). The additional material information can be held in the memory 506 together with or separately from the texture data 512 (e.g., elsewhere). The process of determining the material information of the surface at the point where the light ray intersects the surface is similar to the process of determining the color information of the visible portion of the surface in traditional rasterization. Thus, the texture mapper 503 can be used to determine the material information of the point 126 based on the texture 512. Then, the material information can be provided to the path tracer 501 as part of the mapped texture data.

[0103] Accordingly, the path tracer randomly selects a reflection angle according to Lambertian reflection. In this case, the reflection angle happens to take the light out of window 106. Thus, the path tracer determines that the camera will see the texture at point 126 directly illuminated by the light entering through window 106 at pixel 116 of frame 112. Unlike in a traditional path tracer system, the path tracer 501 does not apply the texture color identified at point 126 to the characteristics of the light received through the window. Instead, the contribution of ray 110 to pixel 116 of frame 112 is given by the color of the light incident at point 126. Thus, in this example, frame 112 represents a raster illumination map of the scene as viewed from the viewpoint of camera 101, not a complete image of the scene.

[0104] Similarly, by tracing a ray through pixel 115 of frame 112, it is determined that the camera will see the texture at point 125 directly illuminated by the ray from window 106. Thus, the contribution of ray 109 to pixel 115 of frame 112 is given by the color of the light incident at point 125.

[0105] It can be seen that the ray 108 traced from the camera through pixel 114 is reflected twice before encountering the light source 103. The contribution of ray 108 to pixel 114 of frame 112 is the color of the light incident at point 127. This is not the color of the light from light source 103, but rather the color of the light from light source 103 modified by a full reflection at 128, where the color of the ray is combined with the color of the wall at point 128. This combination can be performed in any suitable manner depending on the color space defining the colors and the lighting rules defined for the composite environment of the scene. Since wall 117 is dark gray, the ray from the ray 108 incident at 127 is weak. Thus, ray 108 only has a small contribution to pixel 114.

[0106] By pre-booking a quantity of rays (samples) for each pixel and repeating the above path tracing over the pixels of the frame, a raster illumination map 112 of the scene as viewed from the viewpoint of camera 101 is generated. Then, the raster illumination map is filtered by filter 400 in the manner described below to form a filtered illumination map combined with the raster image generated by texture data path 508 in frame generator 505. To generate an output frame 509 of the desired pixel resolution including texture and lighting information, frame generator 505 combines the colors of one or more pixels of the raster illumination map with one or more corresponding pixels of the raster image generated by texture path 508. The pixel dimensions of the raster illumination map can match the pixel dimensions of the raster image generated by texture path 508 such that the color of each pixel of the raster image is combined with the color of the corresponding pixel of the raster illumination map to generate the corresponding pixel of the output frame. The combination performed by the frame generator can be performed in any suitable manner depending on the color space defining the colors and the lighting rules defined for the composite environment of the scene.

[0107] The graphics system is configured to determine the color of each pixel of the raster illumination map; alternatively, the graphics system can be configured to determine the brightness of each pixel of the raster illumination map, but not the color information (such as chromaticity or color saturation). This can be achieved by using the mapped texture data defined for each point where reflection occurs in the scene to determine the effect of the reflection on the brightness of the corresponding light ray without modifying the chromaticity and / or color saturation of the light ray.

[0108] Filtering

[0109] Now, the operation of the filter 400 will be described with reference to Figure 4a and Figure 4b the examples shown in

[0110] Figure 4a and Figure 4b each illustrate an example of the filter 400 suitable for operating on the raster data held at the memory 401. The filter 400 can be configured, for example, to operate on the frame 600 shown in Figure 6 which includes a plurality of pixels each having a pixel value. The pixel value can represent any kind of graphics characteristic (such as color or brightness). The frame 600 can be any kind of raster graphics (such as a grayscale image, a color image, or a raster illumination map). In the example described above with respect to Figure 5 the frame 600 will be the raster illumination map 112 generated by the path tracer 501 of the graphics system.

[0111] The filter 400 includes an input 413 for receiving pixel values from the memory 401, a filter module 405, and filter logic 404. The filter module can include any number of recursive filters configured to operate in parallel or in series. The filter logic 404 operates to combine the filtered pixel values formed at the filter module into a filtered output that can be stored at the memory 406. The function of the filter logic 404 can be integrated with the filter module 405. The memories 401 and 406 can or can not be one and the same memory, and each can include any number of data stores (including, for example, registers, working memory, and non-volatile memory).

[0112] Figure 4a FIG. shows a first example including the filter 400, which includes a filter module 405 having a single recursive filter 402 configured to operate on the raster data held at the memory 401 to provide filtered pixel values to the filter logic 404. In this example, the recursive filter 402 filters the pixels C of the incoming sequence in a first direction (e.g., from left to right according to Equation 1-3 below) to form the filtered pixel CL-R and stored in the row store 409. Then, the same recursive filter 402 filters the pixels C of the incoming sequence in a second direction (e.g., from right to left according to Equation 4-6 below) to form the filtered pixel C R-L . The pixels C filtered in the first direction and stored in the row store 409 L-R are combined at the combining unit 408 of the filter logic 404 with the pixels C filtered in the second direction according to Equation 7 below R-L and the incoming pixels C to form the filtered output C OUT . The switch 412 schematically represents the logic of the filter logic 404, which is configured to direct the pixels filtered in the first direction to the row store 409 and the pixels filtered in the second direction to the combining unit 408 to be combined with the previously filtered pixels in the first direction from the row store 409. In this way, a single recursive filter can be used to perform filtering according to the principles set forth herein.

[0113] Figure 4b FIG. shows a second example of the filter 400, which includes a pair of recursive filters 402 and 403, each recursive filter being configured to operate on the raster data held at the memory 401 to provide filtered pixel values to the filter logic 404. In this example, the incoming pixels C are simultaneously filtered by the recursive filter 402 in a first direction (e.g., from left to right according to Equations 1-3 below) and by the recursive filter 403 in a second direction (e.g., from right to left according to Equation 4-6 below) to form the filtered pixel C L-R , and to form the filtered pixel C R-L . The pixels C filtered in the first direction L-R are combined at the combining unit 408 of the filter logic 404 with the pixels C filtered in the second direction according to Equation 7 below R-L and the incoming pixels C to form the filtered output C OUT . The recursive filters 402 and 403 are arranged to operate in opposite directions on the sequence of pixel values. The sequence can be continuous within the frame 600 such that each pixel value in the sequence is adjacent in the frame to the pixel values immediately before and after it in the sequence (with possible exceptions at the start and end of the sequence).

[0114] As described in more detail below, each recursive filter can include a pair of filters (e.g., the recursive filter 402 includes filters 408 and 409; the recursive filter 403 includes filters 410 and 411), each pair of filters being arranged to perform filtering according to Equations 1 and 2 or Equations 4 and 5 at any given time to generate intermediate pixel values C1 and C2 for each filtering direction. Formed at the recursive filterThe intermediate pixel values C1 and C2 are combined at the combining unit 414 of the recursive filter according to Equation 3 or 6 (depending on the operating direction of the filter) to form their respective filtered outputs C L-R or C R-L .

[0115] In other examples,

[0116] the recursive filter can include a single non-decaying filter such that the output of the recursive filter is the output of its non-decaying filter. For example, if Figure 4b the recursive filter 402 in Figure 4b would include filter 408, then the output of the recursive filter can be given by Equation 1 below. The output of such a recursive filter can be combined at the filter logic according to Equation 7 below in the same manner as described in the example shown in Figure 4b The recursive filter for performing recursive filtering is described in the examples in the following optional discussion of "pre-filtering", where

[0117] one of the filters in the recursive filters 402 / 403 shown in Figure 6 is used to perform recursive filtering and the other filter is used to perform pre-filtering. Figure 6 is indicated by 612 in

[0118] Filtering along the first direction

[0119] Now, the operation of filter 400 on frame 600 will be described in more detail via examples. In frame 600, each of the pixel rows and pixel columns in the frame represents a possible pixel sequence. However, other sequences are possible, including sequences defined within a block of the frame (as exemplified by block 611 in Figure 6 and a single sequence defined for all the pixels of the frame (as indicated by 612 in

[0120] In the examples described below, the color of a pixel is represented using homogeneous color coordinates (the use of homogeneous color coordinates can be considered similar to the use of homogeneous geometric coordinates). A color with three components (i.e., r, g, and b) is represented using homogeneous color coordinates of four component vectors r, g, b, w, where w reflects the "weight" of the color. The weight of a pixel can, for example, be a measure of the number of samples used to generate the color value of the pixel. Alternatively, the weight of a pixel can be derived as a result of the use of a "multiple important sampling" technique, which is known in path tracing and assigns weights to each sample according to the sampling technique used. Typically, the weight indicates the quality or confidence of the value assigned to the pixel. One way to generate a homogeneous color value from RGB colors and weights is to multiply the three color components by the weights to generate a homogeneous color vector {R×w, G×w, B×w, w}. In order to generate a three-component vector from this homogeneous color vector {r, g, b, w}, we divide the color channel by the weight to generate a color vector {r / w, g / w, b / w}. This method is used in the examples described below.

[0121] Using homogeneous coordinates to represent color has several advantages in addition to simplifying the math (no need to update colors and weights separately). Using homogeneous coordinates can, for example, improve performance when implementing filtering on a CPU or GPU by first enabling the use of 4-component vectors that better utilize vector instructions found in many processors and secondly by allowing the expensive division operations described below to be deferred to a single final step. When the path tracer accumulates samples during path tracing, the path tracer can be advantageously configured not to divide the accumulated colors by the number of samples. For example, in a simple case where samples do not carry independent weights, the path tracer can accumulate vectors of the form {r, g, b, 1} for each sample (where r, g, and b are the color components of the sample), and the final weight (w value) will reflect the number of accumulated samples. In other examples, where samples have independent weights, vectors of the form {r, g, b, w} can be accumulated in a similar manner. In an example where only brightness is considered and non-three-component color, the above equivalent homogeneous {r, g, b, w} vector can be expressed as {l, w}, where l is the brightness value of the sample.

[0122] Next we use the symbol Represents the filtered value. is the filtered homogeneous color value of the pixel in the sequence produced by filter m (n is not to be confused with the exponent). By color / intensity and weight composition.

[0123] The recursive filter 402 is constructed to form a matrix with respect to the pixel value C in the sequence by calculating the following formula: nThe first intermediate filtered value of the nth pixel

[0124]

[0125] where α1 is a constant and d n is a measure of the similarity between the nth pixel and the (n - 1)th pixel. Thus, each pixel value receives a contribution from its previous filtered pixel value, which is attenuated according to the similarity to the object pixel value.

[0126] The first intermediate filtered value formed by the recursive filter 402 can be used as the output of the filter 402 for combination with the corresponding first intermediate filtered value formed on the pixels of the sequence in the opposite direction at the filter logic 404. However, if the recursive filter 402 is also configured to form a second intermediate filtered value for the nth pixel having a pixel value C n by calculating the following equation then improved performance can be achieved:

[0127]

[0128] where α2 is a constant and d n is a measure of the similarity between the nth pixel and the (n - 1)th pixel.

[0129] As Figure 4a and Figure 4b shown, the recursive filter 402 can include two filters that form two intermediate filtered values: a first filter 408 configured to calculate the first intermediate filter value and a second filter 409 configured to calculate the second intermediate filter value. The two filters 408 and 409 can operate in parallel to simultaneously form the first filtered value and the second filtered value. Similarly, another recursive filter (e.g., 403) can include two filters 410 and 411 that are set to calculate the first intermediate filtered value and the second intermediate filtered value. One or more of the filters in the recursive filter can be an exponential filter.

[0130] By operating on the sequence of pixel values with the recursive filter 402 according to equations 1 and 2, the first intermediate filtered value and the second intermediate filtered value can be calculated for each pixel in the sequence. Since the first value in the sequence will typically not have a previous pixel value, a weight of zero can be used for the "previous" pixel to calculate the intermediate filtered value of the first pixel or it can be some suitable initial value adapted to the nature of the frame and / or the filtering to be performed. Then, the weight coordinates are updated in the same manner as the color coordinates according to equations 1 and 2, such that the weight of a given pixel is equal to the weight of the current pixel plus αi d n (depending on the formula) the weights of the previous pixels adjusted. The constants α1, α2, and d n are selected such that the quantity α1d n and α2d n lie between 0 and 1. The maximum value within this range indicates high similarity between pixels, and the minimum value within this range indicates low similarity between pixels. Higher values of a1 and α2 will cause the filter 400 to produce a smoother but more blurred output; this means less noise, but may result in soft shadow edges when the filter is configured to operate on a grid illumination map.

[0131] The first intermediate filtered value and the second intermediate filtered value of each pixel are combined at the filter 402 (e.g., at the combining unit 414) by the following formula to form the filtered pixel value in the left - right direction:

[0132]

[0133] where,

[0134] β1 = 1 - α1

[0135] β2 = 1 - α2

[0136] When the filters that generate the first intermediate filtered pixel and the second intermediate filtered pixel values are exponential filters, the filter 402 can have an impulse response approximated to "half - Gaussian", and the weights (the fourth coordinate of the homogeneous system) reflect an estimate of the number of accumulated samples. "Half - Gaussian" describes a filter response that rises to a peak in a Gaussian pattern before a sudden drop. Since the value is constant, it does not need to be calculated during filtering. Another exponential filter can be used to improve the approximation to the Gaussian in the rising part of the response curve.

[0137] Filtering along the second direction

[0138] Now, the operation of the recursive filter 403 in Figure 4b will be described by way of an example of filtering along the second direction. It will be understood that the following description also applies to the recursive filter 402 when a single filter is configured to operate along the second direction on a pixel sequence.

[0139] The recursive filter 403 is constructed in a similar manner to the filter 402, but when the filter 402 operates in the left - to - right direction, the filter 403 operates from right to left on the same pixel sequence to form the filtered value of the pixel according to the value of the pixel and the filtered values of the previous pixels in the sequence in the right - to - left direction. Thus, when operating on the pixel row 602 in the frame 600, Figure 6The filtered value of the middle pixel 604 is formed based on the value of pixel 604 and the filtered value formed for the previous pixel 605. Again, more generally, the filter 403 may form each filtered value based on the contributions from two or more pixels in the sequence that may or may not be immediately adjacent to each other.

[0140] The recursive filter 403 is configured to form a first intermediate filtered value for the k-th pixel in the sequence having a pixel value C k by calculating the following equation (where k = N - n, N is the number of pixels in the sequence, and n = 0 is the leftmost pixel in row 601):

[0141]

[0142] where a3 is a constant, and d k is a measure of the similarity between the k-th pixel and the (k - 1)-th pixel. Thus, each pixel value receives a contribution from its previous filtered pixel value, which is attenuated based on the similarity to the object pixel value.

[0143] The first intermediate filtered value formed by the recursive filter 403 can be used as the output of the filter 403 for combination with the corresponding first intermediate filtered value formed by the recursive filter 402 at the filter logic 404. However, if the recursive filter 403 is also configured to form a second intermediate filtered value for the k-th pixel in the sequence having a pixel value C k by calculating the following equation then improved performance can be achieved:

[0144]

[0145] where α4 is a constant, and d k is a measure of the similarity between the k-th pixel and the (k - 1)-th pixel.

[0146] As Figure 4b shown, the recursive filter 403 may include two filters that form two intermediate filtered values: a first filter 410 configured to calculate the first intermediate filter value and a second filter 411 configured to calculate the second intermediate filter value. The two filters 410 and 411 may operate in parallel to simultaneously form the first filtered value and the second filtered value. One or more of the filters in the recursive filter may be an exponential filter.

[0147] By operating on the sequence of pixel values according to equations 4 and 5 with recursive filter 403, a first intermediate filtered value and a second intermediate filtered value can be calculated for each pixel in the sequence. Since the first value in the sequence will typically not have a previous pixel value, a weight of zero can be used for the "previous" pixel to calculate the intermediate filtered value of the first pixel Or it can be some suitable initial value adapted to the nature of the frame and / or the filtering to be performed. Each similarity measure can take a value within a predefined range between 0 and 1, with the maximum value within this range indicating high similarity between pixels and the minimum value within this range indicating low similarity between pixels.

[0148] The first intermediate filtered and second intermediate filtered values of each pixel are combined at filter 403 (e.g., at combining unit 414) by the following equations to form a filtered pixel value in the right-to-left direction:

[0149]

[0150] β3 = 1 - α3

[0151] β4 = 1 - α4

[0152] wherein,

[0153] When the filters that generate the first intermediate filtered pixel value and the second intermediate filtered pixel value are exponential filters, filter 403 can have an impulse response approximating a "half-Gaussian", and the weights (the fourth coordinate of the homogeneous system) reflect an estimate of the number of samples accumulated. "Half-Gaussian" describes a filter response that rises to a peak in a Gaussian pattern before a sudden drop. Since the value is constant, it does not need to be calculated during filtering. Additional exponential filters can be used to improve the approximation to the Gaussian in the rising part of the response curve.

[0154] To minimize the complexity of recursive filters 402 and 403, it is preferred that they are constructed to form an output for a pixel only based on the values of the pixel and the previous pixel. This allows the n-1th value in the sequence to be discarded from the working memory of the filter when forming the nth value in the sequence.

[0155] It is advantageous if each filter of the recursive filter is constructed to form its first intermediate filtered value and its second intermediate filtered value in a single pass. In other words, since the filter forms each of the first intermediate filtered value and the second intermediate filtered value of a pixel based on the same two pixel values, it is preferred that the filter is constructed to form the two filtered values when the two required pixel values are local to the filter (e.g., held in the registers of each filter after retrieval from memory 401).

[0156] In other examples, the recursive filters 402 and 403 can be configured to form each filtered value based on more than two pixel values.

[0157] The filter logic 404 is configured to combine the filtered pixel values formed by the filters 402 and 403 to generate a filtered output value for each pixel The filter logic is configured to combine the filtered pixel values by calculating the average of the filtered pixel value sums for each pixel of the frame 600. This can be a simple means for the filtered pixel values, but better filter performance can be achieved by forming a weighted average of the filtered pixel values using weights derived from the weights of the pixels. Using homogeneous color coordinates, this can be achieved by performing a simple operation. This is possible in an example where the recursive filters maintain weights for each pixel. For example, good performance can be achieved by configuring the filter logic to generate a filtered output value according to the following equation:

[0158]

[0159] Since we include the pixel value C n in the filtered values from two directions, it is included twice, so we subtract the pixel value C n .

[0160] In this way, the filtered output value for each pixel in row 601 of frame 600 can be formed and stored at the memory 406. By repeating the above steps on the rows of the frame, a new frame filtered in the first (e.g., horizontal) direction along the pixel sequence represented by the rows of the frame can be generated and held at the memory 406.

[0161] Similarity measure

[0162] Now, the similarity measures used by the subsequent behavior of the recursive filters 402 and 403 and the filter 400 will be described.

[0163] The similarity measure of a given pixel represents a certain relationship between that pixel and the previous pixel with respect to the operation direction of the filter. For example, the similarity measure of pixel 604 in Figure 6 at the recursive filter 402 when the filter operates in the left-to-right direction along row 601 will be a measurement of the relationship between the characteristics of pixel 604 and the corresponding characteristics of the previous pixel 603. The similarity measure of pixel 604 in Figure 6The similarity measure for pixel 604 in will be a measurement of the relationship between the characteristics of pixel 604 and the corresponding characteristics of the previous pixel 605. In other examples, the previous pixel may not be the pixel immediately before the pixel being filtered (e.g., the previous pixel may lead two or more pixels in the row or column being filtered). Where the filtered value is formed based on more than two pixel values, there may be more than one similarity measure for a pixel formed with respect to the relationship between the pixel and two or more other pixels. The similarity measure between a given pair of pixels can be the same regardless of the filtering direction, or the similarity measure between a given pair of pixels can be different.

[0164] In Figure 4a and Figure 4b In the example shown in , the similarity generator 405 is configured to form a similarity measure for each pair of pixels independent of the filtering direction (and in the case described herein, the similarity measure is formed with respect to adjacent pixels). The similarity generator calculates the similarity measure for each pair of pixels such that the similarity measure can be used by the filter 400 during filtering. In other examples, the similarity measure can be determined at the filter 400 during filtering, which can be useful where the similarity measure for a pair of pixels varies according to the filtering direction. Figure 4a and Figure 4b The similarity generator is schematically shown in , and the similarity generator may or may not form part of the filter 400.

[0165] The similarity measure can be generated based on any suitable characteristic of the frame or other data related to these pixels. For example, the similarity measure can be formed for adjacent pixel pairs based on one or more of the following: a distance measurement in the color space between the colors of these pixels; a difference measurement of the brightness of the pixels; a difference measurement of the depth in the scene to the surfaces represented by these pixels in the 3D scene of the frame of each surface; and an angle measurement between the normals of the surfaces represented by these pixels in the 3D scene of the frame. The similarity measure will generally be chosen to be expressed as a positive value (e.g., the order of magnitude of some calculated quantity). The similarity measure can be in the range [0, 1], where 0 indicates the minimum similarity and 1 indicates the maximum similarity.

[0166] In the case where depth information is available for the scene, the filter 400 can be used to create a scene depth effect in the raster image instead of performing noise reduction. This can be achieved by changing the similarity measurement d of the recursive filters 402 and 403 according to the depth of each pixel representing the 3D scene related to a certain defined focal plane (effectively changing the filter radius).

[0167] In some examples, the similarity generator is configured to form respective similarity measures based on one or more characteristics of the pixels of the filtered frame 600. For example, the frame 600 can be a graphic that needs noise reduction or an image frame in a camera pipeline, and the similarity generator can be configured to form similarity measures based on one or more characteristics of adjacent pixels (such as their color or brightness). For example, the similarity generator can calculate a similarity measure based on the distance between the colors of adjacent pixels in a color space. With respect to Figure 5 the exemplary graphics system shown in, the image frame can be the output true 509, and the filter 400 can be arranged in the system after the frame generator 505 (alternatively or additionally arranged to Figure 5 the filter 400 shown in).

[0168] In other examples, the similarity generator can be configured to form respective similarity measures for these pairs of adjacent pixels based on other data related to the pixels that is not present in the pixel values themselves. This can be understood by referring back to Figure 5 the exemplary graphics system shown in. Figure 5 The system of generates a frame 509 representing a 3D scene as viewed from the viewpoint of the camera 513. The scene geometry can be used at the data store 511, and the rays traced by the path tracer 501 identify which regions of those surfaces of the scene contribute to the respective pixels of the grid illumination map generated by the path tracer. In the system 500, the normal vector of a pixel of the grid illumination map can be given by the normal vectors of the regions of the surfaces in the scene that contribute to that pixel. Similarly, the depth value of a pixel of the grid illumination map can be given by the scene of the regions of the surfaces in the scene that contribute to that pixel. Techniques for calculating such normal vectors or depths in a 3D scene are well known in the art.

[0169] In cases where normal vector and / or scene depth information is available for the frame on which the filter 400 operates, it is advantageous to use this information to form similarity measures between these nodes. This is typically the case when the frame represents a 3D scene (such as in Figure 5This is acceptable when in the graphics system). Using this basic information about the scene to form the similarity measure used in filter 400 enables the filter to perform filtering based on whether adjacent pixels belong to common features of the scene (e.g., common surfaces). Since there may be adjacent pixels in a frame that happen to have the same color / intensity but are actually related to different features of the underlying scene that generated the frame, it is generally preferred to form the similarity measure based on the color or luminance of the pixels. There may also be cases where adjacent pixels in a frame are related to the same feature in the scene but have very different colors / luminances due to the surface decoration of that feature. Specifically, using the normal vector and / or scene depth information allows the filter to take into account target boundaries (e.g., edges between surfaces) when performing filtering. This can significantly improve the appearance of the resulting filtered frame and allow for high-quality path tracing of lighting for the scene using only a limited number of samples per pixel (SPP) (rays). Values as low as 4 SPP are acceptable when maintaining reasonable quality. This makes path tracing possible on a wide range of customer systems, including mobile devices with limited processing resources. Additionally, forming the similarity measure based on the similarity of the luminance and / or color of pixel pairs in the light map can contribute to preserving shadow edges by causing a drop in the similarity measure when crossing a shadow boundary (e.g., due to a large difference between the luminance / color pixel values of the pixel pair on one of the two sides of the shadow boundary).

[0170] Continue Figure 5 In an exemplary graphics system, the similarity generator is configured to form a similarity measure for adjacent pixels based on the product of the measure of (a) the angle between the normal vectors of the surfaces at the points represented by the pixels in the scene and (b) the difference between the depths of these points in the scene. In frame 112 where pixels 115 and 116 are adjacent pixels Figure 1 Examples of these measures are illustrated. The angle between the normal vectors at points 125 and 126 in the scene (contributing to pixels 115 and 116 respectively) is indicated by angle 150. The measure of angle 150 can be, for example, the angle itself or a function of the angle (such as its sine or cosine). The distance between the depths of these points in the scene is indicated by distance 151. The measure of the distance 151 between the scene depths can be determined in any suitable unit. It can be advantageous to represent the scene depth as 1 / d, where d is the scene depth. This can accelerate the calculations performed with the scene depth. In other examples, the similarity measure can be determined based on more than two depth values, and a discontinuity in the rate of change of the depth values indicates different surface orientations.

[0171] The similarity generator forms as follows Figure 6 (In this example, it can be the grid light map formed by the path tracer 501 in Figure 5 ) the similarity measure d between adjacent pixels in frame 600 shown n :

[0172]

[0173] wherein, is a measure of the magnitude of the angle between the normal vectors of the surface at the point represented by the nth pixel and the (n - 1)th pixel in frame 600 in the scene, and is a measure of the magnitude of the difference in depth at the point represented by the nth pixel and the (n - 1)th pixel in frame 600 in the scene.

[0174] The normal vector of the scene surface and / or the depth information of the scene can be derived from the scene geometry 511 available in the graphics system 500 shown in Figure 5 . The normal vector of the scene and / or the scene depth value can be pre-computed for the scene at the early stage of the graphics pipeline so that the required data is available for the similarity generator to calculate d n . This can save the bandwidth required to access the scene geometry data 511 during the filtering stage.

[0175] In the above equations 1 and 2, it is advantageous if the values of the constants α1 and α2 are chosen such that the difference between the constants is significantly less than the magnitude of either α1 or α2. Similarly, in equations 4 and 5 above, it is advantageous if the values of the constants α3 and α4 are chosen such that the difference between the constants is significantly less than the magnitude of either α3 or α4. For a value of α1 of magnitude 0.8, a difference in the order of 0.001 to 0.01 has been found to be effective. For example, the difference can be at least of the order of less than the magnitude of the i α value. By carefully choosing the constants α1 and α2, α3 and α4, the "semi-Gaussian" responses of two filters operating on the data in different directions are combined such that filter 400 exhibits Gaussian characteristics without the need to perform the large amount of processing resources required for true Gaussian filtering. Thus, filter 400 is an alternative for performing Gaussian filtering on the pixels of a frame at high speed and / or low power.

[0176] Figure 7 The operation of filter 400 is illustrated via a simple example in Figure 7 Frame 700 is shown, which depicts a red ball 702 suspended in an additional planar white frame on which filter 400 is configured to operate. In this example, filter 400 is configured to operate on the red channel values of the pixels and calculate a similarity measure for each pair of adjacent pixels based on the scene depth at the points represented by these pixels in the scene. The recursive filter 402 operates on the pixels of row 701 in the frame from left to right to generate the filtered pixel values 704 of the row. Similarly, the recursive filter 403 operates on the pixels of row 701 in the frame from right to left to generate the filtered pixel values 705 of the row.

[0177] The filtered value of each pixel generated by a recursive filter depends on the filtered value of a previous pixel with respect to the operation direction of the filter, and the filtered value of that previous pixel in turn depends on the filtered value of its previous pixel. Thus, the red color of the pixels depicting the red ball is removed along the direction of operation of each recursive filter. This can be understood according to the curves Figure 7 shown in which illustrate the weights of the accumulated sample values: Filter 402 operates from left to right as shown by curve 704, while filter 403 operates from right to left as shown by curve 705. The similarity measure d n between adjacent pixels ensures that the red color of the ball does not mix into the white background and the shape of the ball is blurred. This is because the similarity measure between pixels on one of the two sides of the ball edge will be close to zero, so that the filter effectively restarts filtering as it crosses the boundary. This will be true regardless of whether the similarity measure is based on surface normals, scene depth, or the similarity of the colors of adjacent pixels.

[0178] By summing the weight values 704 and 705, filter logic 404 generates a weight value 706 which will be symmetric if the similarity measure d n is equal for each pair of adjacent pixels and the recursive filters 402 and 403 start with the pixel weights of the same row 701. It can be seen that the weight value block target 702 is high and relatively constant, which means that the output of the filter will result from the average of several pixel values. This is true even at the edge of the target 702, which shows that the filter can effectively filter up to the edge of the target, but the similarity measure blurs the color of the target 702 into the background 700.

[0179] Regarding Figure 1 shown in the Figure 2 scene 100 Figures 3a - 3g illustrates a more complex example. Figure 2 is an example of a raster illumination map 200 generated by a path tracer 501 of a graphics system 500 for the Figure 1 scene shown in Figure 5 as viewed from the viewpoint of a camera 101. Each pixel of the raster illumination map 200 represents the illuminance received at that pixel from one or more path tracing rays directed to the camera 101 defined for the scene. As discussed, each pixel can express the illuminance it receives, for example, as a color or brightness pixel value.

[0180] Figures 3a - 3gA series of diagrams are shown that illustrate the filtering of pixel values of row 201 of a raster illumination map from pixel 202 to pixel 203. In the drawings, the pixel values are luminance values (e.g., the luminance or Y value of each pixel in the case where the raster illumination map is defined in the YUV color space). Line 301 is a diagram of the unfiltered pixel values of the raster illumination map between pixel 202 and pixel 203, and it can be seen that line 301 includes a large amount of noise due to a small number of per-pixel samples being performed by the path tracer 501. From Figure 2 It can be seen that a segment of pixel values from pixel 202 to pixel 203 passes from pixels representing light received from the polished marble column 104 to pixels representing light received from the dark gray stone wall 117. The transition between the two surfaces occurs at Figure 1 and Figure 2 at the edge 111 shown in, and in a perfect rendering of the scene, this transition would correspond to the true underlying drop in pixel values indicated by the curve 302 in Figure 3a .

[0181] The presence of edge 111 indicates that the similarity between pixels on one side of the edge will be low. This is illustrated by line 303 in Figure 3b which is a diagram of the similarity measure along a segment of pixel values from pixel 202 to pixel 203. For example, consider the case when the similarity of adjacent pixels is determined by the angle between the surface normals of these pixels. Figure 2 The angle between the normals of the pixels on one side of edge 111 in will be approximately 90 degrees (one pixel will represent a part of column 104 and one pixel will represent a part of wall 117). The similarity measure can be chosen, for example, as cosθ, where θ is the angle between the normals, such that an angle of approximately 90 degrees between the surface normals at the points represented by adjacent pixels will result in a similarity measure close to zero.

[0182] The reduction in the similarity measure across edge 111 causes the pixel weights calculated for filtering in the L-R direction (according to equations 1-3) to drop suddenly in the manner shown by line 304 in Figure 3c . Similarly, the pixel weights calculated for filtering in the R-L direction (according to equations 4-6) drop suddenly in the manner shown by line 307 in Figure 3e . Figure 3d and Figure 3f show the effect on the filtered pixel values formed in the L-R and R-L directions.

[0183] Figure 3dThe line 305 in [description] is a graph of the filtered luminance values generated by the recursive filter 402 of the filter 400 along the L-R direction according to the principles described herein (e.g., equations 1-3). It can be seen that the filtered line 305 eliminates many noises present in the line 301. Additionally, since the filtering performed by the recursive filter 402 uses the normal vector and depth information available for the scene 100, it can be seen that the filtered line 305 closely follows the true underlying luminance in the scene, without pushing the high luminance values of pixels receiving light reflected from the marble column to the lower luminance values of pixels receiving light from the darker wall. Most of the noise before the edge 111 is removed by the filtering, but some noise 306 remains after the edge with respect to the direction of the filter operation. This is the result of the filter being "reset" by the reduction of similar values and thus the lower smoothing contribution from adjacent pixel values (see equations 1 and 2).

[0184] Figure 3f The line 308 in [description] is a graph of the filtered luminance values generated by the recursive filter 403 of the filter 400 along the R-L direction according to the principles described herein (e.g., equations 4-6). The line 308 shows the same characteristics as the line 305, but the noise 309 occurs after the edge with respect to the filter direction following the filter "reset" caused by the low similarity across the edge 111 and thus to the left of the edge.

[0185] Figure 3g The line 310 in [description] is a graph of the filtered output of the filter 400 generated at the filter logic 404 by combining the filtered pixel values formed at the recursive filters 402 and 403. It can be seen that Figure 3d and Figure 3f most of the noises 306 and 309 in [description] are attenuated in the output 310 to generate a smooth filtered output in the region 311 that accurately tracks the sharp drop in luminance values due to the edge 111. This is because the noise region 306 corresponds to the low-weight region in the curve 307. Thus, the noise signals from the recursive filters 402 / 403 are suppressed by the cleaner signals from other pairs of filters that dominate in the output 310.

[0186] Return to Figure 5In the example shown, it is particularly advantageous if the filter 400 is set to perform filtering on the rasterized light map generated by the path tracer 501 before it is combined with the raster image generated by the rasterizer 504 at the frame generator 505. For a given output frame 509 with a desired sharpness, this allows the filter 400 to be configured to perform a greater degree of smoothing, for example, by selecting higher constant values for equations 1, 2, 4, and 5 than would be possible if the filter 400 were instead set to operate on the image frame 509 that includes both lighting and texture information. In other words, a greater degree of noise reduction can be performed without blurring the texture contributed by the raster image.

[0187] This allows good performance to be achieved when only a small number (e.g., 4) of per-pixel samples are used at the path tracer 501. Additional filtering using a conventional filter or the filtering techniques described herein can be performed on the raster image before it is combined with the rasterized light map at the frame generator 505 or on the final image at the output of the frame generator 505. For example, filtering that introduces blurring based on the scene depth can be performed.

[0188] Pixel weights

[0189] The use of pixel weights ensures good performance of the filter because it enables the filter to operate with an understanding of the correct confidence in the pixel values (e.g., the color or brightness of the pixels). In the example described herein, the pixel weights are set based on the fourth coordinate of the homogeneous coordinate system of the color components of the pixels, but in other examples, the pixel weights can be treated as parameters for each pixel independently of the color components of the pixels.

[0190] Roughly speaking, the use of pixel weights also provides a measure of the effective number of pixels being averaged by the filter when performing recursive filtering across the last edge (when the similarity factor will have to reset the weight coordinates to some extent). Pixels with low weights contribute less to the accumulated average than pixels with high weights. For example, a pixel with a 70% weight can contribute only 70% of its color (hypothetically) to the accumulated average. Such a pixel can be represented in the pixel weights as pixel 0.7. A pixel with 100% weight can be considered a full pixel.

[0191] The weight of a pixel can be any suitable measure of the quality of the pixel value or the confidence in the accuracy of the pixel value. For example, in the case of a noise reduction filter, the weight of a pixel can be inversely proportional to the expected noise contribution from that pixel. Each weight will generally take any positive value. By using appropriate weights, the filter 400 allows pixel values with high confidence to influence neighboring pixels more than pixel values with low confidence. This behavior can be understood according to equations 1, 2, 4, and 5 described above.

[0192] The weight of a pixel can be a measure of the number of rays contributing to the pixel value. Alternatively or additionally, the weight of a pixel can be a measure of the noise in the pixel, with a higher weight corresponding to lower noise, and such a parameter can be determined for a pixel based on, for example, a statistical analysis of the variation of pixel values of a set of pixels including the pixel of interest (e.g., the filtered row / column) or pixels belonging to a common feature in the underlying 3D scene. Alternatively, the weight of each pixel of the frame to be filtered can initially be set to the same value for all pixels (e.g., set to the value "1"), or set to some measure of the average noise in the frame.

[0193] In this example, the initial weight of a pixel is a measure of the number of rays contributing to the pixel value. In one example, the initial weight of a pixel can be set to take values between 0 and 1. For example, if the path tracer 501 is configured to attempt to trace 4 samples per pixel (SPP), then the weight of each pixel can be given by the ratio of the number of rays actually contributing to the pixel value to the number of samples per pixel (i.e., if 3 rays contribute to a given pixel and the path tracer is configured to operate at 4 SPP, then the weight of that pixel is 0.75). Some of the traced rays will not contribute to the value of the pixel. In other examples, the initial weight of a pixel can take any positive value representing the number of samples contributing to the pixel (e.g., 3 in the foregoing example). Information identifying the number of samples contributing to each pixel can be provided by the path tracer 501 to the filter 400 and held at the memory 401.

[0194] The initial weights determined for the pixels of the frame are filtered in the same manner as the color or luminance coordinates described by the above equations 1 to 7. For example, considering the weight coordinates alone, the recursive filter 402 (e.g., at the filter 408) will be configured to update the initial weight w of the nth pixel according to the following equation n :

[0195]

[0196] where α1 is a constant, a n is a similarity measure between the nth pixel and the (n - 1)th pixel, and is the first intermediate filtered weight of the (n - 1)th pixel.

[0197] Similarly, the recursive filter 402 (e.g., at the filter 409) will be configured to update the initial weight w of the nth pixel according to the following equation n :

[0198]

[0199] where α2 is a constant, a nis the similarity measure between the nth pixel and the (n-1)th pixel, and is the second intermediate filtered weight of the (n-1)th pixel.

[0200] The first intermediate filtered weight and the second intermediate filtered weight of each pixel can be combined at the filter 402 (e.g., at the combining unit 414) by the following equations to form the filtered weight in the left-right direction:

[0201]

[0202] β1 = 1 - α1

[0203] β2 = 1 - α2

[0204] where

[0205] In a similar manner, the filtered weight can be formed in the right-left direction according to equations 4 to 6. This filtering can be performed, for example, at the recursive filter 403. Then, the filtered weights thus formed can be combined at the filter logic 404 described above according to equation 7 to give the output filtered weight of each pixel.

[0206]

[0207] The first weight and the second weight are derived from each other during the operation of the recursive filter due to different values of α1 and α2. To minimize the copying between the filter and the memory 401, the first intermediate weight and the second intermediate weight of the pixel are preferably updated by the recursive filter 402 when the recursive filter 402 forms the first intermediate filtered pixel value and the second intermediate filtered pixel value of the pixel and when the required data is present in the working memory / register of the filter.

[0208] The pixel weights generated from equation 13 can be stored in the memory 406 for subsequent use.

[0209] Filtering sequence in another orientation

[0210] To avoid introducing artifacts into the frame filtered by the filter 400 in the first direction (in the above example, along the rows of the frame), it is advantageous to subsequently filter the frame in the same manner as described above on a sequence having a second orientation (e.g., along the columns of the pixels of the frame). This second stage filtering can be performed by a second filter having a similar construction to the filter 400 described above, or the filter 400 can be configured to perform both the first stage filtering and the second stage filtering in any suitable manner (e.g., continuously). Now, reference will be made again to Figure 6 describe the second stage filtering.

[0211] In the above example, the filter 400 forms the filtered output pixel values of the pixels of the frame 600 row by row and stores these output values at the memory 406. For example, the output pixel values generated according to Equation 7 represent the frame 600 filtered along a row. The output pixel values generated as the output of the filtering performed along a first direction (e.g., along the horizontal direction) are preferably used as the starting point for filtering on the pixel sequence of the frame 600 having a second orientation (in this example, along the pixel columns of the frame). The output pixel values include the output weights generated for the weight coordinates according to Equation 7. In other examples, the same initial pixel values and weights can be used as the input to a second filter constructed similar to the filter 400, and the final output frame is generated by fusing the filtered images generated by the filtering on the pixel sequences of the two orientations together.

[0212] By providing the output pixel values as the input to the filter 400 in place of the unfiltered pixel value C n , and configuring the filter 400 to perform filtering on the columns of the frame 600, the filter 400 can be configured to be a final filtered frame in which the pixel values are filtered along both the first orientation and the second orientation. This method can eliminate artifacts (such as horizontal trailing in the output frame) seen when filtering only along a single orientation, and can achieve excellent filtering performance. Under this method, used to replace C in the equations given above n . The same or different constants can be used by the filter 400 in Equations 1 to 11.

[0213] Regarding Figure 6 and Equations 1 to 11, in the above equations and descriptions, the left side can be made higher and the right side can be made lower, and vice versa. For example, consider filtering along the column 602 of the frame 600. The recursive filter 402 can be configured to filter along the up-down orientation along the pixel sequence represented by the column of the frame, such that under the operation of the filter 402, the pixel value 604 receives contributions from the pixel value 606 according to Equations 1 to 3. The recursive filter 403 can be configured to filter along the down-up orientation along the pixel sequence represented by the column of the frame, such that under the operation of the filter 403, the pixel value 604 receives contributions from the pixel value 607 according to Equations 4 to 6. When performing filtering along the columns of the frame 600, the filter 400 can perform noise reduction on the previously filtered pixel values along a second up-down orientation orthogonal to the right-left orientation of the previous filtering stage.

[0214] In this way, the final filtered output values of the respective pixels of the frame 600 can be formed to serve as the filtered output frame generated by the filter 400. The final filtered output values of the respective pixels As given by the output of arithmetic expression 7 during second-stage filtering in the second orientation by filter logic 404, and wherein the input pixel values for the second stage are the output pixel values from the first-stage filtering

[0215]

[0216] wherein, "L-R" is replaced by "U-D" to reflect that in the second stage, filtering is performed in frame 600 along the up-down orientation along the pixel sequence represented by the columns of the frame

[0217] In Figure 5 the example shown, the output frame including pixel values will represent the filtered grid illumination map used by frame generator 505 to generate output frame 509. Pixel values represented using homogeneous color coordinates can be converted to a conventional representation (e.g., three components R, G, and B) either at the output or by dividing each component by a weight value w

[0218] Unweighted filtering

[0219] In an alternative embodiment, filter 400 can be configured to perform filtering without using weights (e.g., pixel value C does not include a pixel weight and there is also no separate weight provided to describe the confidence of the color value of the pixel). This results in a more noisy filtered image, but can reduce the complexity of the filtering and potentially increase speed / reduce the processing requirements of the filter. Performing filtering according to the principles described herein without using weights requires some modification of the equations given above

[0220] Using the above definitions of α1, α2, and d n the following new constants are defined:

[0221] β1 = 1 - α1d n

[0222] β2 = 1 - α2d n

[0223]

[0224]

[0225]

[0226]

[0227] γ1 and γ2 are used to create a Gaussian-shaped convolution function. δ reflects the contribution of the current pixel to the combined pixel value generated by filtering pixels in two directions, and this contribution is subtracted from the combined pixel value. δ is a correction factor that is selected such that the average value of the filtered signal closely matches the average value of the original signal.

[0228] Then, the above Equation 1 becomes:

[0229]

[0230] And, Equation 2 becomes:

[0231]

[0232] Therefore, the pixel values from the first filter and the second filter are combined in a manner similar to that in Equation 3 as:

[0233]

[0234] And, the pixel values from the first filter and the second filter are combined in a manner similar to that in Equation 7 as:

[0235]

[0236] If filtering is performed along the second direction and the outputs of filters operating in different directions can be combined in any of the ways described in the above weighted examples, similar equations can be used.

[0237] Filter 400 can be configured to operate on any suitable pixel values (including, for example, colors expressed in RGB or YUV formats; one or more components of a color (such as the value of a specific color channel (e.g., the red channel in the RGB color space)); and luminance values (such as brightness (e.g., the Y component in the YUV color space))). If filtering is performed separately for one or more color channels of a frame, the filtered color components of each pixel can be combined to form a filtered color frame.

[0238] The filtering of the frames described here can be performed block by block for frames that are divided into multiple overlapping or non-overlapping blocks.

[0239] Filter 400 can be implemented in hardware, software, or any combination of the two. Graphics system 500 can be implemented in hardware, software, or any combination of the two. In one example, the recursive filter and / or the filter is set as code to be executed at a graphics processing unit (GPU). Multiple code instances can be supported at the GPU to provide multiple filters that are set to operate in parallel on a pixel sequence of a frame to perform the filtering of the frame. The code can be set to appropriately configure the computing units of the GPU to perform the filtering described here.

[0240] Figure 8 The exemplary structure of the data and functional elements of the filter 400 operating at the graphics system is shown. A number of maps used by the filter are generated based on the input pixels of the scene to be filtered:

[0241] · An illumination map 801 that describes the color / brightness of the input pixels in the scene due to the light sources in the scene - this map can be generated, for example, by the path tracer 501 in the manner described herein;

[0242] · A normal map 802 that describes the normal vectors of the respective input pixels - this map can be generated, for example, at the path tracer 501 or the rasterizer 504 based on the scene geometry;

[0243] · A depth map 803 that describes the inverse depth of the respective input pixels - this map can be generated, for example, at the path tracer 501 or the rasterizer 504 based on the scene geometry;

[0244] · A raster image 804 that describes the initial color of each pixel before the application of lighting - this image will typically be produced by texture mapping and rasterization (e.g., by the texture data path 508 in Figure 5 );

[0245] · Horizontal / vertical similarity maps that describe the similarity between horizontal / vertical pixels according to a selected similarity measure - these maps can be generated at the shader.

[0246] These maps can be generated such that each map is available to the filter at an appropriate time. The maps can be generated simultaneously.

[0247] In this example, the similarity measure is the similarity measure defined in Equation 8 above and depends on the normal and depth values of adjacent pixels. The similarity generator 805 operates on the normal and depth maps 802 and 803 to generate a horizontal similarity map 806 and a vertical similarity map 807. The similarity generator can operate at the shader of the graphics system. The horizontal filter 808 corresponds to Figure 4b the filter 402 shown in

[0248] and operates on the illumination map 801 and the horizontal similarity map 806 to form an updated illumination map 809 after filtering along the horizontal orientation (first pass). Figure 4bThe filter 403 shown in

[0249] Color can be any property related to the appearance of a pixel in a frame, including parameters describing the color of a pixel according to a certain color model (e.g., RGB or YUV tuple), one or more components of the pixel color (e.g., red channel parameter), and one or more parameters of an aspect of the color (such as brightness, saturation, luminance, lightness, chroma, or transparency) of the color. Shades of black, white, and gray are colors. Pixel values can be colors.

[0250] A pixel can be any element of a raster frame having an associated pixel value that conveys information about the appearance of the frame. A frame consists of an ordered set of pixels. For the purposes of this disclosure, a pixel need not be related to the screen resolution of the frame. For example, a frame may have a screen resolution of 1280×960 first elements, but the transparency information of the frame may be defined only at a resolution of 640×480 second elements; if the transparency information is filtered, then the transparency value of the 640×480 second elements of the frame represents the pixel value of the pixel and the second elements.

[0251] Figure 9 illustrates the operation of the filter 400 shown in Figure 4a on a sequence of pixels (e.g., a row or column) of a frame. For each pixel of the sequence, the filter forms a filtered pixel value in the left-to-right direction 902 and a filtered pixel value in the right-to-left direction 903. The corresponding filtered pixel values are formed based on the value of the received pixel (e.g., the initial color information of the pixel) and the value of the previous pixel adjusted by a similarity measure between the pixel and its previous pixel in the sequence as shown by steps 904 and 905. For example, filtering in the left-to-right direction 902 can be performed according to equations 1 to 3 such that two intermediate filtered values and are formed and combined in step 902 according to equation 3 to form the filtered value of the pixel in the left-to-right direction. Similarly, for example, filtering in the right-to-left direction 903 can be performed according to equations 4 to 6 such that two intermediate filtered values and are formed and combined in step 903 according to equation 6 to form the filtered value of the pixel in the right-to-left direction.

[0252] Filtering is performed on the received pixels 901 that can be selected from the storage of the input pixels 913 in the left-to-right direction 906. When the filtered pixel values are formed in the left-to-right direction 902, the filtered pixel values can be stored 917, for example, in the row storage 918. Each received pixel 901 is a pixel in a pixel sequence (e.g., a row or column of a frame). The pixels of the sequence can be processed in order such that a pixel in the sequence is processed and then the next pixel 912 in the sequence is received for filtering until all pixels in the sequence are filtered in the left-to-right direction.

[0253] When the filtering of the pixel sequence is completed in the left-to-right direction, the sequence can be processed in the right-to-left direction, receiving 914 the pixels of the sequence from the storage of the input pixels 913 in an appropriate order.

[0254] At step 908, the filtered pixel values formed in the left-to-right direction and the right-to-left direction are combined to form the output pixel value 909. The left-to-right filtered value of a given pixel can be read out from the row storage 918. The combination of the filtered pixel values can be performed, for example, according to equation 7. More generally, the combination can be an average (such as an arithmetic mean) of the left-to-right filtered value and the right-to-left filtered value. The output pixel value 909 can represent, for example, the color of each pixel after filtering. As discussed above, each pixel value can also include a weight that can represent a confidence measure of the pixel characteristics expressed by the initial pixel value (e.g., the initial color of the pixel as determined by path tracing).

[0255] At 920, the filter checks whether the pixel sequence is completed in the right-to-left direction (e.g., whether the filtered pixel is the last pixel in a row or column of the frame). If completed, the filter 400 starts filtering the next sequence of the frame until all sequences of the frame are filtered, and thus the filtering of the frame is completed. If the pixel sequence is not completed, the next pixel 919 in the sequence is received for filtering. When all pixels in the sequence are filtered in the right-to-left direction, the filter can move to the next sequence 911 of the frame.

[0256] In other examples, filtering in the right-to-left direction can be performed first and the result stored. Then, filtering in the left-to-right direction can be performed and the result combined with the stored result from the right-to-left filter to produce the output pixel.

[0257] As indicated by the optional pre-filter steps 915 and 916 in Figure 9 the filter 400 can perform pre-filtering on the received pixel values. The pre-filtering will now be described. The pre-filtering can be used with any configuration of the filter 400, including with any example of the filter 400 described herein.

[0258] Prefiltering

[0259] In an example of forming a similarity measure d of pixel pairs based on the similarity of these pixel values, it can be advantageous to pre-filter the pixel values before using them to form the similarity measure. The pixel values can be, for example, the luminance or color measure of the pixels (wherein the color of the pixel can refer to one or more of its color components). The use of pre-filtering improves noise reduction at the shadow edges at the expense of some blurring of sharp shadows. Additionally, the similarity measure of pixel pairs can be formed based on the similarity of other pixel characteristics, such as the difference measure of the depth in the 3D scene to the respective surfaces represented by the pixels in the frame of the scene and / or the measure of the angle between the normals of the respective surfaces represented by these pixels in the frame of the 3D scene).

[0260] In the above example, the first recursive filtering of the pixels in the first direction is performed according to Equation 1:

[0261]

[0262] where α1 is a constant, and d n is the similarity measure between the nth pixel and the (n - 1)th pixel. Similar filtering is performed using the similarity measures according to Equations 2, 4, and 5. Each such filtering will benefit from using a similarity measure that is calculated using the pre-filtered pixel characteristics in the manner described below.

[0263] To form the similarity measure d based on the luminance or color similarity of these pixel values n , the nth pixel and the (n - 1)th pixel are compared. The (n - 1)th pixel value can be used as the output of a recursive filter for the previous pixel operations in a filtered form, but no equivalent filtered form of the nth pixel is available. If the nth pixel is not filtered, the noise present in the initial pixel value may cause noise in the similarity measure.

[0264] To solve the noise problem in the initial pixel values, in this exemplary example, the pixel value C n of the current nth pixel can be pre-filtered to eliminate the noise present in this initial pixel value. The pre-filtered pixel value is used in the formation of the similarity measure and can also optionally be used by the recursive filter in place of the initial unfiltered pixel value C n。The pre-filtering can be performed in any suitable manner and on one or more components of the pixel values or on the parameters representing the pixel values when calculating the similarity measure. If the similarity measure is formed by calculating the distance between the color values of pixel pairs in a color space, then pre-filtering can be performed on one or more components of the current pixel value contributing to that distance; if the similarity measure is formed by calculating the luminance difference between pixel pairs, then pre-filtering can be performed on at least one parameter representing the luminance of the current pixel (e.g., the luminance component of the current pixel value). The pre-filtering can be performed only on the parameters representing the characteristics of the pixel values or on those components of the current pixel value that are used to calculate the similarity measure. It will be understood that the reference to pixel pre-filtering here is a reference to pre-filtering at least the components / parameters of the pixel or representing the pixel that are used to calculate the similarity measure.

[0265] Pre-filtering the pixel characteristics using the calculation of the similarity measure can be performed at any suitable type of filter (such as a Gaussian filter or an exponential filter). Preferably, the filter used for pre-filtering the current pixel has a small filtering radius. For example, it has been found that a filtering radius of 3×3 pixels provides good performance. The radius of the filter is a measure of the width of the filtering function describing the filter. For example, the radius can be a measure of the region of each pixel or several pixels contributing to a given filtered pixel value. The filtering radius can be expressed in any suitable manner (including: as a measure of the width of the filtering function represented by the filter (e.g., the full width at half maximum of the filtering function); and as a measure of the maximum distance between two pixels in the frame contributing to the filtered value generated for the object pixel).

[0266] The pre-filtering can include taking the simple average (e.g., the arithmetic mean) of the relevant pixel values or characteristics of the pixels located within the filtering radius (e.g., the nine pixels in a 3×3 square grid with the current pixel at the center). The pre-filtering can include calculating the weighted average of the pixels located within the filtering radius, where the weights of the pixels are defined according to a certain predetermined distribution (e.g., the current pixel is assigned a higher weight than its 8 adjacent pixels in the 3×3 square grid to approximate a Gaussian or other filtering envelope concentrated on the current pixel).

[0267] Alternatively, the pre-filtering can be performed by using a recursive filter similar to the filter described by Equation 1 that has a short exponential response (i.e., has a small value) and does not use the similarity measure. Such a recursive filter performs pre-filtering along one dimension, unlike the above-described two-dimensional (e.g., 3×3) filters.

[0268] In the above examples, an additional filter can be set up to perform pixel pre-filtering. Such an additional filter can be set, for example, at Figure 4a or Figure 4b the filter 400 of

[0269] In the example described above regarding Figure 4a the filter is set to have a single filter pair (filter pair 408 and 409 of recursive filter 402). In the example described above regarding Figure 4b the filter is set to have filter pairs (filter pair 408 and 409 of recursive filter 402, and filter pair 410 and 411 of recursive filter 403) that operate in each direction. It can be advantageous to use Figure 4a or Figure 4b either of the structures shown and perform pre-filtering without introducing any additional filters. This minimizes the complexity of the design. This will now be described in more detail regarding Figure 4b this.

[0270] For example, the first filter of each filter pair of recursive filters 402 and 403 (e.g., filters 408 and 410) can be used to perform pre-filtering of pixels, and the second filter of each filter pair of recursive filters 402 and 403 is used to perform recursive filtering (e.g., filter 409 can perform recursive filtering according to equation 1, and filter 411 can perform recursive filtering according to equation 4). The first filter set to perform pre-filtering can be configured to have a smaller filter radius compared to the second filter. For example, the filter radius of the first filter can be at least one order of magnitude smaller than the filter radius of the second filter. To preserve details in a frame (e.g., a light map or a raster image), the filter radius of the first filter can be no more than 20 pixels, no more than 15 pixels, no more than 10 pixels, no more than 5 pixels, no more than 3 pixels, or no more than 2 pixels. The second filter can have an infinite filter radius (e.g., the second filter can be an exponential filter (this filter with a decay of 1 will represent a simple accumulator)).

[0271] The filtered value of each pixel (e.g., the output of filter logic 404) can be calculated as a simple average of the filtered outputs of each pixel from recursive filters 402 and 403 in the first and second directions (simply subtracting the pixel value itself that has been counted twice). In the example of using pixel pre-filtering, the pixel weights as described above that allow the filter to take into account the confidence of the pixel value may or may not be used. If pixel weights are used, the calculation of the average becomes a simple sum according to equation 7 above. This is because the weight values accumulate on the pixels during filtering, just as the pixel values themselves do, and the homogeneous color vector {r, g, b, w} represents the color vector {r / w, g / w, b / w} in color space.

[0272] If the first filter has a small filtering radius, the filtering structure can produce good performance, especially for noisy images - this is true even if the filtering is only performed once in each direction and even when the second filter radius is infinite.

[0273] The pre-filtering can be performed in two dimensions (e.g., on a 3×3 square grid) or in one dimension (e.g., on a 3-pixel row along the direction in which the filtering is to be performed).

[0274] Combined with calculating a similarity measure based on the similarity of the normal and / or depth values of adjacent pixels, further calculating the similarity measure based on the similarity of the pixel values of adjacent pixels enables filtering to be performed while preserving the shadow boundaries and edges of different oriented surfaces and / or boundaries that separate the depths of different scenes. The use of pre-filtering further improves noise reduction near the shadow boundaries.

[0275] In the example shown in performing the pre-filtering steps 915 and 016 Figure 9 when receiving the pixels of the sequence 901, the filter 400 is also configured to pre-filter the received pixel values 901 and 914. The pre-filtered pixel values are used to generate the directional similarity measures for adjusting the "before" pixels at 904 and 905. In Figure 9 the specific example shown, at 902 and 903, the filtering can be performed using the initial pixel values of the received pixels instead of their pre-filtered pixel values. This contributes to maintaining the details in the frame processed by the filter while isolating the output of the recursive filter from the noise of the pixel values. Alternatively, the filtering steps 902 and 903 can be performed using the pre-filtered pixel values. The pixel filtering at steps 902 and 903 can be the recursive filtering as described above.

[0276] Figure 4a and Figure 4b the filters, Figure 5 the graphics systems, and Figure 8 the structures are shown as including several functional blocks. This is only illustrative and is not intended to define a strict division between the different logical elements of such entities. The functional blocks can be arranged in any suitable manner. It is to be understood that the intermediate values described here as being formed by the filters do not need to be physically generated by the filters at any point and can only represent logical values that facilitate the description of the processing performed by the filters between their inputs and outputs.

[0277] Generally, any of the above functions, methods, technologies, or components can be implemented in software, firmware, hardware (e.g., fixed logic circuitry), or any combination thereof. The terms "module", "function", "component", "element", "unit", "block", and "logic" are used herein generically to denote software, firmware, hardware, or a combination thereof. In the case of a software implementation, a module, function, component, element, unit, block, or logic represents program code that, when executed on a processor, performs a specified task. The algorithms and methods described herein can be executed by one or more processors that execute code causing the processor to perform the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical discs, flash memory, hard disk storage, and other storage devices that can use magnetic, optical, and other technologies to store instructions or other data and that are accessible by a machine.

[0278] As used herein, the terms computer, program code, and computer-readable instructions refer to any kind of executable code for a processor (including code expressed in machine language, interpreted language, or script language). Executable code includes binary code, machine code, bytecode, code defining an integrated circuit (such as a hardware description language or a netlist), and code expressed in a programming language (such as C, Java, or OpenCL). Executable code can be, for example, any kind of software, firmware, script, module, or library that, when appropriately executed, processed, interpreted, compiled, or executed at a virtual machine or other software environment, causes a processor of a computer system that supports the executable code to perform the tasks specified by the code.

[0279] A processor, computer, or computer system can be any kind of device, machine, or dedicated circuit or a collection or part thereof having processing capabilities such that it can execute instructions. A processor can be any kind of general-purpose or special-purpose processor (such as a CPU, GPU, system-on-chip, state machine, media processor, application-specific integrated circuit (ASIC), programmable logic array, field-programmable gate array (FPGA), etc.). A computer or computer system can include one or more processors.

[0280] Code defining an integrated circuit can define the integrated circuit in any manner, including as a netlist, code for constructing a programmable chip, and as a hardware description language defining the integrated circuit at any level (including as register transfer level (RTL) code, as a high-level circuit representation such as Verilog or VHDL, and as a low-level circuit representation such as OASIS and GDSII). When processed at a computer system suitably configured to generate a display of the integrated circuit, the code defining the integrated circuit can cause the computer system to generate a display of the integrated circuit represented by the code. Such a computer system can cause generation of a display of the integrated circuit, for example, by providing output for controlling a machine configured to manufacture the integrated circuit or an intermediate representation of the integrated circuit such as a photolithographic mask).

[0281] A higher-level representation (such as RTL) that logically defines the integrated circuit can be processed at a computer system configured to generate a display of the integrated circuit in the context of a software environment that includes definitions of circuit elements and rules for combining those elements for the purpose of generating a display of the integrated circuit so defined by the representation.

[0282] Similar to the case where ordinary software is executed at a computer system to define a machine, one or more intermediate user steps (e.g., providing commands, variables, etc.) may be required for a computer system configured to generate an image of the integrated circuit to execute the code defining the integrated circuit to generate a display of the integrated circuit.

[0283] The applicant hereby discloses each of the independent features described herein and any combination of two or more such features in isolation, to the extent that such feature or combination can be carried out based on the present specification as a whole in view of the common general knowledge of a person skilled in the art, regardless of whether such feature or combination of features solves any of the problems disclosed herein. In view of the foregoing description, it will be apparent to a person skilled in the art that various modifications can be made within the scope of the present invention.

Claims

1. A pixel filter, the pixel filter comprising: An input configured to receive a sequence of pixels, each pixel having an associated pixel value; A filter module configured to perform a first recursive filtering operation on the sequence of pixels in a first direction to form a first filtered pixel value for each pixel, wherein the first recursive filtering operation forms the first filtered pixel value for a given pixel based on the pixel value at the given pixel and the filtered pixel value of a previous pixel in the first direction before the given pixel, and the filtered pixel value of the previous pixel is adjusted by a similarity measure between data related to the given pixel and the previous pixel that is not present in the pixel values of the given pixel and the previous pixel, wherein the similarity measure is formed depending on one or more of the following: A measure of the difference in scene depth in a 3D scene of the respective surfaces represented by the given pixel and the previous pixel in a frame of the 3D scene; and A measure of the angle between the normal vectors of the respective surfaces represented by the given pixel and the previous pixel in a frame of the 3D scene; and Filter logic configured to generate a filter output for each pixel of the sequence based on the first filtered pixel value.

2. The pixel filter according to claim 1, wherein: The filter module is further configured to perform a second recursive filtering operation on the sequence of pixels in a second direction to form a second filtered pixel value for each pixel, wherein the second recursive filtering operation forms the second filtered pixel value for a given pixel based on the pixel value at the given pixel and the filtered pixel value of a previous pixel in the second direction before the given pixel, and the filtered pixel value of the previous pixel is adjusted by a similarity measure between data related to the given pixel and the previous pixel that is not present in the pixel values of the given pixel and the previous pixel; and The filter logic is configured to, for each pixel of the sequence, combine the first filtered pixel value and the second filtered pixel value for the pixel formed by the first recursive filtering operation and the second recursive filtering operation to generate a filter output for the pixel.

3. The pixel filter according to claim 2, wherein The second direction is opposite to the first direction.

4. The pixel filter according to any one of claims 1 to 3, wherein, The data related to the given pixel and the previous pixel includes basic information about the 3D scene such that the similarity measure indicates whether the given pixel and the previous pixel belong to a common feature of the scene.

5. The pixel filter according to any one of claims 1 to 3, wherein The similarity measure depends on the product of the following measures: i) The difference between the scene depths represented by the given pixel and the previous pixel, and ii) The angle between the normal vectors of the respective surfaces represented by the given pixel and the previous pixel.

6. The pixel filter according to any one of claims 1 to 3, wherein, The scene depth is represented as 1 / d, where d is the depth in the 3D scene.

7. The pixel filter according to any one of claims 1 to 3, wherein, The similarity measure is formed depending on more than two scene depth values.

8. The pixel filter according to claim 7, wherein A discontinuity in the rate of change of depth values represents different surface orientations.

9. The pixel filter according to any one of claims 1 to 3, wherein, The similarity measure is additionally formed depending on the similarity of one or more of the luminance and color of the given pixel and the previous pixel.

10. The pixel filter according to claim 2 or 3, wherein, The first recursive filtering operation and the second recursive filtering operation form weights for the first filtered pixel value and the second filtered pixel value, and the weights depend on a confidence measure of pixel values contributing to the filtered pixel values.

11. The pixel filter according to claim 10, wherein, Combining the first filtered pixel value and the second filtered pixel value includes calculating a weighted average of the first filtered pixel value and the second filtered pixel value using the formed weights.

12. The pixel filter according to any one of claims 1 to 3, wherein, The first recursive filtering operation is configured to adjust the filtered pixel value of the previous pixel such that, compared to when the similarity measure indicates low similarity, when the similarity measure indicates high similarity, the filtered pixel value of the previous pixel contributes more to the first filtered pixel value of the given pixel.

13. A method for filtering a sequence of pixels, each pixel having an associated pixel value, the method comprising the following steps: Recursively filtering the sequence of pixels in a first direction by performing the following operations on each pixel to form a first filtered pixel value for each pixel: - Reading the filtered pixel value of the previous pixel in the sequence with respect to the first direction; - Adjusting the filtered pixel value of the previous pixel in the sequence with a similarity measure between data related to the given pixel and the previous pixel, the data not being present in the pixel values of the given pixel and the previous pixel, wherein the similarity measure is formed depending on one or more of the following: A measure of the difference in scene depth in the 3D scene of each surface represented by the given pixel and the previous pixel in a frame of the 3D scene; and A measure of the angle between the normal vectors of each surface represented by the given pixel and the previous pixel in a frame of the 3D scene; and - Forming the first filtered pixel value of the given pixel based on the pixel value of the given pixel and the adjusted filtered pixel value of the previous pixel; and Generating a filter output for the given pixel based on the first filtered pixel value.

14. The method according to claim 13, the method further comprising: Recursively filtering the sequence of pixels in a second direction by performing the following operations on each pixel to form a second filtered pixel value for each pixel: - Reading the filtered pixel value of the previous pixel in the sequence with respect to the second direction; - Adjusting the filtered pixel value of the previous pixel in the sequence with a similarity measure between data related to the given pixel and the previous pixel, the data not being present in the pixel values of the given pixel and the previous pixel; and - Forming the second filtered pixel value of the given pixel based on the pixel value of the given pixel and the adjusted filtered pixel value of the previous pixel; And Combining the first filtered pixel value and the second filtered pixel value formed for each pixel of each sequence to generate a filter output for the pixel.

15. The method according to claim 13 or 14, wherein, The data related to the given pixel and the previous pixel includes basic information about the 3D scene such that the similarity measure indicates whether the given pixel and the previous pixel belong to a common feature of the scene.

16. The method according to claim 13 or 14, wherein, The similarity measure depends on the product of the following measures: i) the difference between the scene depths represented by the given pixel and the previous pixel, and ii) the angle between the normal vectors of the corresponding surfaces represented by the given pixel and the previous pixel.

17. The method according to claim 13 or 14, wherein The scene depth is represented as 1 / d, where d is the depth in the 3D scene.

18. The method according to claim 13 or 14, wherein, The similarity measure is formed depending on more than two scene depth values.

19. The method according to claim 18, wherein, A discontinuity in the rate of change of the depth value represents different surface orientations.

20. A non-transitory computer-readable storage medium having computer-readable instructions stored thereon that, when executed at a processor, cause the processor to perform a method for filtering a sequence of pixels, each pixel having an associated pixel value, the method comprising the steps of: Recursively filtering the sequence of pixels in a first direction by performing the following operations on each pixel to form a first filtered pixel value for each pixel: - Reading the filtered pixel value of the previous pixel in the sequence with respect to the first direction; - Adjusting the filtered pixel value of the previous pixel in the sequence with a similarity measure between data related to the given pixel and the previous pixel, the data not being present in the pixel values of the given pixel and the previous pixel, wherein the similarity measure is formed depending on one or more of the following: a measure of the difference in scene depth in the 3D scene of each surface represented by the given pixel and the previous pixel in a frame of the 3D scene; and a measure of the angle between the normal vectors of each surface represented by the given pixel and the previous pixel in a frame of the 3D scene; and - Forming the first filtered pixel value for the given pixel based on the pixel value of the given pixel and the adjusted filtered pixel value of the previous pixel; and Generating a filter output for the given pixel based on the first filtered pixel value.

Images