A method, storage medium, and device for screen space heat map rendering

Abstract

The application discloses a kind of screen space thermodynamic diagram rendering method, storage medium and equipment, the method includes: obtaining scene hot spot data and scene buffer data, and setting blue noise texture;Screen space is carried out grid grouping, obtain multiple grid units, utilize computing shader to preprocess scene hot spot data, and distribute to corresponding grid unit;Utilize computing shader to carry out world coordinate reconstruction to each pixel of screen space, and sample scene hot spot data in the grid unit where current pixel is located, obtain current pixel color;Utilize scene buffer data and blue noise texture, mix current pixel color and scene color and output to user display page.The application is based on the preprocessing of computing shader acceleration heat map distribution, effectively eliminates invalid data, optimizes final coloring calculation amount, so that final thermodynamic diagram performance is more accurate.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, storage medium, and apparatus for rendering screen space heatmaps based on computational shaders. Background Technology

[0002] Screen space heatmaps reconstruct 3D scenes from screen space data and sample hotspot data to create a heatmap effect. Currently, screen space heatmap rendering primarily involves sampling and coloring scene hotspot data using post-processor shaders. However, existing screen space heatmaps are limited by rendering performance issues, typically restricting the number of hotspots to around one thousand. This results in the heatmap not fitting well onto the scene model, leading to reduced accuracy. Summary of the Invention

[0003] In view of this, the present invention proposes a method, storage medium and device for screen space heatmap rendering based on computation shaders. The method utilizes computation shaders to group hotspot data in screen space and record each group of hotspot data, thereby reducing the number of hotspot samples per pixel and significantly improving the efficiency of screen space heatmap rendering.

[0004] The first aspect of this invention provides a method for rendering a screen space heatmap. The method includes: acquiring scene hotspot data and scene buffer data, and setting a blue noise texture; grouping the screen space into grids to obtain multiple grid cells; preprocessing the scene hotspot data using a computational shader and allocating it to the corresponding grid cells; reconstructing the world coordinates of each pixel in the screen space using the computational shader, and sampling the hotspot data in the grid cell where the current pixel is located to obtain the current pixel color; and mixing the current pixel color and the scene color using the scene buffer data and the blue noise texture and outputting it to the user display page.

[0005] Furthermore, the step of acquiring scene hotspot data and scene buffer data, and setting blue noise texture includes: acquiring user-defined scene hotspot data and storing it in the texture of the graphics processor, wherein the scene hotspot data includes hotspot location, hotspot radiation radius, and hotspot radiation weight; setting downsampling parameters and passing them to the blue noise texture; acquiring scene buffer data provided by the renderer, wherein the scene buffer data includes scene color buffer and scene depth buffer; constructing the graphics processor's counting buffer and grouping container; and setting the transformation matrix from the clipping coordinate system to the world coordinate system.

[0006] Furthermore, the step of dividing the screen space into multiple grid units includes: dividing the screen space into multiple grid units at a set resolution, with each grid unit having a grid number.

[0007] Furthermore, the step of preprocessing the scene hotspot data using a computational shader and allocating it to the corresponding grid cells includes: sampling the hotspot locations and hotspot radiation radii in the scene hotspot data to construct a detection model; based on the detection model, performing intersection detection on each grid cell, and allocating the scene hotspot data to the corresponding grid cells according to the detection results.

[0008] Furthermore, the method for constructing the detection model includes: sampling the hotspot positions and hotspot radiation radii from the preprocessed scene hotspot data, transforming the hotspot positions to screen space to obtain the hotspot positions within the screen space; generating a polyhedron based on the hotspot positions and hotspot radiation radii from the scene hotspot data, and transforming all vertices of the polyhedron to screen space; calculating the maximum distance from all vertices of the polyhedron within screen space to the hotspot positions within screen space; and constructing the detection model based on the hotspot positions and maximum distances within screen space.

[0009] Furthermore, the intersection detection of each grid cell based on the detection model includes: performing intersection detection on each grid cell based on the detection model to determine whether the detection model and the grid cell have an intersection point; if there is an intersection point, incrementing the buffer of the item with the index and grid index of the grid cell in the counting buffer by one, and storing the current hot spot in the grouping container.

[0010] Furthermore, the process of reconstructing world coordinates for each pixel in screen space using a compute shader and sampling scene hotspot data in the grid cell where the current pixel is located to obtain the current pixel color includes: sampling the scene depth buffer; reconstructing the world coordinates corresponding to the current pixel using the scene depth buffer and the transformation matrix from the clipping coordinate system to the world coordinate system; calculating the grid number of the grid cell where the current pixel is located using the screen space coordinates of the current pixel; obtaining the scene hotspot data contained in the current grid from the grouping container; traversing each scene hotspot data contained in the current grid; calculating the current pixel radiance value corresponding to each scene hotspot; accumulating the current pixel radiance values ​​corresponding to all scene hotspots to obtain the current pixel radiance total; calculating the color difference of the current pixel using the current pixel radiance total to obtain the current pixel color; and setting the current pixel radiance total as the current pixel transparency.

[0011] Furthermore, the step of using scene buffer data and blue noise texture to mix the current pixel color and scene color and output it to the user display page includes: using the transparency of the current pixel color to perform linear interpolation calculation on the scene color buffer and the current pixel color to obtain a new color; using blue noise texture to process the new color, and outputting the processed color to the user display page.

[0012] A second aspect of the present invention provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the screen space heatmap rendering method described above.

[0013] A third aspect of the present invention provides a screen space heatmap rendering apparatus, the apparatus comprising: a processor and a memory; the memory storing a computer-readable program executable by the processor; the processor executing the computer-readable program implements the steps of the screen space heatmap rendering method as described above.

[0014] The aforementioned screen space heatmap rendering method, based on computational shaders to accelerate the preprocessing of heatmap distribution, effectively removes invalid data and optimizes the final shading computation to about 0.1% of the existing technology, making the final heatmap more accurate. Attached Figure Description

[0015] For illustrative and not limiting purposes, the invention will now be described with reference to preferred embodiments thereof, and in particular the accompanying drawings.

[0016] Figure 1 This is a flowchart of a screen space heatmap rendering method proposed in an embodiment of the present invention. Detailed Implementation

[0017] To better understand the above-mentioned objects, features, and advantages of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0018] Numerous specific details are set forth in the following description to provide a thorough understanding of the invention. The described embodiments are merely some, not all, of the embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0020] The method for rendering screen space heatmaps according to embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Figure 1 This is a flowchart of a screen space heatmap rendering method proposed in an embodiment of the present invention. Please refer to... Figure 1 The method for rendering the screen space heatmap includes the following steps:

[0021] S100 acquires scene hotspot data and scene buffer data, and sets blue noise texture. In some embodiments, step S100 is implemented as follows:

[0022] S110: Obtain the scene hotspot data H set by the user and store it in the texture of the graphics processor.

[0023] In this embodiment of the invention, the scene hotspot data H includes the hotspot location Hp, the hotspot radiation radius Hr, and the hotspot radiation weight Hw. The scene hotspot data H is stored in the texture of the graphics processing unit (GPU). In some embodiments, the hotspot location Hp is stored in the texture Tp of the graphics processing unit (GPU); the hotspot radiation radius Hr and the hotspot radiation weight Hw are stored in the texture Trw of the graphics processing unit (GPU).

[0024] S120, set the downsampling parameters and pass in the blue noise texture B.

[0025] Among them, the blue noise texture B is used to perturb the downsampling sawtooth.

[0026] The downsampling parameter is the sampling scaling factor of the blue noise texture B.

[0027] S130: Obtain scene buffer data provided by the renderer.

[0028] In this embodiment of the invention, the scene buffer data includes a scene color buffer C and a scene depth buffer D.

[0029] In some embodiments, the scene color buffer C and the scene depth buffer D are obtained through the renderer interface.

[0030] S140 constructs the counting buffer Cb and the grouping container Gb for the graphics processing unit (GPU).

[0031] In this embodiment of the invention, a counting buffer Cb and a grouping container Gb of the graphics processing unit (GPU) are constructed. The counting buffer Cb and the grouping container Gb are unordered access views (UAVs) created through the graphics interface (DirectX 11 / 12).

[0032] S150 sets the transformation matrix T from the clipping coordinate system to the world coordinate system.

[0033] The transformation matrix T from the clipping coordinate system to the world coordinate system is used to convert the coordinate values ​​in screen space to world space for the reconstruction of world coordinates in 3D scenes.

[0034] In this embodiment of the invention, hotspot data of the scene is first acquired, blue noise texture is set, and scene color buffer and scene depth buffer are acquired. A counting buffer and grouping container are constructed. Finally, camera parameters and the transformation matrix from the clipping coordinate system to the world coordinate system are set to facilitate subsequent sampling, preprocessing and rendering of hotspot data.

[0035] S200 groups the screen space into meshes to obtain multiple mesh cells. The computation shader preprocesses the scene hotspot data and assigns it to the corresponding mesh cells. In some embodiments, step S200 is implemented as follows:

[0036] S210, the screen space is divided into grids to obtain multiple grid cells T.

[0037] In this embodiment of the invention, when grouping the screen space into grids, taking a resolution of 1280x720 as an example, the screen space is divided into 40x23 grid units T, each of 32x32 pixels. Each grid unit T has a grid upper left corner Tmin, a grid lower right corner Tmax, and a grid number TId.

[0038] S220 uses a compute shader to generate thread groups of 32x32 compute units each to preprocess scene hotspot data and allocate them to the corresponding mesh units.

[0039] Computation shaders are programs that run outside the rendering pipeline on a graphics card and are used for large-scale parallel computing. In some embodiments, the specific implementation of preprocessing scene hotspot data using computation shaders is as follows:

[0040] S221, sample the hotspot location Hp and hotspot radiation radius Hr from the hotspot data of the sampling scene, and construct a detection model.

[0041] In this embodiment of the invention, the hotspot position Hp and hotspot radiation radius Hr in the preprocessed scene hotspot data are sampled, the hotspot position Hp is transformed to screen space to obtain the hotspot position Hp' in screen space, a polyhedron is generated based on the hotspot position Hp' and the hotspot radiation radius Hr, and all vertices of the polyhedron are transformed to screen space. The maximum distance R from all vertices of the polyhedron in screen space to the hotspot position Hp' is calculated, and a detection model is constructed based on the hotspot position Hp' and the maximum distance R.

[0042] The polyhedron can be a hexahedron, such as an AABB (axis-aligned bounding box). The detection model can be a circle (Ch). In other embodiments, the polyhedron can also be other polyhedral structures, and the detection model can also be other shapes.

[0043] The following example uses a polyhedron as AABB (axis-aligned bounding box) and a circle Ch as the detection model to illustrate the detection model construction method.

[0044] A hotspot position Hp is sampled from the GPU texture Tp, and the hotspot radiation radius Hr is sampled from the GPU texture Trw. The hotspot position Hp is then transformed to screen space to obtain the corresponding hotspot position Hp' in screen space. An AABB (axis-aligned bounding box) is generated in world coordinates, centered on the sampled hotspot position Hp and with the sampled hotspot radiation radius Hr as the radius. The eight vertices of the AABB are then transformed to screen space. The maximum distance R between the eight vertices of the AABB and the transformed hotspot position Hp' in screen space is calculated. A circle Ch is then constructed, centered on the transformed hotspot position Hp' and with the maximum distance R as the radius.

[0045] S222, based on the detection model, performs intersection detection on each grid cell T, and allocates scene hotspot data to the corresponding grid cells according to the detection results. In some embodiments, taking a circular Ch as an example of the detection model, the specific implementation of intersection detection for grid cells T is as follows: Perform an intersection test on the circle Ch and each grid cell T to determine whether the circle Ch and the grid cell T have an intersection point; If there is an intersection, increment the buffer of the item with the index and grid number TId in the counting buffer Cb by one, and store the current scene hotspot data in the grouping container Gb.

[0046] If there is no intersection, it means that the hot data in this scenario does not belong to the current group container Gb and does not need to be stored in the group container Gb.

[0047] In this embodiment of the invention, a compute shader is used to remove invalid hotspots, reducing the computational load on invalid hotspot data; a compute shader is also used to perform screen space grouping preprocessing on scene hotspot data, significantly reducing the computational load per pixel for coloring, thereby accelerating rendering.

[0048] S300 uses a computational shader to reconstruct world coordinates for each pixel in screen space and samples scene hotspot data in the mesh cell T where the current pixel is located to obtain the current pixel color and the current pixel transparency.

[0049] In this embodiment of the invention, a computational shader is used to reconstruct the world coordinates of each pixel in the screen space, and the scene hotspot data in the grid cell T where the current pixel is located is sampled to obtain the current pixel color Pc and the current pixel transparency Pt. In some embodiments, step S300 is implemented as follows:

[0050] S310, sample the scene depth buffer D, and use the scene depth buffer D and the transformation matrix T from the clipping coordinate system to the world coordinate system to reconstruct the world coordinates P corresponding to the current pixel.

[0051] In this embodiment of the invention, firstly, the scene depth buffer D obtained in sampling step S130 is used to reconstruct the world coordinates P corresponding to the current pixel using the scene depth buffer D and the crop coordinate system to world coordinate system transformation matrix T set in step S150.

[0052] In this embodiment of the invention, the world coordinates P corresponding to the current pixel are reconstructed to calculate the distance to the hotspot and determine whether the world coordinates point corresponding to the current pixel is affected by the hotspot.

[0053] S320: Calculate the grid number TId of the grid cell where the current pixel is located using the screen space coordinates of the current pixel, and obtain the scene hotspot data H' contained in the current grid from the grouping container Gb.

[0054] In this embodiment of the invention, the grid number TId of the current pixel is calculated using the screen space coordinates of the current pixel, and the scene hotspot data H' contained in the current grid is obtained from the grouping container Gb of the graphics processor (GPU). The method for calculating the grid number TId of the current pixel using the screen space coordinates of the current pixel is as follows: With a 32x32 grid, where X is the horizontal coordinate of the current pixel and Y is the vertical coordinate of the current pixel, the corresponding Tid for that pixel is: Tid.x = floor(X / 32) Tid.y = floor(Y / 32)

[0055] The floor function performs a floor operation.

[0056] In this embodiment of the invention, the computational cost of the final coloring is optimized, resulting in a more accurate final heatmap representation.

[0057] S330, traverse each scene hotspot data H' contained in the current grid, calculate the current pixel radiation value f corresponding to each scene hotspot, and sum up the current pixel radiation values ​​f corresponding to all scene hotspots to obtain the current pixel radiation total F.

[0058] In some embodiments, each scene hotspot data H' contained in the current grid includes hotspot location Hp, hotspot radius Hr, and hotspot radiance Hw.

[0059] In this embodiment of the invention, the hotspot position Hp, hotspot radius Hr, and hotspot radiance Hw of each scene hotspot data H' contained in the current grid are traversed. The inverse distance interpolation algorithm is used to calculate the current pixel radiance value f corresponding to each scene hotspot contained in the current grid, and the current pixel radiance values ​​f corresponding to all hotspots contained in the current grid are accumulated to obtain the current pixel radiance total F.

[0060] S340: Calculate the color difference of the current pixel using the current pixel's total radiation F to obtain the current pixel's color Pc, and set the current pixel's total radiation F to the current pixel's transparency Pt.

[0061] In this embodiment of the invention, the color of the current pixel is calculated by using the difference between the current pixel's total radiation F and the current pixel's color from green and blue to red, and the current pixel's total radiation F is set as the current pixel's transparency Pt.

[0062] In this embodiment of the invention, computational shader-based preprocessing acceleration is employed, resulting in a hundredfold increase in the number of hotspots compared to existing technologies.

[0063] The S400 uses the scene color buffer C and the blue noise texture B to mix the current pixel color and the scene color and output them to the user's display page.

[0064] In this embodiment of the invention, the scene color buffer C obtained in step S130 and the blue noise texture B set in step S120 are used in the pixel shader to mix the current pixel color and the scene color and output the user display page. In some embodiments, step S400 is implemented as follows:

[0065] S410: Using the current pixel color transparency Pt, perform linear interpolation calculation on the scene color buffer C and the current pixel color Pc to obtain a new color C'.

[0066] In this embodiment of the invention, the scene color buffer C obtained in step S130 and the current pixel color Pc are linearly interpolated using the current pixel color transparency Pt to obtain a new color C'.

[0067] The S420 uses blue noise texture B to process the new color C' and outputs the processed color C' to the user's display page.

[0068] In this embodiment of the invention, the blue noise texture B set in step S120 is used to perturb the color C' in order to eliminate the jaggedness of downsampling and output it to the user display page. The method for processing color C' using blue noise texture B is as follows:

[0069] The red and green channels of the blue noise texture B are sampled using the current screen coordinates as the two-dimensional offset direction Df (perturbation direction); before interpolating the scene color C and the current pixel color Pc, the current pixel is offset, and the color of the corresponding pixel position after the offset is used to quickly eliminate the jagged edges.

[0070] The aforementioned screen space heatmap rendering method, based on computational shaders to accelerate the preprocessing of heatmap distribution, effectively removes invalid data and optimizes the final shading computation to about 0.1% of the existing technology, making the final heatmap more accurate.

[0071] The screen space heatmap rendering method described above uses a compute shader to group scene hotspot data in screen space and records each group of scene hotspot data, effectively reducing the number of final hotspot samples per pixel and significantly improving the efficiency of screen space heatmap rendering.

[0072] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps in the screen space heatmap rendering method as described in any of the above technical solutions.

[0073] This invention also provides an apparatus for rendering screen space heatmaps, comprising: a processor and a memory; the memory storing a computer-readable program executable by the processor; the processor executing the computer-readable program implements the steps of the method for rendering screen space heatmaps as described in any of the above technical solutions.

[0074] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0075] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for rendering a screen space heatmap, characterized in that, include: Acquire scene hotspot data and scene buffer data, and set blue noise texture; wherein, the scene hotspot data includes hotspot location, hotspot radiation radius, and hotspot radiation weight; the scene buffer data includes scene color buffer and scene depth buffer; The screen space is divided into grids to obtain multiple grid cells. The scene hotspot data is preprocessed using a compute shader and then assigned to the corresponding grid cells. The step of preprocessing scene hotspot data using computational shaders and allocating it to corresponding mesh cells includes: Based on the hotspot locations and radiation radii in the hotspot data of the sampling scene, a detection model is constructed. Intersection detection is performed on each grid cell based on the detection model, and scene hotspot data is allocated to the corresponding grid cells according to the detection results; The method for constructing the detection model includes: The hotspot locations and hotspot radiation radii in the preprocessed scene hotspot data are sampled and transformed into screen space to obtain the hotspot locations within the screen space. Based on the hotspot locations and hotspot radiation radii in the scene hotspot data, a polyhedron is generated, and all vertices of the polyhedron are transformed to screen space. Calculate the maximum distance from all vertices of a polyhedron in screen space to the hotspot locations in screen space; A detection model is constructed based on the hotspot locations and maximum distances within the screen space; The intersection detection of each grid cell based on the detection model includes: Based on the detection model, an intersection test is performed on each grid cell to determine whether the detection model and the grid cell have an intersection point. If there is an intersection, increment the buffer of the item whose index and grid cell index are in the counting buffer by one, and store the current hot spot in the group container; The world coordinates of each pixel in screen space are reconstructed using a compute shader, and the scene hotspot data in the grid cell where the current pixel is located are sampled to obtain the color of the current pixel. Using the scene buffer data and the blue noise texture, the current pixel color and the scene color are mixed and output to the user display page.

2. The method for rendering a screen space heatmap according to claim 1, characterized in that, The steps of acquiring scene hotspot data and scene buffer data, and setting blue noise texture include: Retrieve user-defined scene hotspot data and store it in the graphics processor's texture. Set the downsampling parameters and input the blue noise texture; Retrieve scene buffer data provided by the renderer. Construct a counting buffer and grouping container for the graphics processor; Set the transformation matrix from the clipping coordinate system to the world coordinate system.

3. The method for rendering a screen space heatmap according to claim 2, characterized in that, The process of grouping the screen space into grids to obtain multiple grid cells includes: The screen space is divided into multiple grid cells based on a set resolution, and each grid cell has a grid number.

4. The method for rendering a screen space heatmap according to claim 2, characterized in that, The process involves using a computational shader to reconstruct world coordinates for each pixel in screen space and sampling scene hotspot data within the grid cell containing the current pixel to obtain the current pixel color, including: The scene depth buffer is sampled, and the world coordinates corresponding to the current pixel are reconstructed using the scene depth buffer and the transformation matrix from the clipping coordinate system to the world coordinate system. The grid number of the grid cell where the current pixel is located is calculated using the screen space coordinates of the current pixel, and the scene hotspot data contained in the current grid is obtained from the grouping container; Iterate through each scene hotspot data contained in the current grid, calculate the current pixel radiation value corresponding to each scene hotspot, and sum up the current pixel radiation values ​​corresponding to all scene hotspots to obtain the current pixel radiation total. The color of the current pixel is obtained by calculating the color difference using the total radiation of the current pixel, and the total radiation of the current pixel is set as the transparency of the current pixel.

5. The method for rendering a screen space heatmap according to claim 4, characterized in that, The step of using the scene buffer data and the blue noise texture to mix the current pixel color and the scene color and outputting it to the user display page includes: By using the current pixel color transparency, a new color is obtained by linear interpolation between the scene color buffer and the current pixel color; The new color is processed using blue noise texture, and the processed color is output to the user's display page.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the method for rendering a screen space heatmap as described in any one of claims 1-5.

7. A screen space heatmap rendering device, characterized in that, include: Processor and memory; The memory stores a computer-readable program that can be executed by the processor; when the processor executes the computer-readable program, it implements the steps of the method for rendering a screen space heatmap as described in any one of claims 1-5.

Images