A random light sectioning method based on gaze point

Abstract

The present invention proposes a gaze point-based random light cutting method, firstly, a light source tree based on a balanced binary tree is constructed for the light source of the current scene; then a G buffer is constructed, wherein the G buffer contains normals, albedo, velocity, and brightness frame buffer; in order to improve rendering performance without reducing the perceived quality, the method proposes a construction algorithm of a spatiotemporal-brightness perception sensitivity map and light cutting based on the gaze point; in order to further improve rendering performance, the method proposes a selection algorithm based on multi-resolution illumination samples; finally, the diffuse reflection and glossy reflection illumination results of each pixel are calculated, and the diffuse reflection and glossy reflection illumination results are merged to obtain the output image of the current frame. The present invention has a frame rate of 53-110fps, supports fully dynamic scenes containing more than 250,000 dynamic light sources and dynamic diffuse reflection / mirror / glossy objects, and can provide high-quality lighting effects for high-perception sensitive areas of the human visual system in VR head-mounted displays.

Description

Technical Field

[0001] The present invention relates to a gaze point-based random light cutting method, which generates high-quality lighting rendering results in real time for areas with high perception sensitivity of human eyes in a fully dynamic virtual scene containing a large number of light sources. Background Art

[0002] At present, there are four different acceleration strategies for gaze point-based 3D rendering technology: multi-spatial resolution acceleration strategy, detail level acceleration strategy, multi-color resolution acceleration strategy, and multi-illumination resolution acceleration strategy. The gaze point 3D rendering technology based on the multi-illumination resolution acceleration strategy improves rendering efficiency by providing different quality lighting effects for different visual sensitive areas of the human visual system. Stengel et al. [1] proposed a gaze point-based adaptive sampling mode for adaptive sampling of rasterization rendering methods, which is used to adjust the sampling rate of the peripheral areas of the human visual system so as to sample the areas with important features in the output image with higher precision. Tursun et al. [2] proposed a brightness contrast-aware gaze point rendering method, which further saves the computing resources of gaze point rendering by analyzing the local brightness contrast of the image. Wang Lili et al. [3] proposed a gaze point-based immediate radiance method in 2020. This method illuminates the gaze point area by projecting more virtual point light sources, thereby providing high-quality global illumination for the gaze point area, while presenting relatively accurate global illumination in the peripheral areas, thereby improving the rendering efficiency of global illumination. However, the high visual perception sensitive areas of the human visual system include not only the gaze point area, but also some surrounding areas with high saliency features and high brightness contrast features. The traditional gaze point rendering method based on multiple lighting resolutions does not combine the perception sensitivity of the human visual system to time, space and brightness, and considers high-quality lighting rendering in the surrounding high time, space and brightness perception sensitive areas to further improve the quality of concave drawing. The gaze point-based random light cutting method can simultaneously render high-quality lighting rendering in the gaze point area and the surrounding high time, space and brightness perception sensitive areas at a high frame rate.

[0003] In order to efficiently render lighting for scenes containing a large number of light sources, Walter et al. [4] proposed a light cut generation method, which generates a light cut for each pixel and uses the light cut to color the pixel. However, the light cut generation method does not consider combining the visual perception characteristics of the human visual system with the light cut generation algorithm. In order to improve the efficiency of lighting rendering without causing visual perception loss, the present invention proposes a light cut generation method based on spatiotemporal-brightness, which can generate light cuts of different precisions in different perceptual sensitive areas, thereby improving the efficiency of lighting rendering.

[0004] Directly using the evaluation importance criteria to generate light cuts for lighting rendering is not accurate enough, especially when the light cut contains nodes near the root node of the light source tree. In order to improve the rendering quality, Lin et al. [5] selected light source samples for all nodes of the light cut generated for each pixel, and then used the selected light source samples to color the pixel to improve the rendering quality. In order to further improve the efficiency of light source sample selection, the present invention proposes a multi-resolution light sample selection algorithm. This method reuses low-precision light source samples selected by low-resolution lighting rendering for each node of the light cut in pixels in low-perceptual sensitivity areas, reuses low-precision light source samples selected by low-resolution for some nodes of the light cut in pixels in high-perceptual sensitivity areas, and quickly selects high-precision light source samples for the remaining nodes in the light cut.

[0005] [1] M. Stengel, S. Grogorick, M. Eisemann, and M. Magnor. Adaptive imagespacesampling for gaze-contingent real-time rendering. In Computer Graphics Forum, vol. 35, pp. 129–139. Wiley Online Library, 2016.

[0006] [2] OTTursun, E. Arabadzhiyska-Koleva, M. Wernikowski, R. Mantiuk, H.-P. Seidel, K. Myszkowski, and P. Didyk. Luminance-contrast-aware foveatedrendering. ACM Transactions on Graphics (TOG), 38(4):1–14, 2019.

[0007] [3]L.Wang, R.Li,

[0008] [4]Walter B, Fernandez S, Arbree A, et al. Lightcuts: a scalable approach to illumination[M] / / ACM SIGGRAPH 2005Papers.2005:1098-1107.

[0009] [5] D. Lin and C. Yuksel. Real-time stochastic lightcuts. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 3(1):1–18, 2020. Summary of the invention

[0010] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art and provide a gaze point-based random light cutting method to render high-quality lighting for high-perception sensitive areas in a VR head-mounted display at a high frame rate, and to support full dynamic scenes with more than 250,000 dynamic light sources and diffuse reflection, specular reflection, and glossy objects.

[0011] The main innovative points of the present invention are:

[0012] (1) A gaze-based spatiotemporal-brightness perception sensitivity map and light-cutting construction method is implemented. By combining the perception sensitivity of the human visual system to spatiotemporal and brightness, a gaze-based spatiotemporal-brightness perception sensitivity map is constructed. Without sacrificing the perceived rendering quality, light-cuttings of different precisions are generated for different visual perception sensitive areas to improve rendering efficiency.

[0013] (2) A multi-resolution light source sample selection method is implemented, which can more efficiently select light source samples for each node in the light section.

[0014] The technical solution adopted by the present invention is: a random light sectioning method based on gaze point, comprising the following steps:

[0015] Step 1: Construction of light source tree: Given a virtual scene, construct a light source tree based on a balanced binary tree for the light source of the current scene;

[0016] Step 2: G buffer construction: Given the perspective of the current frame and the previous frame, construct the G buffer FB of the current frame g ; The G buffer FB g Contains the normal frame buffer fb n , albedo frame buffer fb a , speed frame buffer fb v and brightness frame buffer The normal frame buffer fb n Each pixel in stores the normal of the pixel on the corresponding object in the virtual scene; the albedo frame buffer fb a Stores the albedo of the pixel on the corresponding object in the virtual scene; the speed frame buffer fb v Store the moving speed of the pixel on the corresponding object in the virtual scene; the brightness frame buffer Store the brightness of the pixel on the corresponding object in the virtual scene. The calculation is performed by rendering a low-resolution rendering image of the current frame. The light section LC′ and the selected light source sample LS′ constructed during the rendering process are saved in the low-resolution image calculation for step 4 to select illumination samples based on multiple resolutions. The size of the low-resolution image is 1 / 8 of the resolution of the output image of the current frame.

[0017] Step 3: Construction of the spatiotemporal-brightness perception sensitivity map and light section based on the gaze point: Using the G buffer FB constructed in step 2 g Extract the gaze-based spatiotemporal feature map FSFM and the gaze-based brightness feature map FLFM, and then construct the gaze-based spatiotemporal-brightness perception sensitivity map SL-FPSM based on FSFM and FLFM; each pixel in FSFM stores a 0-1 value to indicate the spatiotemporal feature sensitivity corresponding to the pixel; each pixel in FLFM stores a 0-1 value to indicate the brightness feature sensitivity corresponding to the pixel; each pixel in SL-FPSM stores a 0-1 value, the larger the value, the higher the spatiotemporal-brightness perception sensitivity and the higher the rendering quality; then generate light sections LC of different precisions for each pixel according to the value of each pixel in SL-FPSM;

[0018] Step 4: Selection of illumination samples based on multi-resolution: Using the light source tree constructed in step 1, the light section LC′ constructed in the process of rendering the low-resolution rendering image in step 2 and the selected light source sample LS′, the gaze point-based spatiotemporal-brightness perception sensitivity map SL-FPSM constructed in step 3, reuse the light source samples in LS′ for some nodes in LC according to SL-FPSM and LC′, traverse the light source tree and select light source samples for the remaining nodes in LC;

[0019] Step 5: Lighting rendering: Using the light source sample selected in step 4, calculate the diffuse reflection lighting result for each pixel of the output image of the current frame. Using the SL-FPSM constructed in step 3, use the SL-FPSM-based path tracing algorithm to calculate the glossy reflection lighting result for each pixel of the output image of the current frame. Finally, combine the diffuse reflection lighting result and the glossy reflection lighting result to obtain the final lighting rendering result.

[0020] Further, in step 2, the brightness frame buffer The result is as follows: first, given the resolution (w, h) of the output image of the current frame, a low-resolution rendering image of the current frame is rendered using the real-time random light sectioning method. The resolution of the low-resolution rendering image is w / 8, h / 8, which is 1 / 8 of the output image resolution of the current frame. The light section LC′ and the selected light source sample LS′ constructed during the rendering process of the low-resolution rendering image are saved for step 4 to select light source samples based on multiple resolutions. Then, the low-resolution rendering image is upsampled using the bicubic upsampling method to a resolution of (w, h). Finally, the brightness frame buffer of the current frame is calculated based on the upsampled low-resolution rendering image.

[0021] Furthermore, in step 3, the construction method of the spatiotemporal-brightness perception sensitivity map and light section based on the gaze point is implemented as follows:

[0022] (1) Using the G buffer FB constructed in step 2 g To construct the spatiotemporal-brightness perception sensitivity map SL-FPSM, first, from FB g Extract the spatiotemporal feature map FSFM based on the gaze point and the brightness feature map FLFM based on the gaze point; for each pixel px in the output image of the current frame, use a three-dimensional Gaussian filter to combine the values ​​of FSFM and FLFM on px, and simulate the degree of reduction of the combined feature value within the maximum imperceptible threshold α; finally, for each pixel px, use the standard deviation of the three-dimensional Gaussian filter corresponding to this pixel to measure the acceptable degree of reduction of the feature value within the range of α, and then calculate the value of SL-FPSM in this px according to α, where α is a decimal in the range of 0-0.5;

[0023] (2) For each pixel px in the output image of the current frame, construct a light section of px, the number of nodes of which is map[px]·MAX lspp , where map[px] is the value of SL-FPSM on px, MAX lspp is the maximum number of light source samples; then a greedy algorithm is used to traverse the root node of the light source tree tree constructed in step 1 to construct a light section for px, until the number of nodes of the light section of px reaches the required number of nodes, and the light section of different precisions is generated for each pixel.

[0024] Further, in step 4, the light source tree tree constructed in step 1, the light section LC′ constructed in the process of rendering the low-resolution rendering image in step 2 and the selected light source sample LS′, and the gaze point-based spatiotemporal-brightness perception sensitivity map SL-FPSM constructed in step 3 are implemented as follows: for each node node in the light section LC[px] of each pixel px in the output image of the current frame, a random number rad is generated; if rad is less than the value of SL-FPSM on the pixel px, and there is a light source sample, i.e., a leaf node, in the light source sample set LS′[px] generated in step 2, then it is the leaf node of node, and the light source sample in LS′[px] is reused as the light source sample of node; if rad is not less than the value of SL-FPSM on the pixel px, or there is no light source sample in LS′[px′] that is a leaf node of node, then the hierarchical importance sampling method is used to traverse the light source tree starting from node until a leaf node l is obtained, and l is used as the light source sample of node.

[0025] Furthermore, in step 4, the lighting rendering method is implemented as follows:

[0026] Using the light source sample selected in step 4, the diffuse reflection lighting result is calculated for each pixel of the output image of the current frame. Using the SL-FPSM constructed in step 3, the glossy reflection lighting result is calculated for each pixel of the output image of the current frame using the SL-FPSM-based path tracing algorithm. Finally, the diffuse reflection lighting result and the glossy reflection lighting result are combined to obtain the final lighting rendering result.

[0027] The principle of the present invention is:

[0028] (1) The perceptual quality of gaze point rendering is further improved by rendering high-quality lighting for surrounding areas with high spatiotemporal-brightness perception sensitivity.

[0029] (2) According to the current viewpoint, a gaze-based spatiotemporal feature map and a gaze-based brightness feature map are extracted, and a three-dimensional Gaussian filter is used to simulate the reduction of each pixel in the spatiotemporal and brightness characteristics within the range that does not produce perceptible loss. A gaze-based spatiotemporal-brightness perceptual sensitivity map is constructed to store the standard deviation of each pixel of the three-dimensional Gaussian filter to describe the maximum degree of spatiotemporal-brightness reduction that each pixel can accept.

[0030] (3) The efficiency of the light source sample selection method is further improved by reusing the optimal light source samples that have been selected in low-resolution lighting rendering for some nodes of the light cut.

[0031] The advantages of the present invention compared with the prior art are:

[0032] (1) Compared with existing methods, the present invention can achieve a frame rate of 53-110fps in a VR head-mounted display and support full dynamic scenes containing more than 250,000 dynamic light sources and dynamic diffuse reflection / mirror / bright objects. This method can not only perform high-quality lighting rendering in the gaze point area, but also render high-quality lighting effects in the surrounding high spatiotemporal-brightness perception sensitive areas;

[0033] (2) The present invention also proposes a light section generation method based on time-space-brightness. Compared with the real-time random light section method proposed by Lin et al., this method combines the visual perception characteristics of the human visual system with light section generation, and can generate light sections with different accuracies in different perception sensitive areas;

[0034] (3) The present invention proposes a multi-resolution light sample selection method, which reuses low-precision light source samples selected by low-resolution lighting rendering for each node of the light cut in the pixels of the low-perception sensitivity area, reuses low-precision light source samples selected by low resolution for some nodes of the light cut in the pixels of the high-perception sensitivity area, and quickly selects high-precision light source samples for the remaining nodes in the light cut. Compared with the real-time random light cut method proposed by Lin et al., the method of the present invention can further improve the efficiency of light source sample selection. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a lighting rendering effect diagram based on the gaze point in a scene containing a large number of light sources according to the present invention;

[0036] Figure 2 is an overall flow chart of the random light sectioning method based on gaze point of the present invention;

[0037] Figure 3 A flowchart for constructing a spatiotemporal-brightness perception sensitivity map based on gaze points in the present invention;

[0038] Figure 4 The flowchart of the selection algorithm of the illumination samples based on multi-resolution in the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described below in conjunction with other drawings and specific embodiments.

[0040] like Figure 1As shown, the first step of the present invention is to construct a light source tree based on a balanced binary tree for the light source of the current scene. The second step is to construct the G buffer of the current frame and save the light section LC′ and the selected light source sample LS′ generated when constructing the brightness frame buffer of the G buffer. The third step is to construct a spatiotemporal-brightness perception sensitivity map (SL-FPSM) and light section based on the gaze point. First, the G buffer constructed in the second step is used to construct the SL-FPSM, and then light sections of different precision are generated for each pixel according to the value of each pixel in the SL-FPSM. The fourth step is to select light source samples based on multi-resolution. Based on the selection of multi-resolution illumination samples, for all nodes of the light section of each pixel in the output image, the light source samples in the selected light source sample set LS′ are directly used for some nodes using SL-FPSM and LC′, and the remaining nodes are traversed from the light source tree and light source samples are selected. The fifth step is lighting rendering. For each pixel px of the output image of the current frame, the light source sample selected by px in the fourth step is used to calculate the diffuse lighting result for px; according to SL-FPSM, a path tracing method with one bounce is used to render the glossy reflection lighting result for px. Finally, the diffuse reflection lighting result and the glossy reflection lighting result are merged to obtain the final lighting rendering result.

[0041] 1. Construction of light source tree

[0042] like Figure 2 As shown in , the first step of the present invention is to build a light source tree. In order to rebuild the light tree in real time to support scenes containing a large number of dynamic light sources, a balanced binary tree is used to build the light source tree. In a light source tree based on a balanced binary tree, all leaf nodes containing a single light source are at the bottom of the tree. Since the number of leaf nodes in a balanced binary tree must be a power of two, empty light sources are added as needed to fill the leaf nodes so that the total number is a power of two. Each node in the light source tree stores the bounding box and total light intensity of the underlying light source. Since a balanced binary tree is used, it is not necessary to store child node pointers because the index of the child node can be directly calculated from the index of the parent node. In addition, each leaf node stores the corresponding light source ID. Since the ID of the light source cannot be directly calculated from the leaf node index in a dynamic scene, the leaf index of the light source may be different for each frame. In order to minimize the construction time, the builder of the present invention sorts the light sources according to the Morton code of their position.

[0043] 2. Construction of G buffer

[0044] like Figure 2 As shown in , the second step of the present invention is to construct a G buffer. Given the perspectives V and V′ of the current frame and the previous frame of the virtual scene S, the resolution (w, h) of the output frame, and the G buffer FB of the current frame, g , FB gContains the normal frame buffer fb n , albedo frame buffer fb a , speed frame buffer fb v , Luminance Frame Buffer fb n Each pixel in fb stores the normal of the corresponding object in the virtual scene. Similarly, fb a Stores the albedo of the pixel on the corresponding object in the virtual scene, fb v The moving speed of this pixel on the corresponding object in the virtual scene is stored. The method of the present invention also requires brightness features to guide the generation of light cuts and the selection of light source samples. It is effective and efficient to extract brightness features from low-resolution rendering results, and the low-precision light cuts and light source samples generated by the low-resolution rendering process can be reused in high-resolution rendering. Therefore, given the resolution (w, h) of the output image of the current frame, we directly use the random light cut method proposed by Yuksel et al. to render a low-resolution rendering image fb′ of the current frame, and the resolution of the low-resolution rendering image is (w / 8, h / 8). At the same time, the light cut LC′ and the selected light source sample LS′ constructed in the rendering process of the low-resolution rendering image are saved for step (4) to perform fast selection of illumination samples based on multi-resolution. Then, the bicubic upsampling method is used to upsample fb′ to a resolution of (w, h), that is, the brightness frame buffer of the current frame is obtained.

[0045] 3. Construction of the spatiotemporal-brightness perception sensitivity map and light section based on gaze point

[0046] like Figure 2 As shown in , the third step of the present invention is to construct a space-time-brightness perception sensitivity map (SL-FPSM) and a light section based on the gaze point. In the method for constructing a space-time-brightness perception sensitivity map (SL-FPSM) and a light section based on the gaze point, the SL-FPSM is first constructed, and then light sections of different precisions are generated for each pixel according to the SL-FPSM. The process of constructing the SL-FPSM is as follows: Figure 3 In the steps of constructing SL-FPSM, firstly, from the G buffer FB gThe FSFM and FLFM are extracted from the gaze point-based spatiotemporal feature map. Then, the FSFM and FLFM are combined using a three-dimensional Gaussian filter, and the degree of reduction of the combined feature value within the maximum imperceptible threshold α is simulated. Finally, the standard deviation of the three-dimensional Gaussian filter in each pixel is used to calculate the value of the SL-FPSM in the pixel. In the step of generating light sections according to the SL-FPSM, for each pixel in the current frame output result fb, the corresponding pixel value in the SL-FPSM is multiplied by a constant to obtain the finite number of nodes required for the light section of this pixel, and then the number of nodes of each light section generates light sections of different precision for each pixel.

[0047] Figure 3 Describes the construction process of SL-FPSM. Given a G buffer FB g And the visual acuity feature map fb based on the gaze point f , output SL-FPSM map. FB g Contains the normal frame buffer fb of the current frame n , albedo frame buffer fb a , speed frame buffer fb v , Luminance Frame Buffer Visual acuity feature map based on gaze point f Each pixel in stores the visual acuity value of the eccentricity corresponding to the visual acuity reduction function at the pixel point. The construction of SL-FPSM includes two sub-steps.

[0048] In sub-step 1, two gaze-based feature maps are extracted: the gaze-based spatiotemporal feature map FSFM fm st And the brightness feature map FLFM fm based on the gaze point l .fm st The eigenvalue of each pixel p in is calculated by the following formula:

[0049]

[0050] where fb f (p) is the value of the visual acuity feature map at pixel p, Sobel filter is used to calculate p in fb n The value of the first-order derivative in is, Calculate p in fb a The value of the first-order derivative in is, Calculate p in fb v The value of the first derivative in. fm l The eigenvalue of each pixel p in is calculated by the following formula:

[0051]

[0052] First of all A 3-level brightness Laplacian image pyramid LP is established. LP1(p) represents the value of p in the first level of the Laplacian image pyramid, and LP3(p) represents the value of p in the third level. ε is used to prevent the low brightness area from causing the LP3(p) value to be 0, which produces a mathematical singularity. ε is set to 0.01.

[0053] In sub-step 2, for each pixel p, a 3D Gaussian filter with the maximum standard deviation ρ is sought to merge the gaze-based spatiotemporal feature map FSFMfm st And the brightness feature map FLFMfm based on the gaze point l , and simulates the maximum reduction of the eigenvalue at each pixel within the maximum imperceptible threshold α. For the Gaussian filter, the larger the standard deviation, the greater the difference between the filtered features and the original features, and the greater the reduction of the eigenvalue. A larger reduction in eigenvalues ​​will result in a greater degree of degradation in rendering quality. Therefore, for each pixel, within the maximum imperceptible threshold α, the larger the standard deviation of its three-dimensional Gaussian filter, the greater the degradation in rendering quality that this pixel can tolerate. The following equation describes the standard deviation ρ of the corresponding three-dimensional Gaussian filter for a pixel p p Maximum reduction in the values ​​of spatiotemporal and brightness features based on the gaze point over the simulated α range.

[0054]

[0055] Where max(fm st (p),fm l (p)) is fm st (p) and fm l (p) is the maximum value between them. It is the reduced eigenvalue after convolution by the three-dimensional Gaussian filter. p The larger the value, the greater the difference between the maximum eigenvalue and the reduced eigenvalue, so the ρ convolved by the three-dimensional Gaussian filter p The reduced eigenvalue of It is represented by:

[0056]

[0057] at the same time, It can also be expressed as the following formula:

[0058]

[0059] in It is fm st (p) is the Gaussian mean of a 5×5 kernel centered at It is fml (p) is the Gaussian mean of a kernel of size 5×5 centered at ρ. Therefore, ρ p It can be calculated by the following formula:

[0060]

[0061] Then, the value of each pixel in the SL-FPSM map is calculated by the following formula:

[0062]

[0063] where ρ p min is the smallest value among all pixels. p , ρ p max is the largest ρ among all pixels p .

[0064] After constructing the SL-FPSM, use it to guide the generation of light slices. For each pixel p in the output frame buffer fb, multiply map[p] by the maximum number of light samples per pixel MAX lspp , get the number of nodes L required for the light cutting of this pixel. Then refine the light cutting according to L until the number of nodes of the light cutting becomes L.

[0065] 4. Selection of multi-resolution lighting samples

[0066] like Figure 2 As shown in , the fourth step of the present invention is to generate illumination samples based on multiple resolutions. This method reuses the optimal light samples selected in low-resolution rendering for some nodes of the light cut, without traversing the light source tree once for each node of the light cut, thereby speeding up the selection of light source samples. First, the light source sample ls of the current pixel px is initialized to empty, and then a light source sample is selected for each node of the light cut of each pixel.

[0067] For a node in a light cut, the light source sample selection flow chart is as follows Figure 4 As shown. First, a random number rad is generated in the range of 0 to 1, and then the corresponding pixel px′ of px in the low-resolution rendering frame buffer fb′ is calculated, px′=px·(w′ / w, h′ / h), where (w, h) is the resolution of the output image of the current frame, and (w′, h′) is the resolution of the low-resolution rendering image of the current frame, which is set to (w′, h′)=(w / 8, h / 8).

[0068] If rad is less than (1.0-map[px]), and there is a low-resolution light source sample (i.e., leaf node) in the low-resolution light source sample set LS′[px′] that is a leaf node of node, select the optimal low-resolution light source sample l in LS′[px′]. Use the function lrLS to quickly determine whether there is a low-resolution illumination sample in LS′[px′] that is a leaf node of the node. Since in the light source tree based on a balanced binary tree, each node on the tree can be quickly indexed by its offset value lId, and the nodes are stored continuously from small to large according to their offset values. Therefore, for each pixel px′, the nodes in the low-resolution light section LC′[px′] are stored from small to large according to their offset values, and the light source samples in the low-resolution light source sample set LS′[px′] are stored from small to large according to their offset values. lrLS calculates the minimum offset lId of the leaf node of the node min and the maximum offset lId max , if lId min Less than the minimum offset or lId of the light source sample in LS′[px′] max Returns true if the value is greater than the maximum offset of light source samples in LS'[px'], otherwise returns false. min and lId max It is calculated by the following formula.

[0069]

[0070] Where n is the total number of tree layers. If a low-resolution light sample in LS′[px′] is a leaf node of node, the function nLight is used to select the best light sample l for node from LS′[px′]. The nLight function first calculates the distance dist between the node and the corresponding node node′ in LC′[px′] of the light sample l in LS′[px′] as follows, and then selects the light sample l with the smallest distance dist.

[0071]

[0072] If rad is not less than (1.0-map[px]), or there is no low-resolution light source sample in LS′[px′] that is a leaf node of node, use the hierarchical importance sampling method to traverse the light source tree starting from node until the leaf node l is obtained; finally, merge l into ls.

[0073] 5. Lighting rendering

[0074] like Figure 2 As shown in , the last step of the present invention is lighting rendering to generate the output result fb of the current frame. First, initialize fb and diffuse reflection frame buffer fb dand the specular frame buffer fb m Then, given the virtual scene S, the current frame's viewing angle V, the output frame's resolution (w, h), and the selected light sample set LS, the traditional lighting rendering method is performed to calculate the diffuse lighting of each pixel and store the result in the diffuse frame buffer fb. d If there are objects with glossy or specular materials in S, according to the virtual scene S, the spatiotemporal-brightness perception sensitivity map SL-FPSM based on the gaze point, and the maximum number of ray samples per pixel MAX rspp , V and (w, h), realize the path tracing rendering of each pixel based on the gaze point, and store the result in the highlight reflection frame buffer fb m The gaze point-based path tracking is a one-shot path tracking implemented under the guidance of SL-FPSM. Specifically, for fb m For each pixel px in the gaze point, the number of ray samples used in the path tracing method based on the gaze point is MAX rspp ·map[px]. Finally, the value of each pixel in fbd is compared with fb m The value of each pixel in is added to get the final rendering result of the pixel.

[0075] The hardware configuration used in the experiment is Intel i7-10700KF processor, Nvidia Geforce 3080Ti graphics card, 64G memory. It runs on Windows 10 64-bit system. All programs involved in the efficiency test are 64-bit programs compiled with O2 level optimization.

[0076] The technical contents not elaborated in detail in the present invention belong to the common knowledge of those skilled in the art.

[0077] Although the above describes the illustrative specific embodiments of the present invention to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A gaze point-based random light sectioning method, characterized in that: The following steps are involved: Step 1: Construction of light source tree: Given a virtual scene, construct a light source tree based on a balanced binary tree for the light source of the current scene ; Step 2: G buffer construction: Given the perspective of the current frame and the previous frame, construct the G buffer of the current frame ; The G buffer Include normals framebuffer , Albedo Frame Buffer , speed frame buffer and brightness frame buffer ; The normal frame buffer Each pixel in stores the normal of the pixel on the corresponding object in the virtual scene; the albedo frame buffer Store the albedo of the pixel on the corresponding object in the virtual scene; the speed frame buffer Store the moving speed of the pixel on the corresponding object in the virtual scene; the brightness frame buffer Store the brightness of the pixel on the corresponding object in the virtual scene. It is calculated by rendering a low-resolution rendered image of the current frame. The light section constructed during the rendering process is also saved in the low-resolution image calculation. and selected light source samples Used for step 4 to select illumination samples based on multiple resolutions, and the size of the low resolution is 1 / 8 of the output image resolution of the current frame; Step 3: Construction of the spatiotemporal-brightness perception sensitivity map and light section based on the gaze point: Using the G buffer constructed in step 2 Extract the gaze-based spatiotemporal feature map FSFM and the gaze-based brightness feature map FLFM, and then construct the gaze-based spatiotemporal-brightness perception sensitivity map SL-FPSM based on FSFM and FLFM; each pixel in FSFM stores a 0-1 value to indicate the spatiotemporal feature sensitivity corresponding to the pixel; each pixel in FLFM stores a 0-1 value to indicate the brightness feature sensitivity corresponding to the pixel; each pixel in SL-FPSM stores a 0-1 value, the larger the value, the higher the spatiotemporal-brightness perception sensitivity and the higher the rendering quality; then generate light sections of different precision for each pixel according to the value of each pixel in SL-FPSM ; Step 4: Multi-resolution based lighting sample selection: Using the light source tree constructed in step 1 , the light section constructed during the rendering of the low-resolution rendered image in step 2 and selected light source samples , the spatiotemporal-brightness perception sensitivity map SL-FPSM based on the gaze point constructed in step 3, according to SL-FPSM and ,for Reuse of some nodes in The light source sample in is The remaining nodes in the light source tree Traverse and select light source samples; Step 5: Lighting rendering: Using the light source sample selected in step 4, calculate the diffuse reflection lighting result for each pixel of the output image of the current frame. Using the SL-FPSM constructed in step 3, use the SL-FPSM-based path tracing algorithm to calculate the glossy reflection lighting result for each pixel of the output image of the current frame. Finally, combine the diffuse reflection lighting result and the glossy reflection lighting result to obtain the final lighting rendering result.

2. The gaze point-based random light sectioning method according to claim 1, characterized in that: In step 2, the brightness frame buffer The result is as follows: First, the resolution of the output image of the current frame is given , use the real-time random light cutting method to render the low-resolution rendering image of the current frame. The resolution of the low-resolution rendering image is , which is 1 of the output image resolution of the current frame , and also saves the light section constructed during the rendering process of the low-resolution rendered image and selected light source samples Used in step 4 to select light source samples based on multi-resolution; The low-resolution rendered image is then upsampled using the bicubic upsampling method to a resolution of ; Finally, the brightness frame buffer of the current frame is calculated based on the upsampled low-resolution rendered image .

3. The gaze point-based random light sectioning method according to claim 1, characterized in that: In step 3, the construction method of the spatiotemporal-brightness perception sensitivity map and light section based on the gaze point is implemented as follows: (1) Using the G-buffer constructed in step 2 To construct the spatiotemporal-brightness perception sensitivity map SL-FPSM, first, Extract the spatiotemporal feature map FSFM based on the gaze point and the brightness feature map FLFM based on the gaze point; for each pixel in the output image of the current frame , using a three-dimensional Gaussian filter to filter FSFM and FLFM The values ​​on the eigenvalues ​​are combined and the combined eigenvalues ​​are simulated at the maximum imperceptible threshold. range; finally, for each pixel , the standard deviation of the three-dimensional Gaussian filter corresponding to this pixel is used to measure The acceptable reduction of characteristic values ​​within the range, and then according to Calculate SL-FPSM in this The value in It is a decimal in the range of 0-0.5; (2) For each pixel in the output image of the current frame , build The number of nodes of the light cut is ,in, For SL-FPSM The value on is the maximum number of light source samples; then the light source tree constructed from step 1 is constructed using the greedy algorithm The root node of traversal is Construct the light section until The number of nodes of the light section reaches the required number of nodes, and the light section with different precision is generated for each pixel.

4. The gaze point-based random light sectioning method according to claim 1, characterized in that: In step 4, the light source tree constructed in step 1 is used , the light section constructed during the rendering of the low-resolution rendered image in step 2 and selected light source samples , the gaze-based spatiotemporal-brightness perception sensitivity map SL-FPSM constructed in step 3 is implemented as follows: for each pixel in the output image of the current frame Light Section Each node in , generate a random number ;like Smaller than SL-FPSM in pixels The value on the light source sample set generated in step 2 There are light source samples, i.e. leaf nodes, in The leaf nodes of The light source samples in light source sample; if Not less than SL-FPSM in pixels The value on There is no light source sample in The leaf nodes of Start traversing the light source tree until you get a leaf node , and As Sample light source.

Images