A temporal upsampling method and device for real-time rendering of participating media

Abstract

The present invention discloses a time domain upsampling method and device for real-time rendering of participating media, including: downsampling a scene depth map to obtain a low-resolution scene depth map; generating a sub-pixel offset based on a time domain image sample; performing discontinuous pixel rendering on the participating media based on the sub-pixel offset and the low-resolution scene depth map to obtain a low-resolution rendered image of the participating media; performing time domain reconstruction on the rendered image of the participating media according to the sub-pixel offset; upsampling the reconstructed image at the reconstruction resolution to obtain a rendered image at the rendering resolution of the participating media. The method and device can greatly improve the rendering efficiency without significantly reducing the image quality.

Description

Technical Field

[0001] The present invention relates to the field of real-time rendering technology, and in particular to a time domain upsampling method and device for real-time rendering of a participating medium. Background Art

[0002] Participating media are widely present in nature. Suspended particles in fluids such as smoke, atmosphere, and clouds can absorb and scatter light, making the propagation of light in participating media diverse and complex from a macroscopic perspective. In the rendering of three-dimensional scenes, participating media, as one of the main elements of the scene, can enhance the sense of presence of the scene. It is widely used by artists when constructing many three-dimensional scenes, and has shined in large-scale 3D games in recent years, greatly enhancing the atmosphere of the scene.

[0003] The rendering of participating media can also be referred to as volume rendering because they usually occupy a certain volume range in space. The definition of participating media in space is generally divided into two parts:

[0004] (1) Define the boundaries of the space, such as the space defined by the Axis Aligned Bounding Box, or the space defined by the material attached to the object in Unreal Engine;

[0005] (2) Define the data in the space. For each position or voxel grid in the space, define its density, temperature and other information. The data in the space can be real-time simulation data, which corresponds to the research in the field of fluid simulation; it can also be programmatically generated values, which is usually done in real-time rendering; or it can be pre-generated data, such as the VDB data format.

[0006] The usual implementation of volume rendering uses a screen-space algorithm, emitting a ray for each pixel on the screen, performing ray marching in the scene, and collecting data of participating media in the scene through sampling points distributed on the line of sight, thereby determining the shape of the participating media. In terms of lighting shading, secondary rays are emitted along the scene lighting direction at the sampling points on the line of sight, and the sampling points of the lighting march are distributed on the secondary rays. The transmittance of each sampling point is accumulated, and the original lighting color is attenuated by the accumulated transmittance, and finally converged to the viewpoint to obtain the color of the participating medium seen by the virtual camera.

[0007] With the continuous development of display devices, more and more high-resolution display screens have been launched, and people's demand for screen display resolution has also been rising. The increase in the number of pixels has put forward higher requirements for many graphics algorithms affected by the number of pixels. For the rendering of participating media, if the screen pixel width is w, the screen pixel height is h, the number of samples used in ray stepping is s, and the number of samples used in light stepping is ls, its time complexity is O(w*h*s*ls). At a display resolution of 1920*1080, a typical implementation of a participating media algorithm takes about 20ms per frame in real-time rendering. For the sake of smoothness, the total time consumed by all links of real-time rendering per frame should not exceed 16.67ms. The time consumed for participating media rendering alone exceeds the total overhead, which is almost unacceptable in real-time rendering scenarios. Moreover, at higher resolutions, the overhead of participating media rendering will continue to rise. Summary of the invention

[0008] In order to improve the performance of participating media in real-time rendering, the present invention proposes a time domain upsampling method and device for real-time rendering of participating media, which greatly improves the rendering efficiency without significantly reducing the picture quality.

[0009] To achieve the above object, an embodiment of the present invention provides a time domain upsampling method for real-time rendering of a participating medium, comprising the following steps:

[0010] Downsampling the scene depth map to obtain a scene depth map of the rendering resolution;

[0011] generating a sub-pixel offset based on the temporal image samples;

[0012] Performing discontinuous pixel rendering on the participating medium based on the sub-pixel offset and the scene depth map of the rendering resolution to obtain a rendered image at the rendering resolution of the participating medium;

[0013] Reconstructing the rendered image of the participating medium in the time domain according to the sub-pixel offset;

[0014] Upsampling the reconstructed image at the reconstruction resolution to obtain a rendered image at the original resolution of the participating medium;

[0015] Among them, original resolution ≥ reconstruction resolution > rendering resolution.

[0016] Preferably, generating the sub-pixel offset based on the time domain image samples comprises:

[0017] A sub-pixel offset is generated on a preset number of time-domain image samples by a uniform offset method to form an offset combination consisting of a preset number of elements. The position of each element in the offset combination represents the offset direction of the sub-pixel within the pixel of the rendering resolution, and the element value represents the frame in which the sub-pixel is rendered. The sub-pixel offset is the pixel width of the rendering resolution divided by the preset order.

[0018] Preferably, the performing non-continuous pixel rendering on the participating medium based on the scene depth map of the sub-pixel offset and the rendering resolution comprises:

[0019] The projection matrix of the current virtual camera is offset according to the sub-pixel offset to obtain the offset projection matrix, and the offset projection matrix is ​​used to perform non-continuous pixel rendering on the participating medium by ray stepping. Among them, the low-resolution scene depth map is used to determine whether the ray stepping method is terminated early. The intersection of the line of sight vector of the virtual camera and the scene is determined by the scene depth map. No rendering is performed after the intersection, and the ray stepping process is terminated early.

[0020] Preferably, the method of performing non-continuous pixel rendering on the participating medium by using an offset projection matrix and adopting a ray-stepping method comprises:

[0021] Using the pre-generated voxel data and combining it with the bounding box area, the sampling points are distributed along the line of sight vector of the virtual camera by ray stepping. The density of the participating medium is sampled at each sampling point to accumulate the change in transmittance at the current position. Then, the illumination shading samples are calculated and accumulated along the illumination direction at each sampling point. Finally, the rendered color image of the participating medium corresponding to each pixel on the screen and the average distance through the participating medium are obtained. The average distance constitutes the rendered depth image.

[0022] Preferably, when shading illumination, custom multiple scattering orders are supported. According to different multiple scattering orders, the extinction coefficient and scattering coefficient at the sampling point are attenuated step by step when calculating illumination, and the phase function used is mixed step by step with the uniformly distributed phase function to simulate the effect of multiple scattering.

[0023] Preferably, the calculation process of the average distance through the participating medium comprises:

[0024] For each sampling point on the line of sight vector of the virtual camera where the density of the participating medium is not zero, the transmittance of the current sampling point is used as the weight, the weighted depth of the participating medium is accumulated, and the accumulated transmittance is used as the weight value. The average distance through the participating medium is the ratio of the weighted depth of all sampling points to the weight value.

[0025] Preferably, the performing time domain reconstruction on the rendered image of the participating medium according to the sub-pixel offset comprises:

[0026] The pixels that need to be reprojected are confirmed based on the sub-pixel offset. For each pixel that needs to be reprojected, it is projected to the pixel position of the previous frame through the reprojection matrix. When the projected pixel position is in the screen space of the previous frame, the reprojection is considered successful, and the color data and depth data of the previous frame are sampled at the projected pixel position. The color data and depth data of the previous frame are processed through the adjacent pixel processing strategy, and the processed data is used as the pixel output of the reconstructed image.

[0027] For pixels that fail to be reprojected, bilinear interpolation is used to collect color data and depth data on the low-resolution rendering image of the participating medium at the pixel position, and the data is used as the pixel output of the reconstructed image.

[0028] Preferably, the adjacent pixel processing strategy includes:

[0029] First, the depth data is classified according to whether it is continuous. The continuity of the depth data is judged by the following formula:

[0030] abs(Depth p -Depth his )>Depth p *β (1)

[0031] Among them, Depth p Depth is the distance between the scene point and the virtual camera restored by the current scene depth map. his is the distance between the scene point and the virtual camera restored by sampling the historical frame depth data, abs(·) represents the absolute value, and β represents a decimal between 0 and 1;

[0032] If equation (1) is satisfied, the pixel depth data after reprojection is considered discontinuous. Depending on whether the pixel depth information after reprojection is continuous, there are two strategies:

[0033] Strategy 1: For pixel positions with discontinuous depth data, the depth data of the current frame is used as the reference, and the color and depth values ​​of the pixel closest to the scene depth of the current frame within a certain radius of the current pixel are used;

[0034] Strategy 2: For locations with continuous depth data, the color data of the historical frame is restricted by the color value of the pixel within a certain radius of the current pixel. The color of the historical frame cannot exceed the maximum and minimum values ​​of the pixels within the range. If it exceeds the range, the maximum or minimum value will prevail.

[0035] Preferably, when upsampling the reconstructed image at the reconstruction resolution, one or a combination of bilinear interpolation upsampling, upsampling using time domain dithering, and bilateral upsampling is adopted.

[0036] To achieve the above-mentioned invention object, the embodiment further provides a time domain upsampling device for participating medium real-time rendering, comprising a downsampling module, an offset generation module, a participating medium rendering module, a time domain reconstruction module, and an upsampling module;

[0037] The downsampling module is used to downsample the scene depth map to obtain a scene depth map of a rendering resolution;

[0038] The offset generation module is used to generate a sub-pixel offset based on the time domain image samples;

[0039] The participating medium rendering module is used to perform non-continuous pixel rendering on the participating medium based on the sub-pixel offset and the scene depth map of the rendering resolution to obtain a rendered image at the rendering resolution of the participating medium;

[0040] The time domain reconstruction module is used to perform time domain reconstruction on the rendered image of the participating medium according to the sub-pixel offset;

[0041] The upsampling module is used to upsample the reconstructed image at the reconstruction resolution to obtain a rendered image at the original resolution of the participating medium.

[0042] Among them, original resolution ≥ reconstruction resolution > rendering resolution.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] Based on the generation of sub-pixel offsets based on time domain image samples, non-continuous pixel rendering of participating media is performed through sub-pixel offset on the low-resolution image, and then the number of samples is accumulated in the time domain, and reconstruction and upsampling operations are performed on the high-resolution image, thereby greatly improving the rendering efficiency without significantly reducing the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 is a flow chart of a time domain upsampling method for real-time rendering of a participating medium provided in an embodiment;

[0047] Figure 2 is a flowchart of a time domain upsampling method for real-time rendering of a participating medium provided in an embodiment;

[0048] Figure 3is a comparison diagram of various downsampling methods provided in the embodiment and the original depth map;

[0049] Figure 4 is an example diagram of temporal sub-pixel offset values ​​provided by an embodiment;

[0050] Figure 5 is a rendering result diagram of smoke provided by the embodiment;

[0051] Figure 6 It is a schematic diagram of the structure of a time domain upsampling device for participating in real-time rendering of a medium provided in an embodiment. DETAILED DESCRIPTION

[0052] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0053] In the embodiment of the present invention, the resolutions of the three stages are first determined:

[0054] Rendering Resolution R pm : refers to the low resolution used when rendering participating media;

[0055] Reconstruction resolution R recon : refers to the resolution used in the time domain upsampling reconstruction process;

[0056] Original resolution scene : refers to the resolution used for original scene rendering;

[0057] It is stipulated that the original resolution ≥ reconstruction resolution > rendering resolution, and the specific resolution size can be set and combined by the user. A typical resolution setting is: the original resolution is set to the full screen resolution (for example: 1920*1080), the reconstruction resolution is set to half the resolution (for example: 960*540), and the rendering resolution is set to one quarter of the resolution (for example: 480*270). The following will take this resolution setting as an example for explanation.

[0058] like Figure 1 Shown and Figure 2 As shown, the time domain upsampling method for real-time rendering of participating media provided in the embodiment includes the following steps:

[0059] S110, downsampling the scene depth map to obtain a scene depth map of a rendering resolution.

[0060] Downsample the scene depth map of the original resolution to obtain the scene depth map of the rendering resolution. A checkerboard distribution downsampling method can be used. For example, for the 4x4 pixels in the scene depth map, after downsampling, it becomes 2x2 pixels, and each pixel corresponds to 4 pixels in the scene depth map. In the 2x2 pixels of the downsampled map, the pixel in the lower left corner is the minimum value of the original four pixels; the pixel in the lower right corner is the maximum value of the original four pixels; the pixel in the upper left corner is the maximum value of the original four pixels; and the pixel in the upper right corner is the minimum value of the original four pixels. This forms a checkerboard pattern, see Figure 3 In comparison, downsampling the maximum depth value will cause the edges of objects to shrink inward, while downsampling the minimum depth value will cause the edges of all objects to expand outward. The chessboard distribution downsampling combines the two methods as a whole, and has a higher degree of preservation and restoration of the original scene.

[0061] S120, generating a sub-pixel offset based on the time domain image samples.

[0062] Based on the distribution strategy of image samples in the time domain, the pixel offset, i.e., the sub-pixel offset, is generated and the projection matrix of the virtual camera is modified using the offset. Through the pixel offset, the actual rendering position of each pixel in each frame is different, and multiple samples are distributed to multiple consecutive frames without increasing the number of samples used in each frame.

[0063] Sub-pixel offset means that the offset is not greater than one pixel of the rendering resolution, and the sample can be distributed at any position within the pixel through the random offset value. There are multiple optional schemes for the distribution strategy of image samples in the time domain. This embodiment adopts uniform offset, that is, the offset of the sub-pixel of each frame is relatively fixed. The sub-pixel offset is generated on the preset order of time domain image samples by uniform offset, forming an offset combination composed of preset order elements. Each element position in the offset combination represents the offset direction of the sub-pixel within the pixel of the rendering resolution, and the element value represents the frame where the sub-pixel is rendered. The sub-pixel offset is the pixel width of the rendering resolution divided by the preset order.

[0064] Assume that there is order order=(R recon / R cloud ) 2 , where R recon represents the reconstruction resolution, R cloud Indicates the rendering resolution. The rendering of the participating medium samples one pixel per frame. The number of pixels of the reconstruction resolution is order times the number of pixels of the rendering resolution. Therefore, in order to obtain an image of the reconstruction resolution for each consecutive order frame, order image samples need to be distributed in sub-pixels. In the example, order = 4, a 4-order frame offset combination. See Figure 4In the example frame offset combination, the 0th frame corresponds to the element in the upper left corner, so the sampling point is offset to the upper left sub-pixel in the pixel. When the reconstruction resolution is twice the rendering resolution, one pixel of the rendering resolution corresponds to four pixels of the reconstruction resolution. Assuming the width of each pixel of the rendering resolution is 1, the offset of the upper left pixel is (-0.25, 0.25).

[0065] S130, performing discontinuous pixel rendering on the participating medium based on the sub-pixel offset and the scene depth map of the rendering resolution to obtain a rendered image at the rendering resolution of the participating medium.

[0066] After obtaining the sub-pixel offset, the projection matrix of the current virtual camera is offset according to the sub-pixel offset to obtain the offset projection matrix. The specific offset processing methods include:

[0067] Let the original virtual camera projection matrix be M proj , the offset matrix composed of offsets is M offset , then multiply the two matrices to get the final offset projection matrix:

[0068]

[0069] Among them, j x j y Represents the horizontal and vertical offset of the sub-pixel on the screen, n represents the near plane distance of the virtual camera cone, f represents the far plane distance of the virtual camera cone, w represents the near plane half-width of the virtual camera cone, and h represents the near plane half-height of the virtual camera cone.

[0070] The offset projection matrix of the virtual camera is used to perform discontinuous pixel rendering of participating media. After the offset, the rendering of participating media images is discontinuous on the reconstructed resolution image. The discontinuous sample distribution combined with different sub-pixel offsets per frame enables the participating media rendering at low resolution to distribute more samples in the time domain, thus solving the problem of insufficient sample number to a certain extent.

[0071] There are many ways to implement real-time rendering of participating media in the industry. In this embodiment, pre-generated voxel data is used in combination with a bounding box area defined in the world space. Sampling points are distributed along the line of sight vector of the virtual camera by ray stepping. The density of the participating medium is sampled at each sampling point to accumulate the change in transmittance at the current position. Then, at each sampling point, the illumination shading samples are calculated and accumulated along the illumination direction. Finally, a rendered color image of the participating medium corresponding to each pixel on the screen and an average distance through the participating medium are obtained. The average distance constitutes a rendered depth image.

[0072] During the ray-stepping process, a low-resolution scene depth image is used to determine whether the ray-stepping method is terminated early. Specifically, the scene depth image is used to determine the intersection of the virtual camera's sight vector and the scene. After the intersection, the participating medium is blocked by scene objects, so rendering is not continued and the ray-stepping process is terminated early.

[0073] When shading light, the embodiment supports customized multiple scattering orders. According to the different multiple scattering orders, the extinction coefficient (Extinction Coefficience) and scattering coefficient (Scattering Coefficience) at the sampling point are attenuated step by step when calculating the light, and the phase function used is mixed step by step with the uniformly distributed phase function to simulate the effect of multiple scattering.

[0074] In addition to outputting the rendered color image of the participating medium, it is also necessary to output the average distance through the participating medium to form a rendered depth image. The calculation process of the average distance through the participating medium includes:

[0075] For each sampling point x on the line of sight of the virtual camera where the density of the participating medium is not zero n , with the transmittance Tr(x n ) as the weight, accumulating the weighted depth Depth(x n ), and the cumulative transmittance is used as the weight value, and the average distance Depth through the participating medium cloud is the ratio of the weighted depth of all sampling points to the weight value, expressed as:

[0076]

[0077] S140, reconstructing the rendered image of the participating medium in the time domain according to the sub-pixel offset.

[0078] Since S120 sets a different offset for each frame, the image rendered by S130 is different for each frame. It corresponds to the pixel information of the offset position of S120 at the reconstruction resolution. Therefore, when performing time domain reconstruction, it is necessary to use the image of the rendering resolution to generate the image of the reconstruction resolution. The specific process is:

[0079] The pixels that need to be reprojected are confirmed based on the sub-pixel offset. For each 2x2 pixel block in the reconstructed resolution image, there is a pixel generated by rendering in the current frame S130. This pixel corresponds to the sub-pixel offset and is the latest data. Therefore, the rendering value is directly used as the pixel output of the reconstructed resolution image. For the other three pixels in the 2x2 pixel block, they do not correspond to the sub-pixel offset value of the current frame and need to be obtained from the previous frame of the reconstructed resolution image through reprojection.

[0080] For each pixel P to be reprojected i , through the reprojection matrix M proj ·M offset Projected to the pixel position of the previous frame, when the projected pixel position P i 'If it is in the screen space of the previous frame, the reprojection is considered successful, otherwise the reprojection fails.

[0081] Pixels that fail to be reprojected cannot reuse data from historical frames, so data can only be obtained from the current frame. In this embodiment, for pixels that fail to be reprojected, bilinear interpolation is used to collect color data and depth data on a low-resolution rendering image of the participating medium at the pixel position, and the data is used as the pixel output of the reconstructed image.

[0082] The pixels that are successfully reprojected use the data in the historical frame. i The pixel position P after reprojection i ', at the projected pixel position P i 'Sample the color data and depth data of the previous frame, process the color data and depth data of the previous frame through the adjacent pixel processing strategy, and use the processed data as the pixel output of the reconstructed image.

[0083] Among them, the adjacent pixel processing strategy includes: first, classify according to whether the depth data is continuous, and the continuity of the depth data is judged by the following formula:

[0084] abs(Depth p -Depth his )>Depth p *β (3)

[0085] Among them, Depth p Depth is the distance between the scene point and the virtual camera restored by the current scene depth map. his is the distance between the scene point and the virtual camera restored by sampling the historical frame depth data, abs(·) represents the absolute value, and β represents a decimal between 0 and 1;

[0086] If equation (3) is satisfied, the pixel depth data after reprojection is considered discontinuous. Depending on whether the pixel depth information after reprojection is continuous, there are two strategies:

[0087] Strategy 1: For pixel positions with discontinuous depth data, the depth data of the current frame is used as the reference, and the color and depth values ​​of the pixel closest to the scene depth of the current frame within a certain radius of the current pixel are used;

[0088] Strategy 2: For locations with continuous depth data, the color data of the historical frame is restricted by the color value of the pixel within a certain radius of the current pixel. The color of the historical frame cannot exceed the maximum and minimum values ​​of the pixels within the range. If it exceeds the range, the maximum or minimum value will prevail.

[0089] S150 , upsampling the reconstructed image at the reconstruction resolution to obtain a rendered image at the original resolution of the participating medium.

[0090] After constructing the reconstructed image, further upsampling is required to restore the reconstructed resolution to the original resolution. This process can use a variety of upsampling strategies, such as bilinear interpolation upsampling, upsampling using time domain dithering, and bilateral upsampling, or a combination of multiple. In this embodiment, a bilateral upsampling method based on depth perception + a method of time domain dithering is used, and the specific process includes:

[0091] In the depth-aware bilateral upsampling method, consider the 2x2 pixels where the current pixel is located:

[0092] (1) Read the scene depth and the participating medium depth d from the participating medium reconstructed depth image c ;

[0093] (2) Obtain the scene depth d corresponding to the current pixel through the scene depth map and camera position s ;

[0094] (3) Calculate scene depth d s and the participating medium depth d within a 2x2 pixel block c The difference diff;

[0095] (4) Reading the bilinear interpolation upsampling threshold input by the user;

[0096] (5) When diff is greater than the threshold, bilinear interpolation upsampling is performed on the 2x2 pixel block;

[0097] (6) Otherwise, sample the participating medium color information color within the 2x2 pixel block, let t be the user-defined weighting coefficient, according to the formula:

[0098]

[0099] color=dot(ω,color recon ) (5)

[0100] Among them, diff is a four-dimensional vector, each dimension represents the scene depth d s and the participating medium depth d within a 2x2 pixel block cThe difference between ω and ω represents the weight of each pixel color in the 2x2 pixel block in the form of a four-dimensional vector, color recon Represents the pixel color of a 2x2 pixel block.

[0101] (7) After weighted mixing according to formula (5), the upsampled pixel color value color is obtained;

[0102] Temporal jitter refers to adding a random offset of each frame change to the upsampled pixel color value, adding more image samples in the time domain through the offset sampling points, and determining the superposition weight based on the consistency of the transmittance, and finally obtaining the final color of the participating medium rendering. The steps of temporal jitter are divided as follows:

[0103] (1) The sampling points are shifted by random jitter, and the reconstructed image is sampled using the shifted n sampling points;

[0104] (2) The weight of the sampling result is determined by interpolating the transmittance stored in the participating medium color after sampling (stored in the transparency channel of the participating medium color) and the transmittance in the current participating medium color, and multiplying and accumulating it with the sampling result in (1) to obtain the final color value. The calculation formula is as follows:

[0105]

[0106] Among them, extinction is the extinction coefficient set by the user, Alpha i Represents the transparency channel information of the upsampled color image sampled by the i-th sample, Color i Represents the RGBA channel information of the upsampled color image sampled by the i-th sample, and Alpha represents the average value of the transparency channel information of the upsampled color image sampled by n samples.

[0107] After obtaining the rendered image of the participating medium, the upsampling result of the participating medium is mixed with the original scene image. The original scene image refers to the result of rendering the opaque objects in the scene. The premultiplied transparency method is used to superimpose the image with the color of the scene to achieve the final rendering of the participating medium. This mixing will produce the following results: for pixels without scene objects, the final color of the pixel is the color of the participating medium; for pixels with scene objects, the final color of the pixel depends on the thickness of the participating medium between the virtual camera and the scene object. If the spatial range of the participating medium is thicker, the density is higher, or the distance traversed is longer, the scene objects behind will be obscured by the participating medium; if the spatial range of the participating medium is thinner, the density is lower, or the distance traversed is shorter, the scene objects will be clearly visible. The result of rendering smoke using the present invention is as follows. Figure 5 shown.

[0108] Based on the same inventive concept, the embodiment also provides a time domain upsampling device for participating in real-time rendering of a medium, such as Figure 6 As shown, the device 600 includes a downsampling module 610, an offset generation module 620, a participating medium rendering module 630, a time domain reconstruction module 640, and an upsampling module 650;

[0109] Among them, the downsampling module 610 is used to downsample the scene depth map to obtain a scene depth map of the rendering resolution; the offset generation module 620 is used to generate a sub-pixel offset based on the time domain image samples; the participating medium rendering module 630 is used to perform non-continuous pixel rendering on the participating medium based on the sub-pixel offset and the scene depth map of the rendering resolution to obtain a rendered image at the rendering resolution of the participating medium; the time domain reconstruction module 640 is used to perform time domain reconstruction on the rendered image of the participating medium according to the sub-pixel offset; the upsampling module 650 is used to upsample the reconstructed image at the reconstruction resolution to obtain a rendered image of the original resolution of the participating medium.

[0110] It should be noted that the time domain upsampling device for participating in real-time rendering of media provided in the above-mentioned embodiments should be illustrated by the division of the above-mentioned functional modules when performing time domain upsampling for participating in real-time rendering of media, and the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the time domain upsampling device for participating in real-time rendering of media provided in the above-mentioned embodiments and the time domain upsampling method embodiment for participating in real-time rendering of media belong to the same concept, and the specific implementation process thereof is detailed in the time domain upsampling method embodiment for participating in real-time rendering of media, which will not be repeated here.

[0111] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A temporal upsampling method for real-time rendering of a participating medium, characterized in that: The following steps are involved: Downsampling the scene depth map to obtain a scene depth map of the rendering resolution; generating a sub-pixel offset based on the temporal image samples; Based on the sub-pixel offset and the scene depth map of the rendering resolution, non-continuous pixel rendering is performed on the participating medium to obtain a rendered image at the rendering resolution of the participating medium, including: offsetting the projection matrix of the current virtual camera according to the sub-pixel offset to obtain the offset projection matrix, using the offset projection matrix and adopting the ray stepping method to perform non-continuous pixel rendering on the participating medium, wherein the low-resolution scene depth map is used to determine whether the ray stepping method is terminated early, and the intersection of the line of sight vector of the virtual camera and the scene is determined by the scene depth map, and no rendering is performed after the intersection, and the ray stepping process is terminated early; Performing time-domain reconstruction on the rendered image of the participating medium according to the sub-pixel offset, including: confirming the pixels that need to be reprojected according to the sub-pixel offset, projecting each pixel that needs to be reprojected to the pixel position of the previous frame through the reprojection matrix, and when the pixel position after projection is in the screen space of the previous frame, it is considered that the reprojection is successful, and sampling the color data and depth data of the previous frame at the pixel position after projection, processing the color data and depth data of the previous frame through the adjacent pixel processing strategy, and using the processed data as the pixel output of the reconstructed image; for the pixel that fails to be reprojected, using bilinear interpolation to collect the color data and depth data on the low-resolution rendered image of the participating medium at the pixel position, and using it as the pixel output of the reconstructed image; Among them, the adjacent pixel processing strategies include: First, the depth data is classified according to whether it is continuous. The continuity of the depth data is judged by the following formula: abs(Depth p -Depth his )>Depth p *β (1) Among them, Depth p Depth is the distance between the scene point and the virtual camera restored by the current scene depth map. his is the distance between the scene point and the virtual camera restored by sampling the historical frame depth data, abs(·) represents the absolute value, and β represents a decimal between 0 and 1; If equation (1) is satisfied, the pixel depth data after reprojection is considered discontinuous. Depending on whether the pixel depth information after reprojection is continuous, there are two strategies: Strategy 1: For pixel positions with discontinuous depth data, the depth data of the current frame is used as the reference, and the color and depth values ​​of the pixel closest to the scene depth of the current frame within a certain radius of the current pixel are used; Strategy 2: For locations with continuous depth data, the color data of the historical frame is restricted by the color value of the pixel within a certain radius of the current pixel. The color of the historical frame cannot exceed the maximum and minimum values ​​of the pixels within the range. If it exceeds the range, the maximum or minimum value prevails. Upsampling the reconstructed image at the reconstruction resolution to obtain a rendered image at the original resolution of the participating medium; Among them, original resolution ≥ reconstruction resolution > rendering resolution.

2. The temporal upsampling method for real-time rendering of participating media according to claim 1, characterized in that: The generating of sub-pixel offsets based on the time domain image samples comprises: A sub-pixel offset is generated on a preset number of time-domain image samples by a uniform offset method to form an offset combination consisting of a preset number of elements. The position of each element in the offset combination represents the offset direction of the sub-pixel within the pixel of the rendering resolution, and the element value represents the frame in which the sub-pixel is rendered. The sub-pixel offset is the pixel width of the rendering resolution divided by the preset order.

3. The temporal upsampling method for real-time rendering of participating media according to claim 1, characterized in that: The method of using the offset projection matrix and adopting a ray-stepping method to perform non-continuous pixel rendering on the participating medium includes: Using the pre-generated voxel data and combining it with the bounding box area, the sampling points are distributed along the line of sight vector of the virtual camera by ray stepping. The density of the participating medium is sampled at each sampling point to accumulate the change in transmittance at the current position. Then, the illumination shading samples are calculated and accumulated along the illumination direction at each sampling point. Finally, the rendered color image of the participating medium corresponding to each pixel on the screen and the average distance through the participating medium are obtained. The average distance constitutes the rendered depth image.

4. The temporal upsampling method for real-time rendering of participating media according to claim 3, characterized in that: When shading with light, it supports customized multiple scattering orders. According to the different multiple scattering orders, the extinction coefficient and scattering coefficient at the sampling point are attenuated step by step when calculating the light, and the phase function used is mixed with the uniformly distributed phase function step by step to simulate the effect of multiple scattering.

5. The temporal upsampling method for real-time rendering of participating media according to claim 3, characterized in that: The calculation process of the average distance through the participating media includes: For each sampling point on the line of sight vector of the virtual camera where the density of the participating medium is not zero, the transmittance of the current sampling point is used as the weight, the weighted depth of the participating medium is accumulated, and the accumulated transmittance is used as the weight value. The average distance through the participating medium is the ratio of the weighted depth of all sampling points to the weight value.

6. The temporal upsampling method for real-time rendering of participating media according to claim 1, characterized in that: When upsampling the reconstructed image at the reconstruction resolution, one or a combination of bilinear interpolation upsampling, upsampling using time domain dithering, and bilateral upsampling is used.

7. A temporal upsampling device for real-time rendering of a participating medium, characterized in that: It includes a downsampling module, an offset generation module, a participating medium rendering module, a time domain reconstruction module, and an upsampling module; The downsampling module is used to downsample the scene depth map to obtain a scene depth map of a rendering resolution; The offset generation module is used to generate a sub-pixel offset based on the time domain image samples; The participating medium rendering module is used to perform non-continuous pixel rendering on the participating medium based on the sub-pixel offset and the scene depth map of the rendering resolution to obtain a rendered image at the rendering resolution of the participating medium, including: performing an offset processing on the projection matrix of the current virtual camera according to the sub-pixel offset to obtain a post-offset projection matrix, using the post-offset projection matrix and adopting a ray stepping method to perform non-continuous pixel rendering on the participating medium, wherein the low-resolution scene depth map is used to determine whether the ray stepping method is terminated in advance, and the intersection of the line of sight vector of the virtual camera and the scene is determined by the scene depth map, and no rendering is performed after the intersection, and the ray stepping process is terminated in advance; The time domain reconstruction module is used to perform time domain reconstruction on the rendered image of the participating medium according to the sub-pixel offset, including: confirming the pixels that need to be reprojected according to the sub-pixel offset, projecting each pixel that needs to be reprojected to the pixel position of the previous frame through the reprojection matrix, when the pixel position after projection is in the screen space of the previous frame, it is considered that the reprojection is successful, and sampling the color data and depth data of the previous frame at the pixel position after projection, processing the color data and depth data of the previous frame through the adjacent pixel processing strategy, and using the processed data as the pixel output of the reconstructed image; for the pixel that fails to be reprojected, using bilinear interpolation to collect color data and depth data on the low-resolution rendered image of the participating medium at the pixel position, and using it as the pixel output of the reconstructed image; Among them, the adjacent pixel processing strategies include: First, the depth data is classified according to whether it is continuous. The continuity of the depth data is judged by the following formula: abs(Depth p -Depth his )>Depth p *β (1) Among them, Depth p Depth is the distance between the scene point and the virtual camera restored by the current scene depth map. his is the distance between the scene point and the virtual camera restored by sampling the historical frame depth data, abs(·) represents the absolute value, and β represents a decimal between 0 and 1; If equation (1) is satisfied, the pixel depth data after reprojection is considered discontinuous. Depending on whether the pixel depth information after reprojection is continuous, there are two strategies: Strategy 1: For pixel positions with discontinuous depth data, the depth data of the current frame is used as the reference, and the color and depth values ​​of the pixel closest to the scene depth of the current frame within a certain radius of the current pixel are used; Strategy 2: For locations with continuous depth data, the color data of the historical frame is restricted by the color value of the pixel within a certain radius of the current pixel. The color of the historical frame cannot exceed the maximum and minimum values ​​of the pixels within the range. If it exceeds the range, the maximum or minimum value prevails. The upsampling module is used to upsample the reconstructed image at the reconstruction resolution to obtain a rendered image at the original resolution of the participating medium; Among them, original resolution ≥ reconstruction resolution > rendering resolution.

Images