An interactive panoramic spatial ray tracing method based on RGBD panoramic video

Abstract

The present invention proposes an interactive panoramic space ray tracing method based on RGBD panoramic video. The method is able to render photo-realistic lighting and shadow effects in real time when inserting virtual objects into 360° RGBD panoramic videos. First, a panoramic depth buffer and a screen space depth buffer are used to approximate the geometry of the real scene. Then, a sparse sampling ray generation method is proposed to accelerate the tracing process by reducing the number of rays that need to be emitted in ray tracing. After that, an incident radiance estimation algorithm is used to generate noisy Monte Carlo images. Finally, the final result is smoothed by interpolation, spatiotemporal filtering, and differential rendering. The results show that the method of the present invention can generate visually realistic frames for virtual objects in 360° RGBD videos in real time, making the mixed reality rendering results more natural and credible.

Description

Technical Field

[0001] The invention relates to an interactive panoramic space ray tracing method based on RGBD panoramic video, belonging to the technical field of computer graphics and mixed reality virtual-real fusion real-time rendering. Background Art

[0002] In mixed reality applications, it is challenging to insert virtual objects into natural world environments in a photo-realistic way. One of the problems is that it is difficult to capture the lighting, geometry, and material information of the entire environment. The missing natural environment information often leads to some artificial lighting or geometric assumptions, which are ultimately inconsistent. Another problem is that when simulating the interaction of virtual objects with natural environment lighting, some approximations are often made to meet the high frame rate requirements of mixed reality applications. However, some details such as surface shadows, spatially varying lighting, etc. will be lost.

[0003] With a general head-mounted AR device, real environment information including lighting and geometry can be captured within the narrow field of view of the front camera. In the absence of environmental information outside the field of view, many researchers focus on simulating the light and shadow interaction between virtual objects and the captured part of the natural environment. One method attempts to simulate the light transfer between the virtual world and the real world using a differential irradiance cache. Another method attaches additional light sources to the target scene area and uses the Kinect camera and geometry processing methods to simulate the radiation transfer between virtual objects and the real world. Both methods only capture part of the environment and assume external field of view directional lighting conditions. They cannot simulate the interaction between virtual objects and external environment geometry.

[0004] There are also some methods that choose to estimate external view environment information to obtain complete real-world environment lighting and geometry information. One of them assumes that the indoor scene is a six-sided cube and calculates the external view light source to illuminate the inserted virtual objects. Another work uses a CNN network to estimate multi-scale volumetric scene lighting with a single RGBD image and renders virtual objects using a spherical volume tracking method. Both methods output spatially varying realistic merged results, but there are still some problems. One is that there may be a big difference between the real world and the estimated external environment. Another reason is that some artifacts such as color stretching and missing reflections may occur when virtual objects interact with the depth gap in a single RGBD image. Summary of the invention

[0005] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art and provide an interactive panoramic space ray tracing method based on RGBD panoramic video, a novel mixed reality real-time interactive virtual-real fusion rendering, which can utilize the 360° RGBD video of the input scene, user perspective information and virtual object position, orientation, material and other information to simulate the natural interaction of light and shadow between the virtual object and the real scene, and generate natural and realistic interactive mixed scene images in real time.

[0006] In order to solve the above problems, an interactive panoramic spatial ray tracing method based on RGBD panoramic video is proposed, which can support spatially varying global illumination of multiple virtual object materials, including diffuse reflection, gloss and specular reflection, allowing dynamic virtual objects and changing real-world geometry. The method of the present invention significantly improves the user's sense of reality and presence by providing a consistent photo-realistic appearance of the integrated scene.

[0007] The technical solution adopted by the present invention is as follows Figure 3 As shown, the steps include:

[0008] Step 1: Input the panoramic image of the current frame of the RGBD panoramic video, the user's perspective and the information of the virtual object to be inserted, generate the screen space geometry buffer and the panoramic image space geometry buffer under the current perspective, and store the template function image, position image and normal image of the virtual object in the screen space and the panoramic image space in the buffer. At the same time, process the depth channel in the RGBD panoramic image to generate the corresponding hierarchical refinement depth texture map (MIPMAPS);

[0009] Step 2: Based on the position in the geometry buffer in the screen space under the user's current perspective in step 1 and the pixel-by-pixel position and normal information in the normal image, a corresponding multi-scale structured sparse sampling template cache is generated, and a sparse tracking acceleration algorithm is used to further accelerate rendering;

[0010] Step 3: Using a back-face-aware ray tracing algorithm, ray tracing is performed on the images in the screen space geometry buffer and the panoramic image space geometry buffer under the current viewing angle generated in step 1. The density distribution of the light emitted during ray tracing is controlled by the multi-scale structured sparse sampling template cache in step 2, and the intersection points of the light and the three-dimensional geometry are calculated to simulate the two-way interaction between the virtual object and the real world geometry, and obtain the noisy Monte Carlo incident radiance image of the current scene;

[0011] Step 4: The noisy Monte Carlo incident radiance image of the current scene obtained in step 3 is interpolated using the sampling points in the multi-scale structured sparse sampling template buffer generated in step 2 to generate a dense incident radiance image;

[0012] Step 5: Use a joint bilateral spatial filter to perform spatiotemporal filtering and denoising on the dense incident radiance image in step 4, and then perform temporal filtering on the spatial filtering result in the time domain through the motion vector in the screen space to obtain a reconstructed Monte Carlo incident radiance image of the scene at the current viewing angle before and after the virtual object is added;

[0013] Step 6: Using the Monte Carlo incident radiance image of the scene at the current perspective before and after adding the virtual object reconstructed in step 5, combined with the data of the screen space geometry buffer in step 1, the differential rendering method is used to render the virtual-reality fusion result image of the virtual object and the real world at the user's current perspective.

[0014] In step 3: the back-face-aware ray tracing algorithm is specifically implemented as follows:

[0015] (1) When ray tracing is performed on the image space geometry buffer and the panoramic image space geometry buffer through the control of the multi-scale structured sparse sampling template cache, in order to prevent the depth gap points in the image in the screen space geometry buffer from causing erroneous highlights or stretching artifacts, each time a ray is traced and a pixel point intersects with the image in the screen space geometry buffer, it is detected whether the pixel point belongs to a depth gap point;

[0016] (2) If the detected pixel belongs to a depth gap point, start sampling from the back side of the three-dimensional surface where the pixel is located, and select geometrically related replacement points as the true intersection points. The sampling of replacement points follows the following principles: the replacement point should be in the same plane as the light, line of sight, and original intersection pixel point in three dimensions; the distance from the replacement point to the original intersection pixel point should be equal to the physical limit;

[0017] (3) Ultimately, through such a back-face-aware ray tracing algorithm, the radiance value of each pixel corresponding to each ray is calculated, thereby realizing a two-way interaction between the simulated object and the real-world geometry, and finally obtaining a noisy Monte Carlo incident radiance image of the current scene.

[0018] In step 2, the sparse tracking acceleration algorithm is specifically implemented as follows:

[0019] Sparse tracking acceleration generates a multi-scale structured sparse sampling template cache to control whether to track the light of a specific pixel; for each pixel in screen space, the position and normal distribution of its neighboring pixels and whether it intersects with the virtual object determine whether to track the light. The calculation method of the result tracking weight buffer is described as follows:

[0020]

[0021] Where p is any pixel in the current screen space image, q is the neighboring pixel in p’s neighborhood Ω; in practice, Ω is set to a 5×5 pixel range, which is p’s neighborhood; for frame i, d i (p) is a binary parameter calculated in the virtual object template function image, which is used to indicate whether the pixel p has been illuminated by the virtual object in the last few frames, n p and pos p represents the normal and position of p, n q and pos q represents the normal and position of q; σ p It is a manually set normalization parameter. After all pixels in the screen space image are calculated, the tracking weight buffer is obtained;

[0022] According to formula (1), the tracking weight buffer is obtained, and then the pixels in the tracking weight buffer are divided into three regions by two manually set thresholds δ1 and δ2, 0<δ1<δ2<1; for any pixel p, the tracking level l of p is p The record is as follows:

[0023]

[0024] w p is the tracking weight calculated for p pixels, 0, 1, and 2 are specific values ​​of the tracking level, marking the tracking level of p pixels. In practice, δ1 is set to 0.3 and δ2 is set to 0.5. In order to make the sparse tracking acceleration algorithm suitable for the GPU, the screen space image is constructed into a multi-scale grid, that is, a multi-scale structured sparse sampling template cache, and different densities of tracking rays are emitted in these areas according to the areas to which the multi-scale grids belong.

[0025] The advantages of the present invention compared with the prior art are:

[0026] (1) In a real scene, the incident light on any point on a surface comes from the indirect illumination of all visible surfaces in the direction of its own principal normal hemisphere. Therefore, the illumination influence received by the current point can be calculated by performing screen space ray tracing in the RGBD panoramic image space. At the same time, the present invention proposes an interactive panoramic space ray tracing method based on RGBD panoramic video. Compared with other existing methods, the rendering speed is fast and the calculation of light and shadow interaction of virtual-reality fusion scene can be completed in a very short time. The rendering frame rate can reach more than 100fps, supporting interactive virtual-reality fusion rendering.

[0027] (2) Depth faults in RGBD panoramic images can cause visible defects in the lighting of virtual objects, especially virtual objects with highlight materials. By using a back-aware screen-space ray tracing algorithm to infer the possible lighting effects between faults, such rendering defects can be greatly reduced or even eliminated. Compared with existing methods, the present invention takes into account the influence of the depth gaps that are widely present in RGBD panoramic images when performing ray tracing, and proposes a back-aware screen-space ray tracing algorithm to achieve a more reasonable and realistic virtual-real fusion lighting effect.

[0028] (3) For different surface points that are spatially adjacent and have similar normals in the scene, the lighting information they receive in the scene is also similar. Using this principle, a sparse sampling template can be structured in the screen space to perform ray tracing, thereby reducing the sampling rate and increasing the rendering speed. The present invention takes into account the similarity between the global illumination between spatially adjacent and feature-similar surface points in real scenes, and uses this property to propose a method for constructing a multi-scale structured sparse sampling template cache, which further accelerates the virtual-reality fusion rendering speed by reducing the number of samples and maintains an acceptable rendering quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Flow chart for realizing the method of the present invention;

[0030] Figure 2 This is a virtual-real fusion effect diagram of the present invention;

[0031] Figure 3 It is the overall flow chart of the present invention;

[0032] Figure 4 The schematic diagram of screen space ray tracing for back-side perception; (a) is a target tracing ray moving to the right and entering the gap between the edge of the table and the cabinet behind it, and (b) is a top view of (a);

[0033] Figure 5 Screen space sparse sampling template and interpolation schematic. DETAILED DESCRIPTION

[0034] The present invention is further described below in conjunction with other drawings and specific embodiments.

[0035] like Figure 1 As shown, the real-time process of the present invention is divided into six steps. The first step is to input the panoramic image of the current frame of the RGBD panoramic video, the user's perspective and the information of the virtual object to be inserted for preprocessing; the second step is to establish a multi-scale structured sparse sampling template map (cache); the third step is to perform screen space ray tracing in the panoramic image space; the fourth step is to interpolate the ray tracing image; the fifth step is to perform spatiotemporal filtering on the interpolated image; the sixth step is to perform differential rendering to obtain the final virtual-real fusion rendering image result.

[0036] Figure 2 The virtual-real fusion rendering result of inserting a vase, a gift box and a metal clothes hanger in a 360° bedroom scene is given. First, the current frame is extracted from the 360° RGBD video, and the screen space image buffer is obtained according to the user's viewing direction. Then, the virtual object to be inserted is added to the real scene, and ray tracing and related light and shadow interaction calculation and simulation are performed in the screen space and panoramic image space. Finally, the result image after the virtual object is inserted is obtained, and it is fed back to the display device worn by the user in real time to achieve the natural fusion of virtual and real objects. The main innovations of the present invention are: (1) an interactive panoramic image space ray tracing pipeline is proposed and implemented, which can dynamically insert and render virtual objects into 360° panoramic videos. (2) a back-aware screen space ray tracing algorithm is proposed and implemented, which takes into account the gaps in the 360° RGBD panoramic video frame and achieves more reasonable and realistic virtual and real object lighting effects. (3) an acceleration method for sparse tracking template generation and interpolation is proposed and implemented, which further accelerates the screen space ray tracing process and maintains acceptable image quality.

[0037] Figure 3 The specific processing flow of interactive panoramic space ray tracing based on RGBD panoramic video is given.

[0038] Step 1: Correspondence Figure 3 In the geometric approximation part, the panoramic image of the current frame of the RGBD panoramic video, the user's perspective and the information of the virtual object to be inserted are input to generate the screen space geometry buffer and the panoramic image space geometry buffer under the current perspective. The buffer stores images such as the position and normal in the corresponding space. At the same time, the depth channel in the RGBD panoramic image is processed to generate the corresponding hierarchical refinement depth texture map (MIPMAPS).

[0039] Step 2: Correspondence Figure 3 In the scale structured sampling template cache generation part, based on the pixel-by-pixel position, normal and other information of various images in the geometric buffer in the screen space under the user's current perspective in step 1, a corresponding multi-scale structured sparse sampling template cache is generated.

[0040] Specific implementation details such as Figure 5 As shown in Figure 1, the main idea of ​​sparse tracking acceleration is to generate a sparse sampling mask to control whether to track the light of a specific pixel. For each pixel on the screen, the position and normal vector distribution of its neighboring pixels and whether it intersects with the virtual object are considered to decide whether to track the light. The resulting tracking weight function is described as follows:

[0041]

[0042] Where p is the target pixel and q is the neighboring pixel of p in the neighborhood Ω. In practice, Ω is set to a 5×5 pixel range. For frame i, d i (p) is a binary parameter computed in the two-step inward radiance estimation pass that indicates whether pixel p has been illuminated by a virtual object in the last few frames. p and pos p represents the normal and position of p, and similarly for q. i If di(p) = 1, then p is an active pixel, which means it interacts with the virtual object and the weight is set to 1.0. If di(p) = 0, for each q in Ω, the maximum of the normal and position difference of p is calculated and set as the tracking weight. p is the normalization parameter of the pixel position difference, which is set to 10.0 in practice. According to equation (7), the tracking weight buffer is obtained, and then the screen pixels are divided into three regions by two thresholds δ1 and δ2 (0<δ1<δ2<1). For a pixel p, its tracking level l p The record is as follows:

[0043]

[0044] In practice, δ1 is set to 0.3 and δ2 is set to 0.5. In order to make the algorithm suitable for GPU, the entire screen space is constructed into a multi-scale grid, and different densities of tracing rays are emitted in these areas according to the area they belong to. Figure 5 As shown, for area levels 0 (innermost layer), 1 (middle layer) and 2 (outermost layer), 1, 1 / 4 and 1 / 16 rays are traced for each pixel, that is, one ray is traced for every 1, 4 and 16 pixels.

[0045] Step 3: Correspondence Figure 3 The incident radiance estimation in step 1 is performed; a back-aware ray tracing algorithm is used to perform ray tracing on the images in the screen space geometry buffer and the panoramic image space geometry buffer under the current viewing angle generated in step 1. The density distribution of the light emitted during ray tracing is controlled by the multi-scale structured sparse sampling template cache in step 2. The intersection points of the light and the three-dimensional geometry are calculated to simulate the two-way interaction between the virtual objects and the real-world geometry, and the noisy Monte Carlo incident radiance image of the current scene is obtained.

[0046] The specific implementation details are as follows: When inserting a virtual object in mixed reality, the virtual object is illuminated by the surrounding real-world surfaces, and at the same time, the real-world surfaces are also affected by the virtual objects. The fact is that the real surface covered by the shadow of the virtual object will not illuminate the virtual object as much as the exposed one. If the rays are traced into them at the same time, the influence of the inserted virtual object will be lost. Therefore, the present invention divides the tracking into two steps. In the first step, the rays are traced into the pixels of the real object area of ​​the screen, and then the colors of the real object pixels are calculated and updated. In the second step, the rays are traced into the pixels of the virtual object, and the updated color image is used to calculate the appearance of the virtual object. Therefore, there are two types of rays in the tracking channel, one for the real surface area and the other for the virtual object area.

[0047] On the other hand, the details of the back-face-aware ray tracing algorithm are as follows: Since the 360° RGBD image only stores a single layer of the real-world environment, some artifacts may appear on the surface of the inserted virtual object, especially for glossy and specular materials. The reason is that when tracing rays in image space, sometimes the intersection point is located on the depth gap. The geometry behind the front object suddenly disappears, which usually causes color jumps or stretching on the virtual object. To this end, the present invention introduces a back-face-aware screen-space ray tracing method to mitigate the damage of back-face missing to the authenticity of the rendering results. The method checks whether there is a back-face missing situation and finds a suitable alternative intersection point for tracing rays.

[0048] like Figure 4 As shown, in (a), a target tracking ray travels to the right and enters the gap between the edge of the table and the cabinet behind it. For this case, it is difficult to know which point the ray intersects with because the back of the table is missing in the RGBD image buffer. Traditional screen-space ray tracing sets the intersection point of the edge of the table (g1 in b) as the alternative intersection point, which usually results in incorrect highlights or stretching artifacts on virtual objects. Therefore, the present invention introduces a back-face processing method to alleviate this unnatural appearance. Figure 4 As shown in (b), a top view of (a) is given. l1 and l2 are the nearest layers of a frame in the 360° RGBD video captured from the center point e. In this case, l1 is the front surface of the desktop and l2 is the surface of the cabinet. g1 and g2 are the segmentation points of the depth layer, where g1 belongs to l1 and g2 belongs to l2. M is the virtual sphere to be inserted. During the ray tracing process, the light reflected from point s on the virtual sphere M is entering the gap between g1 and g2. The main idea of ​​the back-face-aware screen-space ray tracing method of the present invention is to assume that the back face should be similar to the front face of the object. For example, in Figure 4In (a), the back of the table should also be white and, like the front, have a wooden top, which means that, in (b), the appearance of the back side g1p is similar to that of the surface l1. Then, if an alternative intersection point is chosen from l1, the appearance on the virtual sphere M will be natural. Based on this assumption, the search for alternative intersection points begins. First, a manipulation plane is constructed with the ray sp and the viewpoint e, as shown in (b). On this plane, the intersection of the ray sp and the back side g1p is found. g1p is perpendicular to the tangent plane g1p′. p′ is positioned equal to the length |g1p| by setting the length |g1p|. If the lengths are not equal, the final image of the back side of the virtual object may be stretched or shortened. Finally, ep′ is connected and extended to intersect with l1 at point q, which is set as the alternative intersection point between the ray sp and the 360° RGBD frame.

[0049] Step 4: Correspondence Figure 3 In the interpolation part, based on the multi-scale structured sparse sampling template cache generated in step 2, the noisy Monte Carlo incident radiance image of the current scene obtained in step 3 is interpolated through the sampling points in the multi-scale structured sampling template cache to generate a dense incident radiance image.

[0050] Step 5: Correspondence Figure 3 In the spatiotemporal filtering part, a joint bilateral spatial filter is used to perform spatiotemporal filtering and denoising on the dense incident radiance image in step 4, and then the spatial filtering result is temporally filtered in the time domain by the motion vector in the screen space to obtain a reconstructed Monte Carlo incident radiance image of the scene at the current viewing angle before and after the virtual object is added;

[0051] Step 6: Correspondence Figure 3 In the differential rendering part, the Monte Carlo incident radiance image of the scene at the current perspective before and after the addition of the virtual object is used to reconstruct the scene in step 5, combined with the data of the screen space geometry buffer in step 1, and the differential rendering method is used to render the virtual and real fusion result image of the virtual object and the real world at the user's current perspective.

[0052] The technical contents not elaborated in detail in the present invention belong to the common knowledge of those skilled in the art.

[0053] Although the above describes the illustrative specific embodiments of the present invention to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. An interactive panoramic spatial ray tracing method based on RGBD panoramic video, characterized in that: The steps include: Step 1: Input the panoramic image of the current frame of the RGBD panoramic video, the user's perspective and the information of the virtual object to be inserted, generate the screen space geometry buffer and the panoramic image space geometry buffer under the current perspective, and store the template function image, position image and normal image of the virtual object in the screen space and the panoramic image space in the buffer. At the same time, process the depth channel in the RGBD panoramic image to generate the corresponding hierarchical refinement depth texture map MIPMAPS; Step 2: Based on the position in the geometry buffer in the screen space under the user's current perspective in step 1 and the pixel-by-pixel position and normal information in the normal image, a corresponding multi-scale structured sparse sampling template cache is generated, and a sparse tracking acceleration algorithm is used to further accelerate rendering; Step 3: Using a back-face-aware ray tracing algorithm, ray tracing is performed on the images in the screen space geometry buffer and the panoramic image space geometry buffer under the current viewing angle generated in step 1, and the corresponding hierarchically refined depth texture map MIPMAPS. The density distribution of the light emitted during ray tracing is controlled by the multi-scale structured sparse sampling template cache in step 2, and the intersection points of the light and the three-dimensional geometry are calculated to simulate the two-way interaction between the virtual object and the real-world geometry, and obtain the noisy Monte Carlo incident radiance image of the current scene; Step 4: The noisy Monte Carlo incident radiance image of the current scene obtained in step 3 is interpolated using the sampling points in the multi-scale structured sparse sampling template buffer generated in step 2 to generate a dense incident radiance image; Step 5: Use a joint bilateral spatial filter to perform spatiotemporal filtering and denoising on the dense incident radiance image in step 4, and then perform temporal filtering on the spatial filtering result in the time domain through the motion vector in the screen space to obtain a reconstructed Monte Carlo incident radiance image of the scene at the current viewing angle before and after the virtual object is added; Step 6: Using the Monte Carlo incident radiance image of the scene at the current perspective before and after adding the virtual object reconstructed in step 5, combined with the data of the screen space geometry buffer in step 1, the differential rendering method is used to render the virtual-reality fusion result image of the virtual object and the real world at the user's current perspective.

2. The interactive panoramic spatial ray tracing method based on RGBD panoramic video according to claim 1, characterized in that: In step 3: the back-face-aware ray tracing algorithm is specifically implemented as follows: (1) When ray tracing is performed on the image space geometry buffer and the panoramic image space geometry buffer through the control of the multi-scale structured sparse sampling template cache, each time a ray is traced and an image pixel in the screen space geometry buffer intersects, it is detected whether the pixel belongs to a depth gap point; (2) If the detected pixel belongs to a depth gap point, start sampling from the back side of the three-dimensional surface where the pixel is located, and select a geometrically related replacement point as the true intersection point. The sampling of the replacement point follows the following principles: the replacement point should be in the same plane as the light, line of sight, and original intersection pixel point in three dimensions; the distance from the replacement point to the original intersection pixel point should be equal to the physical limit; (3) Finally, through such a back-face-aware ray tracing algorithm, the radiance value of each pixel corresponding to each ray is calculated, thereby realizing a two-way interaction between the simulated object and the real-world geometry, and finally obtaining a noisy Monte Carlo incident radiance image of the current scene.

3. The interactive panoramic spatial ray tracing method based on RGBD panoramic video according to claim 1, characterized in that: In step 2, the sparse tracking acceleration algorithm is specifically implemented as follows: Sparse tracking acceleration generates a multi-scale structured sparse sampling template cache to control whether to track the light of a specific pixel; for each pixel in screen space, the position and normal distribution of its neighboring pixels and whether it intersects with the virtual object determine whether to track the light. The calculation method of the result tracking weight buffer is described as follows: (1) in, is any pixel in the current screen space image, yes Neighborhood In practice, Set to Pixel range, ; For frame , is a binary parameter calculated in the virtual object template function image, which is used to indicate the pixel Whether it has been illuminated by virtual objects in the last few frames, and represent The normal and position of and represent The normal and position of It is a manually set normalization parameter. After all pixels in the screen space image are calculated, the tracking weight buffer is obtained; From formula (1), we get the tracking weight buffer, and then we set the threshold manually by two and , , divide the pixels in the tracking weight buffer into three regions; for any pixel ,Will Trace level The record is as follows: (2) yes The tracking weight calculated by the pixel point. 0, 1, and 2 are specific values ​​of the tracking level. The tracking level of pixels, in practice, will be Set to 0.3, Set to 0.5; construct the screen space image into a multi-scale grid, that is, a multi-scale structured sparse sampling template cache, and emit tracing rays of different densities in these areas according to the areas to which the multi-scale grid belongs.

Images