Virtual viewpoint rendering method, system and device based on multiple rgb-d images

Abstract

The present application belongs to the technical field of computer graphics and computer vision, and discloses a virtual view rendering method, system and device based on multiple RGB-D images, which adopts an instant truncated signed distance function method, generates a new depth map through virtual rays, introduces a multi-weight projection texture mapping method, and fuses color information from multiple views together; calculates a depth confidence map to determine the participation of inaccurate measurement in the TSDF fusion, depth map filtering and projection texture mapping process during the rendering process, reduces the influence of noise and inaccurate depth measurement on the rendering quality; through the image-driven TSDF fusion method, fusion is carried out along the rays, real-time TSDF value calculation is realized, and the projection texture mapping color fusion method with a reduced search range and multiple weights is adopted. The present application can efficiently generate high-quality virtual view rendering results in real time.

Description

Technical Field

[0001] This invention belongs to the fields of computer graphics and computer vision technology, and particularly relates to a virtual perspective rendering method, system and device based on multiple RGB-D images. Background Technology

[0002] Currently, Virtual Viewpoint Rendering Technology is a hot research topic in computer graphics. This technology aims to synthesize new viewpoint images using limited viewpoint data, thereby achieving the effect of observing scene targets from different angles. The method is simple in principle and easy to implement, and can be widely applied in film and television production, sports broadcasting, distance education, online conferencing, medical imaging, and many other fields. While existing technologies can generate realistic rendering results through complex calculations and inferences, their high computational cost results in low operating efficiency, making them unsuitable for applications with high real-time requirements, thus hindering the widespread application of this technology in real-world scenarios. Furthermore, traditional virtual viewpoint technologies, while offering a variable viewpoint range that closely approximates the original real image, limit the rendered viewpoint and lack the ability to freely roam from any angle, resulting in a lack of spatial depth and immersion. Therefore, researching a high-quality real-time virtual viewpoint rendering algorithm not only has significant theoretical research value but will also fully realize the practical application value of this method.

[0003] Improving the rendering quality and speed of rendering algorithms has always been a focus of academic and industrial attention. With ongoing research, many virtual perspective rendering technologies have emerged in recent years. These research results can be broadly categorized into two types: model-based rendering and image-based rendering. Model-based rendering algorithms employ multi-view solid geometry techniques to reconstruct the entire scene in 3D, allowing free control of the virtual perspective within the reconstructed scene to obtain the final rendering result. In 2019, Chen et al. first constructed a sparse point cloud of the target and determined the 3D mask of the reconstructed target using a connected component labeling algorithm, then utilized the 3D mask to accelerate the dense reconstruction using a visual shell algorithm. This traditional algorithm has a complex and cumbersome data processing pipeline; although it can generate high-quality 3D reconstruction results, its real-time performance is poor, and the optimization space for algorithm speed is relatively limited. Image-based rendering methods directly render a new perspective image from the input image. In recent years, several virtual perspective rendering technologies based on RGBD images have emerged. In 2017, Kaiwen Guo et al. proposed a real-time method using a single-view RGB-D input (a depth sensor integrated with a color camera) to simultaneously reconstruct a leisure scene with detailed geometric models, surface albedo, and non-rigid motion per frame. In the same year, a virtual mirror rendering system using a commercial structured light RGB-D camera network utilized depth information to track the viewpoint, then rendered dynamic scenes by tracking light rays from each scene point to reflection points on a virtual mirror, finally determining the appropriate location and color values ​​on the display surface for the area the virtual reflected light rays traversed before reaching the viewpoint. Overall, image-based new perspective generation algorithms have a simple architecture and can quickly generate new perspective images even with dense camera arrays. However, due to the lack of spatial geometric information, their virtual perspective has limited degrees of freedom, and it is difficult to obtain high-quality output results in scenes with sparse camera arrangements. How to fully utilize the geometric relationships between multiple views to achieve high-quality virtual perspective rendering with sparse camera arrangements is a problem that urgently needs to be solved.

[0004] Based on the above analysis, the problems and shortcomings of the existing technology are as follows: the virtual perspective of the existing technology has limited freedom, and it is difficult to obtain high-quality output results in scenarios with sparse camera arrangement. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a virtual perspective rendering method, system, and device based on multiple RGB-D images.

[0006] This invention is implemented as follows: Under the condition of providing sparse RGB-D viewpoints as raw input, a virtual viewpoint rendering method based on multiple RGB-D images achieves real-time, high-quality free-viewpoint rendering. This is mainly achieved through a combination of two parts: online truncated signed distance function (TSDF) based on ray tracing technology and multi-weighted texture projection. First, starting from the virtual viewpoint, the distance from each sampling point to the object surface is calculated online while stepping along the ray. To increase the efficiency of ray stepping, the original point cloud is rasterized to determine and narrow the sampling interval, enabling rapid acquisition of the depth map of the virtual viewpoint. Then, the obtained 3D points on the surface are projected and calculated. The color information from multiple viewpoints is fused together, taking into account parameters such as viewpoint angle, normal vector, and visibility, to generate the final high-quality rendering result. Furthermore, this invention introduces depth confidence calculation during the calculation process to reduce the impact of noise from the original acquisition device on TSDF fusion, depth map filtering, and projected texture mapping during rendering.

[0007] Furthermore, the virtual viewpoint rendering method based on multi-RGB-D images includes the following steps:

[0008] Step 1: Data preparation, including scene color images, depth images, and multi-camera calibration intrinsic and extrinsic parameters;

[0009] Step two: Determine the subset of cameras participating in rendering based on the virtual viewpoint. For each participating camera, calculate a depth confidence map to evaluate the reliability of each depth pixel. The weight of the depth confidence map is determined by two factors: first, the angle between the camera's optical axis and the normal vector of each depth pixel; second, the degree of fluctuation of the depth pixel value in its neighborhood. The depth confidence map serves as a weighting coefficient; the larger the weight value, the greater the contribution of the camera's viewpoint to the rendering result, and it is used to adjust the degree of participation of inaccurate points in subsequent rendering processes.

[0010] Step 3: Determine the optimal search interval for each ray tracing to avoid traversing the entire sampling space. Back-project the depth map into the world space to obtain the point cloud set of the human body. Then, rasterize the point cloud from a virtual perspective to find a suitable search interval so that the surface intersections of the virtual perspective can be accurately found during the ray tracing process.

[0011] Step 4: Starting from the virtual viewpoint's two-dimensional imaging plane, emit a search ray equivalent to the image resolution; obtain the 3D spatial position and depth information corresponding to each pixel point of the virtual viewpoint on the imaging plane;

[0012] Step 5: By using ray tracing to track the intersection of the ray and the target surface, a virtual view surface is obtained without explicitly creating voxels. The ray is moved along the ray with a fixed step size within the search interval. The TSDF value is calculated online for each sampling point, and the intersection of the ray and the surface is determined.

[0013] Step six: Integrate the depth confidence map into the image-based rendering algorithm mentioned earlier, filter the original depth map, and use the depth confidence map to adjust the participation level of inaccurate points;

[0014] Step seven involves employing a hybrid weighting method to mitigate the impact of noise and accuracy. First, a weight map for each camera relative to the virtual view needs to be calculated. This weight map is generated from three influencing factors: visibility test, view contribution, and the correlation of surface normal vectors. In the visibility test, each pixel in the scene is projected onto the virtual viewpoint, and the depth value of the projected pixel is compared with the depth map of the virtual viewpoint. If the depth value of the projected pixel is smaller, it indicates that the pixel is visible in the virtual viewpoint; otherwise, it indicates that it is occluded. The view contribution is usually related to the angle between the camera viewpoint and the virtual viewpoint. All reference viewpoints are sorted; the smaller the spatial angle, the more similar the reference viewpoint is to the virtual viewpoint. Each camera provides different surface normal information, and the correlation between the surface normal information provided by different cameras and the surface normal information of the virtual viewpoint needs to be considered.

[0015] Step 8: After obtaining the weight map of each camera's contribution to the virtual view, a multi-weighted projection texture mapping method is used to accurately fuse the original color information into the reconstructed 3D points using weighted blending, and to fuse the texture information captured by different cameras together, thereby generating the final virtual view rendering result.

[0016] Furthermore, step one includes: using a sparse set of RGB-D cameras to capture geometric and texture information, performing simple depth filtering based on a depth threshold, and calibrating the camera array to obtain the three-dimensional world point P = (X, Y, Z, 1). T The pixel coordinates on the imaging plane are p = (u, v, 1). T Mapping relationship:

[0017] sp=MP

[0018] Where s is the scale factor and M is the mapping matrix;

[0019] Step 2: Calculate a depth confidence map, denoted as W. i Where i∈M, the reliability of the original depth measurement depends on the angle between the camera's line of sight and the surface normal. The depth of pixel u obtained from an almost orthogonal viewpoint is more accurate, and the weights are used in the following way. Calculate the depth confidence value for each pixel u:

[0020]

[0021] The normalized coordinates of the point corresponding to the depth value are represented as: At this point, the surface normal N is calculated by taking the cross product of adjacent pixels. u , Introduction The definition is as follows:

[0022]

[0023] Here, ε1 and ε2 represent the minimum and maximum tolerable errors, respectively. Represents pixel d i N i The depth confidence map weights for each depth value are obtained by combining two factors: nearest neighbors. After calculating the weighted graph, a threshold is applied to remove depth values ​​with lower weights.

[0024] Furthermore, step three first defines a cubic bounding box that surrounds the sampling space, denoted as [B]. min B max ]∈R 3 Subsequently, the axis-aligned bounding box (AABB) algorithm is used to calculate each ray p = o + td, p ∈ R. 3 The intersection point t n and t f Let o represent the starting point of the ray, d represent the direction of the ray, and t represent the distance from the starting point o to the intersection point. Project each pixel onto the virtual viewpoint and use the perspective projection equation to obtain the projected coordinates u. proj and the corresponding depth z proj :

[0025] p proj =K v T v {T i} -1 δ i -1 (u,d)

[0026]

[0027] z proj =p proj.z

[0028] K v and T v These represent the intrinsic and extrinsic parameters of the virtual camera, respectively. Two caches, F, were created. n and F fIn CUDA, atomic operations are used to update the nearest and farthest depths of the virtual view projected from the point cloud, denoted as z. n and z f The specific update process is as follows:

[0029] z n =F n (u proj =atomicMin(F) n (u proj ),z proj )

[0030] z f =F f (u proj =atomicMax(F f (u proj ),z proj )

[0031] Here, the size of each projection point is set to 5. Then the final search interval is adjusted to [max(t)]. n ,z n )-ψ,min(t f ,z f )+ψ], where ψ represents an offset value used to expand the interval outward;

[0032] Step four: Starting from the virtual viewpoint optical center, emit a cluster of search rays connecting the optical center and the pixels of the two-dimensional imaging plane:

[0033] p(t,u,v)=p0+tn(u,v)

[0034] Where p0 is the position of the virtual viewpoint in the world coordinate system, n(u,v) is the direction vector of the ray projected from pixel (u,v), and t is the search interval.

[0035] Furthermore, step five applies to any point p∈R on the ray. 3 Use the corresponding depth value d i Calculate the z-coordinate of a point in camera i to calculate the corresponding TSDF value s. i . s i The calculation formula is as follows:

[0036]

[0037] Where τ represents the cutoff distance. The final value s of the current point is obtained by fusing the TSDF values ​​contributed by the selected M cameras, and the fusing formula is as follows:

[0038]

[0039] For si For points ≤-1 or points whose projections fall outside the image, this invention sets w. i =0, a positive value of s indicates that the point is outside the surface, while a negative value indicates that the point is inside the surface. The algorithm terminates when the sign of the TSDF value changes from positive to negative.

[0040] Step six involves projecting each surface point on the virtual view onto M cameras, and back-projecting the pixels into 3D space. If no occlusion occurs, each point is considered a neighbor to the corresponding virtual point, and weights are assigned to all neighboring points using a depth confidence map. For any pixel u on the virtual view... v The processing procedure is as follows:

[0041] u i =δ i (T i {T v} -1 δ v -1 (u v ,d v ))

[0042]

[0043] Where, ω v This indicates an occlusion state.

[0044] Furthermore, step seven includes:

[0045] (1) Visibility Test: A depth test strategy is used to determine the weights. For each camera view, the actual depth map is considered as the nearest surface. To determine the surface point p w Whether a point is visible in the camera view is determined by projecting it onto each camera and assigning the projected point's value d to the depth map. proj Depth d relative to the z-axis coordinate in the camera coordinate system z For comparison, the visibility weight ω vis Calculate using the following formula:

[0046]

[0047] λ represents a tolerance level, which is usually set to a constant.

[0048] (2) Viewpoint contribution: The contribution of each camera's viewpoint is determined by connecting the actual camera center p. r and virtual camera center p v Point P ω The angle between them determines the cosine value ω of the angle. l Used to indicate contribution:

[0049]

[0050] (3) Surface normal correlation: To calculate the surface normal correlation ω for each camera n Use point p ω Find the vertex normal vector n, and calculate the connection between the real camera center p. r and p ω The lines between The angle between the two sides, the cosine of which is used to represent ω. n ;

[0051]

[0052] For each camera i, the final blending weight factor ω i It is calculated by multiplying the three weighting factors mentioned above, and the depth confidence W is... i This weighting factor, added as an additional weight in the calculation, is used for color blending in the virtual view:

[0053] ω i =W i (u)·ω v i ·ω l i ·ω n i

[0054] Step 8: Among the M viewpoints involved, select the B viewpoints with higher weights to participate in the final color mixing (B≤M), while the weights of the remaining M–B viewpoints are set to zero, and the virtual viewpoint p is... ω The texture information is projected onto each camera and calculated based on the normalized weights. To mix the colors of the final pixels;

[0055]

[0056] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the virtual perspective rendering method based on multiple RGB-D images.

[0057] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the virtual perspective rendering method based on multiple RGB-D images.

[0058] Another objective of this invention is to provide an information data processing terminal for implementing the virtual perspective rendering method based on multiple RGB-D images.

[0059] Another object of the present invention is to provide a virtual perspective rendering system based on multiple RGB-D images, employing the aforementioned virtual perspective rendering method based on multiple RGB-D images, wherein the virtual perspective rendering system based on multiple RGB-D images comprises:

[0060] The information fusion module is used to generate a new depth map through virtual rays using the real-time truncated signed distance function (TSDF) method; and introduces a multi-weighted projection texture mapping method to fuse color information from multiple viewpoints.

[0061] The information calculation module is used to calculate the depth confidence map to determine the involvement of inaccurate measurements in the rendering process of TSDF fusion, depth map filtering, and projected texture mapping.

[0062] The numerical calculation module is used to perform fusion along the ray using an image-driven TSDF fusion method, realizing real-time TSDF value calculation, and employing a projection texture mapping color fusion method with narrowed search range and multiple weights.

[0063] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0064] First, achieving real-time, high-quality free-view rendering has always been a challenge in existing technologies. Traditional methods typically require long computation times when processing large-scale RGB-D data, and the rendering results may be affected by noise and inaccurate points. These problems complicate providing high-quality free-view rendering in real-time applications. This invention addresses this issue by employing a series of technical solutions. First, by using the online truncated symbolic distance field (TSDF) in ray tracing technology to calculate the distance from each sampling point to the object surface in real time, the rendering speed is effectively improved. Furthermore, rasterization of the original point cloud is performed to determine and narrow the sampling interval, further accelerating the acquisition of the depth map. Most importantly, a multi-weighted texture projection method fuses color information from multiple perspectives to generate a final high-quality rendering result, thus resolving the contradiction between real-time and high-quality rendering. Simultaneously, the influence of noise from the original acquisition device on the rendering process is reduced by introducing the calculation of depth confidence. The depth confidence map is used to weigh the contributions of different camera perspectives to adjust the participation of inaccurate points in subsequent rendering processes. This step effectively reduces the impact of noise on TSDF fusion, depth map filtering, and projected texture mapping processes, improving the quality of the rendering results.

[0065] The technical solution of this invention achieves real-time, high-quality free-viewpoint rendering, making it widely applicable in virtual reality, augmented reality, and 3D modeling. Secondly, the application of ray tracing and TSDF improves rendering speed, making it suitable for real-time applications. Then, the introduction of depth confidence calculation reduces the impact of noise, improves the accuracy of rendering results, and makes the visualization clearer and more realistic. Finally, the multi-weighted texture projection method effectively fuses color information from multiple perspectives, providing more possibilities for generating high-quality virtual viewpoint rendering results—a truly innovative technical achievement. In summary, this invention overcomes the difficulties in existing technologies, providing an innovative virtual viewpoint rendering method aimed at offering higher-quality rendering effects and broader application prospects for virtual reality and related fields.

[0066] Secondly, one of the core advantages of this invention is its ability to achieve a balance between real-time performance and high-quality rendering. Through the combined application of ray tracing technology, online truncated symbolic distance field (TSDF), and multi-weighted texture projection, high-quality virtual perspective rendering is achieved, enabling it to provide satisfactory visual effects in fields such as virtual reality, augmented reality, and 3D modeling. Simultaneously, the introduction of depth confidence effectively suppresses device noise, significantly improving the accuracy of rendering results, reducing inaccuracies and visual noise, and making the final rendering result more realistic and clear. Therefore, the technical solution of this invention has broad application potential and is applicable to multiple fields, including virtual reality, augmented reality, video games, 3D modeling, and medical imaging. Its real-time and high-quality rendering capabilities, as well as noise suppression technology, provide more opportunities for innovation and development in these application areas. In summary, the technical solution of this invention provides a promising solution for achieving real-time, high-quality free-viewpoint rendering, possessing broad application prospects and the potential to improve rendering accuracy and user experience, thus having a positive impact on the technological development of virtual reality and related fields.

[0067] Third, as supplementary evidence of the inventive step of the claims of this invention, it is also reflected in the following important aspects:

[0068] (1) The virtual perspective rendering technology solution of the present invention will have broad commercial value in multiple fields. Industries such as virtual reality, augmented reality, video games, 3D modeling, and medical imaging can all benefit from high-quality free-viewpoint rendering. Commercial applications include virtual tourism, virtual training, medical surgical simulation, architectural design and visualization, and the entertainment industry, and these fields have huge market potential.

[0069] (2) The industry, both domestically and internationally, has been seeking solutions to provide more realistic and lifelike experiences in virtual reality and augmented reality. This invention fills a technological gap in the field of real-time, high-quality free-viewpoint rendering. Existing technologies often involve a trade-off between achieving real-time rendering and accuracy, while this invention achieves superior performance in both aspects through an innovative method.

[0070] (3) This invention solves the technical challenge of real-time, high-quality free-view rendering. This challenge involves providing real-time performance without sacrificing rendering quality. This is an ongoing challenge in virtual reality and augmented reality applications, and this invention provides an effective solution to achieve this goal. Furthermore, this invention successfully addresses the impact of noise on rendering results, improving the accuracy of the rendering results.

[0071] (4) This invention employs various technical means, such as ray tracing, TSDF, and depth confidence calculation, to overcome the limitations of traditional technologies in real-time high-quality rendering. Traditional methods may be limited by computational complexity and noise issues, while this invention uses a novel approach to overcome these biases, providing a more competitive solution. Overcoming these technical biases gives this invention greater market potential and commercial competitiveness, providing the industry and users with a more innovative virtual perspective rendering experience. Attached Figure Description

[0072] Figure 1 This is a flowchart of a virtual perspective rendering method based on multiple RGB-D images provided in an embodiment of the present invention;

[0073] Figure 2 This is a schematic diagram of the virtual viewpoint rendering method based on multiple RGB-D images provided in this embodiment of the invention;

[0074] Figure 3 This is a data preparation example diagram provided in an embodiment of the present invention;

[0075] Figure 4 This is a schematic diagram illustrating the impact of unreliable depth values ​​provided in this embodiment of the invention on the final rendering result;

[0076] Figure 5 This is a weighted graph-based fusion rendering result image provided in an embodiment of the present invention;

[0077] Figure 6 This is a schematic diagram of the structure of a virtual perspective rendering system based on multiple RGB-D images provided in an embodiment of the present invention. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0079] like Figure 1 As shown, the virtual viewpoint rendering method based on multi-RGB-D images provided in this embodiment of the invention includes the following steps:

[0080] S101: Employs the instantaneous truncated signed distance function (TSDF) method to generate a new depth map using virtual rays; introduces a multi-weighted projective texture mapping method to fuse color information from multiple viewpoints. Specifically;

[0081] S102: Calculate the depth confidence map to determine the involvement of inaccurate measurements in TSDF fusion, depth map filtering and projected texture mapping during the rendering process, in order to reduce the impact of noise and inaccurate depth measurements on rendering quality.

[0082] S103: The image-driven TSDF fusion method is used to fuse along the ray, realizing real-time TSDF value calculation, and adopting a projection texture mapping color fusion method with narrowed search range and multiple weights.

[0083] Example 1:

[0084] The virtual viewpoint rendering method based on multiple RGB-D images provided in this embodiment of the invention includes the following steps:

[0085] Step 1: Data preparation, including scene color images, depth images, and multi-camera calibration intrinsic and extrinsic parameters;

[0086] Step two involves determining the subset of cameras participating in rendering based on the virtual viewpoint. For each participating camera, a depth confidence map is calculated to evaluate the reliability of each depth pixel. The weights of the depth confidence map are determined by two factors: first, the angle between the camera's optical axis and the normal vector of each depth pixel; and second, the degree of fluctuation of the depth pixel value in its neighborhood. These depth confidence maps serve as weighting coefficients; a larger weight value indicates a greater contribution of the camera's viewpoint to the rendering result, and can be used to adjust the participation of inaccurate points in subsequent rendering processes.

[0087] Step three involves determining the optimal search interval for each ray tracing path to avoid traversing the entire sampling space, thereby improving rendering efficiency and performance. This invention back-projects the depth map into world space to obtain a point cloud set of the human body, then rasterizes the point cloud from a virtual perspective to find a suitable search interval. This allows for the rapid and accurate identification of surface intersections from the virtual perspective during ray tracing.

[0088] Step four: Starting from the virtual viewpoint's two-dimensional imaging plane, emit a search ray equivalent in size to the image resolution; this process is to obtain the 3D spatial position and depth information corresponding to each pixel point of the virtual viewpoint on the imaging plane.

[0089] Step 5: By using ray tracing to track the intersection of the ray and the target surface, a virtual view of the surface is obtained without explicitly creating voxels. The ray is traversed along the search interval using a fixed step size, and the TSDF value is calculated online for each sampling point to determine the intersection of the ray and the surface.

[0090] Step six involves incorporating the depth confidence map into the previously mentioned image-based rendering algorithm. The original depth map is filtered, and the depth confidence map is used to adjust the participation of inaccurate points. This can significantly reduce interference caused by RGB-D camera noise and inaccurate depth measurements, and generate high-quality rendering results.

[0091] Step seven employs a hybrid weighting method to mitigate the impact of noise and accuracy. First, a weight map for each camera relative to the virtual view needs to be calculated. This weight map is generated from three influencing factors: visibility test, view contribution, and the correlation of surface normal vectors. In the visibility test, each pixel in the scene is projected onto the virtual viewpoint, and the depth value of the projected pixel is compared with the depth map of the virtual viewpoint. If the depth value of the projected pixel is smaller, it indicates that the pixel is visible in the virtual viewpoint; otherwise, it indicates that it is occluded. The view contribution is usually related to the angle between the camera viewpoint and the virtual viewpoint. All reference viewpoints are sorted; the smaller the spatial angle, the more similar the reference viewpoint is to the virtual viewpoint. Each camera provides different surface normal information; the correlation between the surface normal information provided by different cameras and the surface normal information of the virtual viewpoint needs to be considered.

[0092] Step eight involves obtaining a weighted map of each camera's contribution to the virtual view, then employing a multi-weighted projection texture mapping method to accurately fuse the original color information onto the reconstructed 3D points using weighted blending. This process involves fusing texture information captured by different cameras to generate the final virtual view rendering result.

[0093] Example 2:

[0094] The virtual viewpoint rendering method based on multiple RGB-D images provided in this embodiment of the invention includes the following steps:

[0095] Step one: The method of this invention requires the following data: multi-view target color images, depth images, and multi-camera internal and external calibration parameters, such as... Figure 3As shown, a sparse array of RGB-D cameras is first used to capture geometric and texture information. Since this invention primarily focuses on rendering human-related data, background elements other than the human body need to be removed. Specifically, depth is simply filtered based on a depth threshold. Furthermore, the camera array needs to be calibrated to obtain the 3D world point P = (X, Y, Z, 1). T The pixel coordinates p = (u, v, 1) on the imaging plane T Mapping relationship:

[0096] sp=MP

[0097] Where s is the scale factor and M is the mapping matrix.

[0098] Step two: To mitigate the impact of sensor noise on rendering quality, this invention introduces a confidence assessment into the original depth map, constructing a confidence map to represent the reliability of each depth pixel. A depth confidence map is calculated for each participating camera to evaluate the reliability of each depth pixel. These depth confidence maps serve as weighting coefficients to adjust the participation of inaccurate points in subsequent rendering processes. First, a depth confidence map, denoted as W, is calculated. i Where i ∈ M. The reliability of the original depth measurement depends on the angle between the camera's line of sight and the surface normal. In other words, the depth of pixel u obtained from a nearly orthogonal viewpoint is more accurate. Therefore, this invention uses weights in the following manner. Calculate the depth confidence value for each pixel u:

[0099]

[0100] The normalized coordinates of the point corresponding to the depth value are represented as: At this point, the surface normal N is calculated by taking the cross product of adjacent pixels. u To account for regions in the measurement data that exhibit significant variations compared to their surrounding neighborhood and have lower reliability, and to mitigate the effects of edge and high-variance noise, this invention introduces... Its definition is as follows:

[0101]

[0102] Here, ε1 and ε2 represent the minimum and maximum tolerable errors, respectively. Represents pixel d i N i The nearest neighbors. The depth confidence map weight for each depth value is obtained by combining two factors: After calculating the weighted graph, this invention applies a threshold to remove depth values ​​with lower weights. Figure 3 This demonstrates the impact of unreliable depth values ​​on the final rendering result, highlighting the need to effectively reduce their influence to improve rendering quality.

[0103] Step three: This invention proposes a method to determine the search interval for each ray, thereby avoiding the need to traverse the entire sampling space. This invention first defines a cubic bounding box enclosing the sampling space, denoted as [B...]. min B max ]∈R 3 Subsequently, this invention uses the Axis-Aligned Bounding Box (AABB) algorithm to calculate each ray p = o + td ∈ R. 3 The intersection point t n and t f Here, o represents the starting point of the ray, d represents the direction of the ray, and t represents the distance from the starting point o to the intersection point. To further reduce the search interval, this invention rasterizes the point cloud obtained from the original depth map. Specifically, this invention projects each pixel onto a virtual viewpoint and uses the perspective projection equation to obtain the projected coordinates u. proj and the corresponding depth z proj :

[0104] p proj =K v T v {T i} -1 δ i -1 (u,d)

[0105]

[0106] z proj =p proj,z

[0107] Here, K v and T v These represent the intrinsic and extrinsic parameters of the virtual camera, respectively. Subsequently, this invention creates two caches, F... n and F f In CUDA, atomic operations are used to update the nearest and farthest depths of the virtual view projected from the point cloud, denoted as z. n and z f The specific update process is as follows:

[0108] z n =F n (u proj =atomicMin(F) n (u proj ),z proj )

[0109] z f =F f (u proj =atomicMax(F f(u proj ),z proj )

[0110] This invention sets the size of each projection point to 5. Then, the final search interval is adjusted to [max(t)]. n ,z n )-ψ,min(t f ,z f )+ψ], where ψ represents an offset value used to expand the interval outward.

[0111] Step four: Starting from the virtual viewpoint optical center, emit a cluster of search rays connecting the optical center and the pixels of the two-dimensional imaging plane:

[0112] p(t,u,v)=p0+tn(u,v)

[0113] Where p0 is the position of the virtual viewpoint in the world coordinate system, n(u,v) is the direction vector of the ray projected from pixel (u,v), and t is the search interval.

[0114] Step five: The present invention proceeds along the ray within the search interval, calculates the TSDF value using a fixed step size, and determines the intersection point of the ray with the surface. For any point p∈R on the ray... 3 The corresponding depth value d can be used. i Calculate the z-coordinate of a point in camera i to calculate the corresponding TSDF value s. i . s i The calculation formula is as follows:

[0115]

[0116] Where τ represents the cutoff distance. The final value s of the current point is obtained by fusing the TSDF values ​​contributed by the selected M cameras. In the method of this invention, the contribution of each camera is not calculated uniformly. Instead, the contribution is weighted according to the previously calculated depth confidence map to reduce the influence of inaccurate depth measurements on the final surface representation. The fusion formula is as follows:

[0117]

[0118] For s i For points ≤-1 or points whose projections fall outside the image, this invention sets w. i =0. A positive value of s indicates that the point is outside the surface, while a negative value indicates that the point is inside the surface. The algorithm terminates when the sign of the TSDF value changes from positive to negative. To obtain a more accurate surface representation, this invention performs linear interpolation between adjacent voxels.

[0119] Step six: Unlike traditional TSDF methods that utilize neighborhood information to achieve smoother surfaces, the image-based method of this invention independently computes each ray, potentially resulting in less smooth surface depth. However, obtaining neighboring points incurs significant computational overhead. Therefore, this invention proposes a point filtering method based on a 2D depth map for processing within local regions. First, this invention projects each surface point on the virtual view onto M cameras. Next, this invention considers the 2D neighborhood of neighboring pixels around the projected point. Then, these pixels are back-projected into 3D space. If no occlusion occurs, this invention treats these points as neighborhood points corresponding to the virtual point. This invention uses a depth confidence map to assign weights to all neighboring points, thereby obtaining smoother points. For any pixel u on the virtual view... v The processing procedure is as follows:

[0120] u i =δ i (T i {T v} -1 δ v -1 (u v ,d v ))

[0121]

[0122] Where, ω v This indicates an occlusion state.

[0123] Step seven: This invention employs a hybrid weighting method to mitigate the impact of noise and accuracy. This weighting map is generated by considering three influencing factors: visibility testing, viewpoint-related contributions, and the correlation of surface normal vectors, all of which contribute to obtaining more accurate rendering results.

[0124] (1) Visibility Testing: The binary weighting method avoids color weighting errors due to occlusion. It employs a depth testing strategy to determine weights. For each camera view, the actual depth map is considered the nearest surface. To determine the surface point p... w Whether a point is visible in the camera view is determined by its projection onto each camera, and the value d of the projected point on the depth map is recorded. proj Depth d relative to the z-axis coordinate in the camera coordinate system z Compare the visibility weights ω. vis The following formula can be used to calculate:

[0125]

[0126] λ represents a tolerance level and is usually set to a constant.

[0127] (2) Viewpoint contribution: The contribution of each camera's viewpoint is determined by connecting the actual camera center p. r and virtual camera center p v Point P ω The angle between them determines the value. The cosine of this angle is ω. l Used to indicate contribution:

[0128]

[0129] (3) Surface normal correlation: To calculate the surface normal correlation ω for each camera n This invention uses point p ω Find the vertex normal vector n, and calculate the connection between the real camera center p. r and p ω The lines between The angle between the two sides. The cosine of this angle is used to represent ω. n .

[0130]

[0131] For each camera i, the final mixing weight factor ω i It is calculated by multiplying the three weighting factors mentioned above. To mitigate the impact of inaccurate pixels (such as pixels at edges) on the final color blending, this invention uses the depth confidence W. i This weighting factor is added to the calculation as an additional weight. Subsequently, this weighting factor is used for color blending in the virtual view.

[0132] ω i =W i (u)·ω v i ·ω l i ·ω n i

[0133] Step eight: After obtaining the weight map of each camera's contribution to the virtual view, this invention uses weighted blending to achieve the final rendering result. Of the M views involved, B views with higher weights are selected to participate in the final color blending (B≤M), while the weights of the remaining M–B views are set to zero. Next, this invention will apply weighted blending to the virtual view point p... ω The texture information is projected onto each camera and calculated based on the normalized weights. This is used to blend the colors of the final pixels. The weighted graph-based fusion rendering result is as follows: Figure 5 As shown.

[0134]

[0135] This invention proposes a virtual viewpoint rendering algorithm based on multiple RGB-D images. The image-driven TSDF fusion method of this invention fuses along rays, achieving real-time TSDF value calculation. It also employs a narrowed search range and a multi-weighted projection texture mapping color fusion method, thus efficiently generating high-quality virtual viewpoint rendering results in real-time. The parallel rendering algorithm of this invention can process input from eight RGB-D cameras in real-time, performing high-quality rendering at speeds exceeding 100fps. Furthermore, it uses a depth confidence map to reduce the involvement of inaccurate measurements in TSDF fusion, depth map smoothing, and projection texture mapping during the rendering process, thereby reducing the impact of noise and inaccurate depth measurements on rendering quality.

[0136] like Figure 6 As shown, the virtual viewpoint rendering system based on multi-RGB-D images provided in this embodiment of the invention includes:

[0137] The information fusion module is used to generate new depth maps using virtual rays by employing the real-time truncated signed distance function (TSDF) method; and introduces a multi-weighted projective texture mapping method to fuse color information from multiple viewpoints.

[0138] The information calculation module is used to calculate the depth confidence map to determine the involvement of inaccurate measurements in the TSDF fusion, depth map filtering and projected texture mapping processes during rendering, in order to reduce the impact of noise and inaccurate depth measurements on rendering quality.

[0139] The numerical calculation module is used to perform fusion along the ray using the image-driven TSDF fusion method, realizing real-time TSDF value calculation, and adopting a projection texture mapping color fusion method with narrowed search range and multiple weights.

[0140] II. Application Examples. To demonstrate the inventiveness and technical value of the technical solution of this invention, this section provides application examples of the technical solution of the claims on specific products or related technologies.

[0141] This invention can be widely applied to various scenarios. For example, in live streaming of remote meetings, participants typically use video conferencing tools for interaction, but traditional tools often offer limited perspectives, making it difficult for participants to gain a comprehensive meeting experience. The technical solution of this invention deploys multiple RGB-D cameras within the meeting venue to capture panoramic color and depth data. Remote participants can freely browse and select different perspectives within a virtual reality device, as if they were truly present. Simultaneously, the technology of this invention ensures real-time performance and high-quality rendering, allowing remote participants to obtain a clear and realistic meeting scene without being affected by latency or visual noise. Ultimately, by employing the virtual perspective rendering technology of this invention, live streaming of remote meetings becomes more attractive and immersive. Remote participants can better understand and navigate the meeting venue, experiencing a more realistic meeting atmosphere. This improves the quality of remote meetings and is expected to attract more clients and users.

[0142] In traditional sports broadcasting, limited perspectives are often provided, potentially causing viewers to miss key moments or angles. The virtual perspective rendering technology provided by this invention offers viewers multiple viewpoints for a more engaging viewing experience. Specifically, multiple RGB-D cameras are set up within the stadium to capture various angles and details of the game. This technology allows viewers to freely choose different perspectives, such as player view, global view, or audience view, for a better understanding of the match. The entire rendering process is real-time, ensuring smooth perspective switching without noticeable delays. By employing this virtual perspective rendering technology, sports broadcasting companies offer viewers more viewing angles and a more immersive experience. Viewers are able to participate more deeply in the game, appreciating multiple aspects, improving the quality of sports broadcasts, and increasing viewer interaction and engagement.

[0143] The advantages of this invention are reflected in several aspects: In the data acquisition stage, traditional multi-view reconstruction rendering techniques require a large number of cameras, leading to high data acquisition costs and complex camera deployment. The proposed solution only requires sparse RGB-D cameras, reducing data acquisition costs and simplifying equipment deployment. Furthermore, multiple cameras can work simultaneously to obtain more angle and viewpoint information. Regarding rendering efficiency, previous methods for multi-view rendering consumed significant time for target modeling and image post-processing, resulting in long rendering times unsuitable for real-time applications. The algorithm proposed in this invention is concise and efficient, using GPU hardware parallelization to allow real-time rendering, achieving high-quality rendering within milliseconds, suitable for applications requiring low latency, and significantly improving rendering efficiency. Regarding rendering quality, traditional cameras may struggle to achieve high-quality rendering results in multi-view rendering due to complex data fusion in post-processing, potentially resulting in significant noise and distortion. The technology of this invention offers higher rendering quality, allowing for more accurate fusion of depth information and multi-viewpoint data, reducing noise and distortion. Regarding interactivity, traditional methods typically provide limited interactivity because viewers can only passively view content captured from a single angle. Using the technology of this invention, viewers can interact with the virtual environment, freely choosing different perspectives, making interactive comments, or participating in the experience. This enhances user interactivity and increases user engagement.

[0144] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0145] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A virtual viewpoint rendering method based on multiple RGB-D images, characterized in that, The virtual viewpoint rendering method based on multiple RGB-D images employs the real-time truncated signed distance function (TSDF) method to generate a new depth map through virtual rays; it introduces a multi-weighted projection texture mapping method to fuse color information from multiple viewpoints; it calculates a depth confidence map to determine the involvement of inaccurate measurements in the TSDF fusion, depth map filtering, and projection texture mapping processes during rendering; it uses an image-driven TSDF fusion method to fuse along the rays, achieving real-time TSDF value calculation, and employs a projection texture mapping color fusion method with a narrowed search range and multiple weights. The virtual viewpoint rendering method based on multi-RGB-D images includes the following steps: Step 1: Data preparation, including scene color images, depth images, and multi-camera calibration intrinsic and extrinsic parameters; Step two: Determine the subset of cameras participating in rendering based on the virtual viewpoint. For each participating camera, calculate a depth confidence map to evaluate the reliability of each depth pixel. The weight of the depth confidence map is determined by two factors: first, the angle between the camera's optical axis and the normal vector of each depth pixel; second, the degree of fluctuation of the depth pixel value in its neighborhood. The depth confidence map serves as a weighting coefficient; the larger the weight value, the greater the contribution of the camera's viewpoint to the rendering result, and it is used to adjust the degree of participation of inaccurate points in subsequent rendering processes. Step 3: Determine the optimal search interval for each ray tracing to avoid traversing the entire sampling space. Back-project the depth map into the world space to obtain the point cloud set of the human body. Then, rasterize the point cloud from a virtual perspective to find a suitable search interval so that the surface intersections of the virtual perspective can be accurately found during the ray tracing process. Step 4: Starting from the virtual viewpoint's two-dimensional imaging plane, emit a search ray equivalent to the image resolution; obtain the 3D spatial position and depth information corresponding to each pixel point of the virtual viewpoint on the imaging plane; Step 5: By using ray tracing to track the intersection of the ray and the target surface, a virtual view surface is obtained without explicitly creating voxels. The ray is moved along the ray with a fixed step size within the search interval. The TSDF value is calculated online for each sampling point, and the intersection of the ray and the surface is determined. Step six: Integrate the depth confidence map into the image-based rendering algorithm mentioned earlier, filter the original depth map, and use the depth confidence map to adjust the participation level of inaccurate points. Step seven involves employing a hybrid weighting method to mitigate the impact of noise and accuracy. First, a weight map for each camera relative to the virtual view needs to be calculated. This weight map is generated from three influencing factors: visibility test, view contribution, and the correlation of surface normal vectors. In the visibility test, each pixel in the scene is projected onto the virtual view and its depth value is compared to the depth map of the virtual view. If the depth value of the projected pixel is smaller, it indicates that the pixel is visible in the virtual view; otherwise, it indicates that it is occluded. The view contribution is usually related to the angle between the camera view and the virtual view. All reference views are sorted; the smaller the spatial angle, the more similar the reference view is to the virtual view. Each camera provides different surface normal information; therefore, the correlation between the surface normal information provided by different cameras and the surface normal information of the virtual view needs to be considered. Step 8: After obtaining the weight map of each camera's contribution to the virtual view, a multi-weighted projection texture mapping method is used to accurately fuse the original color information into the reconstructed 3D points using weighted blending, and to fuse the texture information captured by different cameras together, thereby generating the final virtual view rendering result.

2. The virtual viewpoint rendering method based on multiple RGB-D images as described in claim 1, characterized in that, Step one includes: using a sparse set of RGB-D cameras to capture geometric and texture information, performing simple depth filtering based on a depth threshold, and calibrating the camera array to obtain 3D world points. to the pixel coordinates of the imaging plane Mapping relationship: ； Where s is the scale factor and M is the mapping matrix; Step 2: Calculate a depth confidence map, denoted as... ,in The reliability of the raw depth measurement depends on the angle between the camera's line of sight and the surface normal. The depth of pixel u obtained from a nearly orthogonal viewpoint is more accurate, and weights are used in the following way. Calculate the depth confidence value for each pixel u: ； The normalized coordinates of the point corresponding to the depth value are represented as: At this point, the surface normal is calculated by taking the cross product of adjacent pixels. , Introduction The definition is as follows: ； Here, and These represent the minimum and maximum tolerable errors, respectively. Represents pixels of The depth confidence map weights for each depth value are obtained by combining two factors: nearest neighbors. After calculating the weighted graph, a threshold is applied to remove depth values ​​with lower weights.

3. The virtual viewpoint rendering method based on multiple RGB-D images as described in claim 1, characterized in that, Step three first defines a cubic bounding box that encloses the sampling space, denoted as . Subsequently, the axis-aligned bounding box (AABB) algorithm was used to calculate each ray. intersection and t f , Indicates the starting point of the ray. Indicates the direction of the ray. Indicates the starting point The distance to the intersection point is used to project each pixel onto the virtual viewpoint, and the projection coordinates are obtained using the perspective projection equation. and the corresponding depth : ； and These represent the intrinsic and extrinsic parameters of the virtual camera, respectively, and two caches are created. and And in CUDA, atomic operations are used to update the nearest and farthest depths of the virtual view projected from the point cloud, denoted as: and The specific update process is as follows: ； Set the size of each projection point to 5, and then adjust the final search interval to... in This represents an offset value used to expand the interval outwards; Step four: Starting from the virtual viewpoint optical center, emit a cluster of search rays connecting the optical center and the pixels of the two-dimensional imaging plane: ； in, This refers to the position of the virtual viewpoint in the world coordinate system. For pixels The direction vector of the projected ray, where t is the search interval.

4. The virtual viewpoint rendering method based on multiple RGB-D images as described in claim 1, characterized in that, Step five applies to any point on the ray. Use the corresponding depth value d i Calculate the z-coordinate of a point in camera i to determine the corresponding TSDF value. , The calculation formula is as follows: ; For s i Points ≤-1 or points whose projections fall outside the image, set A positive value of s indicates that the point is outside the surface, while a negative value indicates that the point is inside the surface. The algorithm terminates when the sign of the TSDF value changes from positive to negative. Step 6: Project each surface point on the virtual view onto M cameras, back-project the pixels into 3D space. If no occlusion occurs, treat each point as a neighboring point to the virtual point, and assign weights to all neighboring points using the depth confidence map. For any pixel on the virtual view... The processing procedure is as follows: ; in, This indicates an occlusion state.

5. The virtual viewpoint rendering method based on multiple RGB-D images as described in claim 1, characterized in that, Step seven includes: (1) Visibility test: A depth test strategy is used to determine the weights. For each camera view, the actual depth map is regarded as the nearest surface in order to determine the surface points. Whether a point is visible in the camera view is determined by projecting it onto each camera and assigning the projected point's value to the depth map. Depth relative to the z-axis coordinate in the camera coordinate system Comparison, visibility weight Calculate using the following formula: ; This represents a tolerance level and is usually set to a constant. (2) Viewpoint contribution: The contribution of each camera viewpoint is determined by connecting the actual camera center. and virtual camera center p v Point P ω The angle between them determines the cosine value of the angle. Used to indicate contribution: ; (3) Surface normal correlation: In order to calculate the surface normal correlation of each camera Point of use Find the vertex normal vector n, and calculate the connection between the real camera center p. r and P w The lines between The angle between the two sides, the cosine of the angle is used to represent ; ; For each camera i, the final blending weight factor ω i It is calculated by multiplying the three weighting factors mentioned above, thus determining the depth confidence score. This weighting factor, added as an additional weight in the calculation, is used for color blending in the virtual view: oh i =W i (u)·ω v i ·oh l i ·oh n i ; Step 8: Among the M perspectives involved, select B perspectives with higher weights to participate in the final color mixing. And the rest The weight of each viewpoint is set to zero, and the virtual viewpoint is... The texture information is projected onto each camera and adjusted according to the normalized weights. To mix the colors of the final pixels; 。 6. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the virtual perspective rendering method based on multiple RGB-D images as described in any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the virtual view rendering method based on multiple RGB-D images as described in any one of claims 1 to 5.

8. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the virtual perspective rendering method based on multiple RGB-D images as described in any one of claims 1 to 5.

9. A virtual perspective rendering system based on multiple RGB-D images, employing the virtual perspective rendering method based on multiple RGB-D images as described in any one of claims 1 to 5, characterized in that, The virtual viewpoint rendering system based on multi-RGB-D images includes: The information fusion module is used to generate a new depth map through virtual rays using the real-time truncated signed distance function (TSDF) method; and introduces a multi-weighted projection texture mapping method to fuse color information from multiple viewpoints. The information calculation module is used to calculate the depth confidence map to determine the involvement of inaccurate measurements in the rendering process of TSDF fusion, depth map filtering, and projected texture mapping. The numerical calculation module is used to perform fusion along the ray using an image-driven TSDF fusion method, realizing real-time TSDF value calculation, and employing a projection texture mapping color fusion method with narrowed search range and multiple weights.

Images