A method and system for video rendering based on voxel compression
By constructing a voxel model that closely resembles the object's surface by pruning invalid voxels, and then converting it into a point cloud or mesh model, the problem of fast but large-capacity voxel model rendering is solved, achieving efficient rendering compatibility with edge devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING YUANCHUANGLI TECH CO LTD
- Filing Date
- 2022-12-01
- Publication Date
- 2026-04-28
AI Technical Summary
Existing voxel models offer fast rendering speeds but require large file sizes and are incompatible with traditional rendering pipelines, limiting their widespread application at the edge.
By pruning invalid voxels during training, a high-quality voxel model that closely resembles the object's surface is constructed and treated as a point cloud or mesh model, which is then adapted to the traditional rendering pipeline for edge rendering.
It achieves efficient rendering of voxel models, is compatible with traditional rendering pipelines, and is suitable for edge devices such as mobile phones and VR headsets, thus expanding its application scope.
Smart Images
Figure CN119213467B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer graphics and 3D reconstruction technology, and particularly to a video rendering method and system based on voxel compression. Background Technology
[0002] Neural Radiation Field (NeRF) can reconstruct static 3D voxel models from image inputs from multiple viewpoints. NeRF uses neural networks to represent 3D models. Given any point in space and any viewing direction, the NeRF model can output the corresponding volume density and color RGB information. Therefore, for a given viewpoint and camera parameters, a ray can be emitted from the camera's optical center towards a pixel, points can be sampled on the ray, and the corresponding volume density and color information can be calculated. Using the volume rendering formula, the color and depth information of the corresponding pixel can then be calculated, such as... Figure 1 As shown, Neural Radiation Field (NeRF), combined with a time dimension, allows for the reconstruction of dynamic 3D models based on video input from multiple perspectives. However, due to the implicit representation of NeRF, rendering images from new perspectives requires significant computational power; rendering a 1080p image from a new perspective on a high-performance graphics rendering unit (GPU) can take minutes.
[0003] Voxel-based 3D static models directly store volume density and color information at spatial grid points. During rendering, the volume density and color information of sampling points on a given ray can be directly queried and obtained through simple calculations, avoiding neural network inference and thus greatly improving rendering efficiency. Voxel 3D models can also extend the time dimension and be used to represent dynamic 3D models, one example being the Fourier PlenOctree. However, this direct extension of the voxel model leads to a very large model size; a Fourier PlenOctree reconstruction of a dynamic character taking about 2 seconds results in a model size of several gigabytes. Although voxel models render very quickly, their rendering method is incompatible with traditional point cloud or mesh-based rendering pipelines, causing problems with edge rendering and limiting the widespread application of the model. Invention disclosure
[0004] While researching edge rendering of voxel models, the inventors discovered that the aforementioned technical shortcomings were due to the voxels in the voxel model being too dispersed and not closely aligned with the surface of real objects. This necessitates writing computation pipelines on specialized graphics cards for graphics rendering based on this voxel model, limiting its application to edge rendering.
[0005] To address the shortcomings of existing technologies, this invention proposes a video rendering method based on voxel compression, which includes:
[0006] Step 1: Obtain the frame synchronization video for training the voxel model, and use the specified starting video frame in the frame synchronization video as the current frame;
[0007] Step 2: Input the current frame into the neural radiation field, construct the initial voxel model of the current frame, and perform regularization processing on the initial voxel model to remove voxels other than the surface of the target object in the initial voxel model, so as to obtain the compressed voxel model of the current frame.
[0008] Step 3: Based on the frame order of the synchronized video, select the next frame of the current frame as the current frame and execute Step 2 again until the compressed voxel model of the specified final video frame is obtained.
[0009] Step 4: Treat all compressed voxel models as point cloud models or treat the voxels of compressed voxel models as vertices of mesh models. Render each compressed voxel model as an image from a specified viewpoint using the point cloud rendering pipeline or mesh rendering pipeline, thus forming the video rendering result from the starting video frame to the final video frame.
[0010] The video rendering method based on voxel compression, wherein the regularization process includes: when the volume density of voxels in the initial voxel model is less than a preset volume density threshold, it is determined to be an invalid voxel and deleted.
[0011] The aforementioned voxel compression-based video rendering method, wherein the training process of the neural radiation field includes: setting the importance of a specified parameter w in the neural radiation field, using L(θ; w) as the loss function when training the neural radiation field, and if... If the value is less than a certain threshold, the specified parameter w is deleted; where θ represents all parameters in the neural radiation field other than the specified parameter w.
[0012] The video rendering method based on voxel compression, wherein the process of generating the synchronized video frame includes:
[0013] The target object is photographed using a camera array, resulting in an initial video including the video streams of each physical camera after frame synchronization. A new virtual camera video stream is generated for this initial video using a pre-trained model based on implicit representation. This virtual camera video stream is then integrated into the initial video in a frame-synchronized manner to obtain the frame-synchronized video.
[0014] The video rendering method based on voxel compression, wherein the camera array includes a depth camera, and the regularization process includes: determining invalid voxels between the target object and the depth camera in the initial voxel model and deleting them based on the depth information collected by the depth camera.
[0015] The video rendering method based on voxel compression, wherein step 3 includes: after obtaining the compressed voxel model of the final video frame, saving all compressed voxel models in a way that saves the inter-frame model changes.
[0016] In the aforementioned video rendering method based on voxel compression, after the cloud device performs steps 1 to 3, the compressed voxel model is sent to the edge device to perform step 4.
[0017] This invention also proposes a video rendering system based on voxel compression, which includes:
[0018] The initial module is used to acquire frame-synchronized video for training the voxel model, and uses the specified starting video frame in the frame-synchronized video as the current frame.
[0019] The compression module is used to input the current frame into the neural radiation field, construct the initial voxel model of the current frame, and perform regularization processing on the initial voxel model to remove voxels other than the surface of the target object in the initial voxel model, so as to obtain the compressed voxel model of the current frame.
[0020] The loop module is used to select the next frame of the current frame as the current frame based on the frame order of the synchronized video and call the compression module again until the compressed voxel model of the specified final video frame is obtained.
[0021] The rendering module is used to treat all compressed voxel models as point cloud models or to treat the voxels of compressed voxel models as vertices of a mesh model. Through the point cloud rendering pipeline or the mesh rendering pipeline, each compressed voxel model is rendered as an image from a specified viewpoint, constituting the video rendering result from the starting video frame to the final video frame.
[0022] The present invention also proposes a storage medium for storing a program that executes any of the voxel compression-based video rendering methods.
[0023] The present invention also proposes a client for the aforementioned voxel compression-based video rendering system.
[0024] As can be seen from the above solutions, the advantages of the present invention are:
[0025] 6DoF volumetric videos created based on voxel models that closely resemble the object's surface can be rendered using point cloudification or meshing methods. This approach is compatible with traditional rendering pipelines, suitable for edge rendering, and can be widely adopted. Brief description of the attached figures
[0026] Figure 1 This is a schematic diagram illustrating the rendering and training principle of the Neural Radiation Field (NeRF) of this invention.
[0027] Figure 2A schematic diagram of video shooting using a camera array with frame synchronization function;
[0028] Figure 3 The diagram shows data augmentation aids in training voxel models. The dashed lines in the diagram represent new perspectives generated by other NeRF models (such as the Instant-ngp model).
[0029] Figure 4 To help train high-quality voxel models using LiDAR or depth cameras;
[0030] Figure 5 This is a schematic diagram of effective voxels in adjacent frames. Solid voxels represent voxels that do not change between adjacent frames, while dashed voxels represent voxels that change between adjacent frames. Best way to implement the present invention
[0031] The inventors discovered that this defect can be improved by pruning useless voxels during training. Based on some voxel compression algorithms, effective voxels can be distributed close to the object's surface. In this case, the voxel model can be treated as a 3D point cloud, or each voxel point can be combined with its nearest neighbor voxels to form a triangular face. Then, a shader can be written for rendering, adapting to edge calculations. Therefore, when we can train a voxel model that closely resembles the surface of a real object, obtaining a voxel-based 3D dynamic model, we can then treat it as a point cloud or directly form a mesh to write a shader for edge rendering. The core invention of this application is to train a high-quality voxel model that closely resembles the object's surface, treating the voxel model as a dense point cloud or directly forming a mesh model, to achieve 6DoF (Degree of Freedom) volumetric video edge rendering.
[0032] To make the above features and effects of the present invention clearer and easier to understand, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.
[0033] Step S1, as follows Figure 2 As shown, a camera array with frame synchronization function is used to capture video and obtain frame-synchronized video.
[0034] Step S2: Extract the first frame of the video captured by each camera and train the voxel model. During voxel training, this invention uses regularization to remove voxels with low importance to the model's representation, and may also merge some voxels with low importance. Specifically, the following method can be used to train a voxel model that closely resembles the surface of a real object:
[0035] a) Set a threshold for volume density. During training, if the volume density of a voxel is less than the threshold, the voxel is considered invalid and can be deleted. For example, when shooting... Figure 2The voxels shown in the scene with a volume density less than the threshold may be objects such as air that do not significantly obstruct light in the actual scene. Therefore, deleting them will not affect the final viewing effect. Thus, they can be deleted to reduce the amount of data in the voxel model and train a voxel model that closely resembles the surface of a real object.
[0036] b) During training, the importance of a certain parameter is defined, such as the volume density parameter of the voxel model or the importance of color information parameters. For example, given the parameter w and a given loss function L(θ; w), if If the value is less than a certain threshold, then this parameter is deleted. In the formula, θ in L(θ; w) represents all parameters except w. Represents the partial derivative.
[0037] c) For image inputs with overly sparse viewpoints in the forward-facing scene, voxel models may fail to train well. In such cases, data augmentation methods can be used to assist training. Since NeRF models based on network representations are relatively easy to train and achieve good results, we can use pre-trained models based on implicit representations (NeRF or Instant-ngp) to generate new viewpoints, helping voxel models improve training performance, such as... Figure 3 As shown.
[0038] d) In camera arrays, depth cameras, LiDAR, etc., can be added to provide depth information, effectively providing prior information to voxels. For example, when we obtain depth information at a certain location, we know that there are no valid voxels before that location, but valid voxels exist at the corresponding depth. Figure 4 As shown.
[0039] Step S3: Considering the real dynamic scene, when the time is very close, the change area of the model is very small. Therefore, this invention compresses the expression of the voxel dynamic model. The voxel model of the next frame is finely adjusted from the voxel model of the previous frame, and only the key voxels of the transformation are preserved, achieving a model compression of about 200 times.
[0040] Specifically, the second frame of each camera array is extracted to train the corresponding voxel model for that second frame. Fine-tuning is then performed directly based on the voxel model from the first frame. Since the time interval is very short, the dynamic changes between the two frames should be relatively small, so only a small number of meshes will change, such as... Figure 5 As shown. At this point, the second voxel model will have a small number of voxel movements relative to the first voxel model, with new effective voxels being generated or disappearing. The voxel model in the second frame only needs to store these changes, which can greatly reduce the model size.
[0041] Next, following the steps from the previous step, train each frame sequentially.
[0042] Step S4: Treat the trained voxel model as a point cloud model and write shaders to adapt it to the traditional rendering pipeline. Alternatively, treat voxels as vertices of a mesh model, find neighboring voxels for each voxel to form a triangular mesh, and write corresponding shaders for the triangular mesh to make it compatible with the traditional rendering pipeline. The rendering pipeline is a specification for the rendering process of 3D models. However, the pipeline has a certain degree of freedom, and some steps can be programmed by yourself.
[0043] The model is sent to the edge (such as a mobile phone, VR headset, etc.) for edge rendering.
[0044] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.
[0045] This invention also proposes a video rendering system based on voxel compression, which includes:
[0046] The initial module is used to acquire frame-synchronized video for training the voxel model, and uses the specified starting video frame in the frame-synchronized video as the current frame.
[0047] The compression module is used to input the current frame into the neural radiation field, construct the initial voxel model of the current frame, and perform regularization processing on the initial voxel model to remove voxels other than the surface of the target object in the initial voxel model, so as to obtain the compressed voxel model of the current frame.
[0048] The loop module is used to select the next frame of the current frame as the current frame based on the frame order of the synchronized video and call the compression module again until the compressed voxel model of the specified final video frame is obtained.
[0049] The rendering module is used to treat all compressed voxel models as point cloud models or to treat the voxels of compressed voxel models as vertices of a mesh model. Through the point cloud rendering pipeline or the mesh rendering pipeline, each compressed voxel model is rendered as an image from a specified viewpoint, constituting the video rendering result from the starting video frame to the final video frame.
[0050] The regularization process includes: if the volume density of a voxel in the initial voxel model is less than a preset volume density threshold, it is determined to be an invalid voxel and deleted.
[0051] The training process of the neural radiation field includes: setting the importance of a specified parameter w in the neural radiation field; using L(θ; w) as the loss function when training the neural radiation field; and training the neural radiation field if... If the value is less than a certain threshold, the specified parameter w is deleted; where θ represents all parameters in the neural radiation field other than the specified parameter w.
[0052] The process of generating this frame of synchronized video includes:
[0053] The target object is photographed using a camera array, resulting in an initial video including the video streams of each physical camera after frame synchronization. A new virtual camera video stream is generated for this initial video using a pre-trained model based on implicit representation. This virtual camera video stream is then integrated into the initial video in a frame-synchronized manner to obtain the frame-synchronized video.
[0054] The camera array includes a depth camera, and the regularization process includes: determining invalid voxels in the initial voxel model between the target object and the depth camera based on the depth information acquired by the depth camera, and deleting them.
[0055] This loop module is used to save all compressed voxel models in a way that saves the inter-frame model changes after obtaining the compressed voxel model of the final video frame.
[0056] The initialization module, compression module, and loop module are located on cloud devices, while the rendering module is located on edge devices.
[0057] The present invention also proposes a storage medium for storing a program that executes any of the voxel compression-based video rendering methods.
[0058] The present invention also proposes a client for the aforementioned voxel compression-based video rendering system.
[0059] Industrial application
[0060] This invention proposes a video rendering method based on voxel compression, comprising: inputting the current frame of a frame-synchronized video into a neural radiation field to construct an initial voxel model for the current frame; performing regularization on the initial voxel model to remove voxels other than those on the surface of the target object, thus obtaining a compressed voxel model for the current frame; selecting the next frame as the current frame according to the frame order of the frame-synchronized video, until a compressed voxel model for a specified final video frame is obtained; treating all compressed voxel models as point cloud models or treating the voxels of the compressed voxel models as vertices of a mesh model, and rendering each compressed voxel model as an image from a specified viewpoint through a point cloud rendering pipeline or a mesh rendering pipeline, thus constituting the video rendering result from the initial video frame to the final video frame. This invention, by truncating the voxel model, allows effective voxels to be distributed close to the object surface. At this point, the voxel model can be considered as a 3D point cloud, or each voxel point and its nearest neighbor voxels can form a triangular face. Then, a shader can be written for rendering, and edge-adapted calculations can be performed.
Claims
1. A video rendering method based on voxel compression, characterized in that, include: Step 1: Obtain the frame synchronization video for training the voxel model, and use the specified starting video frame in the frame synchronization video as the current frame; Step 2: Input the current frame into the neural radiation field, construct the initial voxel model of the current frame, and perform regularization processing on the initial voxel model to remove voxels other than the surface of the target object in the initial voxel model, so as to obtain the compressed voxel model of the current frame. Step 3: Based on the frame order of the synchronized video, select the next frame of the current frame as the current frame and execute Step 2 again until the compressed voxel model of the specified final video frame is obtained. Step 4: Treat all compressed voxel models as point cloud models or treat the voxels of compressed voxel models as vertices of mesh models. Render each compressed voxel model as an image from a specified viewpoint using the point cloud rendering pipeline or mesh rendering pipeline, thus forming the video rendering result from the starting video frame to the final video frame.
2. The video rendering method based on voxel compression as described in claim 1, characterized in that, The regularization process includes: if the volume density of a voxel in the initial voxel model is less than a preset volume density threshold, it is determined to be an invalid voxel and deleted.
3. The video rendering method based on voxel compression as described in claim 1, characterized in that, The training process of this neural radiation field includes: setting the importance of a specified parameter w in the neural radiation field; using L(θ; w) as the loss function when training the neural radiation field; and training the neural radiation field if... If the value is less than a certain threshold, the specified parameter w is deleted; where θ represents all parameters in the neural radiation field other than the specified parameter w.
4. The video rendering method based on voxel compression as described in claim 1, characterized in that, The process of generating this frame of synchronized video includes: The target object is photographed using a camera array, resulting in an initial video including the video streams of each physical camera after frame synchronization. A new virtual camera video stream is generated for this initial video using a pre-trained model based on implicit representation. This virtual camera video stream is then integrated into the initial video in a frame-synchronized manner to obtain the frame-synchronized video.
5. The video rendering method based on voxel compression as described in claim 4, characterized in that, The camera array includes a depth camera, and the regularization process includes: determining invalid voxels in the initial voxel model between the target object and the depth camera based on the depth information acquired by the depth camera, and deleting them.
6. The video rendering method based on voxel compression as described in claim 4, characterized in that, Step 3 includes: after obtaining the compressed voxel model of the final video frame, saving all compressed voxel models in a way that saves the inter-frame model changes.
7. The video rendering method based on voxel compression as described in claim 4, characterized in that, After the cloud device performs steps 1 to 3, it sends the compressed voxel model to the edge device to perform step 4.
8. A video rendering system based on voxel compression, characterized in that, include: The initial module is used to acquire frame-synchronized video for training the voxel model, and uses the specified starting video frame in the frame-synchronized video as the current frame. The compression module is used to input the current frame into the neural radiation field, construct the initial voxel model of the current frame, and perform regularization processing on the initial voxel model to remove voxels other than the surface of the target object in the initial voxel model, so as to obtain the compressed voxel model of the current frame. The loop module is used to select the next frame of the current frame as the current frame based on the frame order of the synchronized video and call the compression module again until the compressed voxel model of the specified final video frame is obtained. The rendering module is used to treat all compressed voxel models as point cloud models or to treat the voxels of compressed voxel models as vertices of a mesh model. Through the point cloud rendering pipeline or the mesh rendering pipeline, each compressed voxel model is rendered as an image from a specified viewpoint, constituting the video rendering result from the starting video frame to the final video frame.
9. A storage medium for storing a program that executes any one of the voxel compression-based video rendering methods as described in claims 1 to 7.
10. A client for the voxel compression-based video rendering system of claim 8.
Citation Information
Patent Citations
Indoor scene three-dimensional reconstruction system and method based on neural radiation field
CN114004941A
Free viewpoint video synthesis method and device, electronic equipment and storage medium
CN114627223A