Video rendering method and system based on voxel compression

Abstract

The invention provides a video rendering method based on voxel compression, and the method comprises the steps: inputting a current frame of a frame synchronization video into a neural radiation field, constructing an initial voxel model of the current frame, and carrying out the regularization processing of the initial voxel model, so as to remove voxels outside the surface of a target object in the initial voxel model, obtaining a compressed voxel model of the current frame; selecting a next frame of the current frame as the current frame according to a frame sequence of the frame synchronization video until a compression voxel model of a specified final video frame is obtained; and taking all the compressed voxel models as a point cloud model or taking voxels of the compressed voxel models as vertexes of a grid model, rendering each compressed voxel model as a picture of a specified view angle through a point cloud rendering pipeline or a grid rendering pipeline, and forming a video rendering result from the initial video frame to the final video frame. According to the method, the voxel model is cut, so that effective voxels can be distributed at positions close to the surface of an object, and the edge end can be rendered conveniently.

Description

A video rendering method and system based on voxel compression Technical Field

[0001] The present invention relates to the field of computer graphics and three-dimensional reconstruction technology, and in particular to a video rendering method and system based on voxel compression. Background Art

[0002] Neural Radiance Fields (NeRF) can reconstruct static three-dimensional (3D) voxel models based on image input from multiple perspectives. NeRF uses a neural network to represent 3D models. Given any point in space and any viewing direction, the NeRF model can output the corresponding volume density and RGB color information. Therefore, for a given viewpoint and camera parameters, a ray can be emitted from the camera's optical center toward the pixel, sampling points on the ray and calculating the corresponding volume density and color information. The color and depth information of the corresponding pixel can then be calculated using the volume rendering formula, as shown in Figure 1. Neural Radiance Fields (NeRF), with the addition of a time dimension, can reconstruct dynamic 3D models based on video input from multiple perspectives. However, based on NeRF's implicit representation, rendering images from new perspectives requires a significant amount of computing power. Rendering a 1080p image from a new perspective on a high-performance graphics rendering unit (GPU) can take minutes.

[0003] Voxel-based static 3D models store volume density and color information directly on the grid points in space. During rendering, the volume density and color information for a given sampling point on a ray can be directly queried and obtained through simple calculations, avoiding neural network inference and greatly improving rendering efficiency. Voxel 3D models can also expand the time dimension and be used to express dynamic 3D models. One representative example is the Fourier PlenOctree. However, this direct expansion of the voxel model results in a very large model capacity. The Fourier PlenOctree reconstruction of a dynamic character takes about 2 seconds, and the model capacity reaches several GB. Although the voxel model renders quickly, its rendering method is incompatible with traditional point cloud or mesh rendering pipelines, resulting in problems with edge rendering and limiting the promotion and application of the model.

[0004] Invention Disclosure

[0005] While researching edge rendering with voxel models, the inventors discovered that the aforementioned technical limitations are caused by the voxel model's overly dispersed voxels, which don't conform closely to the surface of real objects. Consequently, rendering graphics based on this voxel model requires programming a computational pipeline on specialized graphics cards, limiting the voxel model's widespread adoption at the edge.

[0006] In view of the shortcomings of the existing technology, the present invention proposes a video rendering method based on voxel compression, which includes:

[0007] Step 1: Obtain a frame-synchronized video for training a voxel model, and use a specified starting video frame in the frame-synchronized video as the current frame;

[0008] Step 2: Input the current frame into the neural radiation field, construct an initial voxel model of the current frame, and perform regularization on the initial voxel model to remove voxels other than the surface of the target object in the initial voxel model to obtain a compressed voxel model of the current frame;

[0009] Step 3: According to the frame sequence of the frame synchronization video, select the next frame of the current frame as the current frame and execute step 2 again until the compressed voxel model of the specified final video frame is obtained;

[0010] Step 4: Treat all compressed voxel models as point cloud models or treat the voxels of the compressed voxel models as vertices of the mesh model. Render each compressed voxel model into an image of a specified perspective through the point cloud rendering pipeline or the mesh rendering pipeline to form the video rendering result from the starting video frame to the final video frame.

[0011] In the video rendering method based on voxel compression, the regularization processing includes: when the volume density of a voxel in the initial voxel model is less than a preset volume density threshold, determining it as an invalid voxel and deleting it.

[0012] The video rendering method based on voxel compression, wherein the training process of the neural radiation field includes: setting the importance of the specified parameter w in the neural radiation field, the loss function used when training the neural radiation field is L(θ; w), and when training the neural radiation field, if If θ is less than a certain threshold, the specified parameter w is deleted; where θ represents all parameters other than the specified parameter w in the neural radiation field.

[0013] In the voxel compression-based video rendering method, the process of generating the frame-synchronized video includes:

[0014] A camera array is used to shoot the target object to obtain an initial video that includes the video stream of each physical camera after frame synchronization. A pre-trained implicit expression-based model is used to generate a new virtual camera video stream for the initial video, and the virtual camera video stream is integrated into the initial video in a frame-synchronized manner to obtain the frame-synchronized video.

[0015] In the voxel compression-based video rendering method, the camera array includes a depth camera, and the regularization processing includes: based on the depth information collected by the depth camera, determining the voxels between the target object and the depth camera in the initial voxel model as invalid voxels and deleting them.

[0016] In the video rendering method based on voxel compression, step 3 includes: after obtaining the compressed voxel model of the final video frame, saving all compressed voxel models in a manner of saving the model changes between frames.

[0017] In the voxel compression-based video rendering method, after the cloud device executes steps 1 to 3, the compressed voxel model is sent to the edge device to execute step 4.

[0018] The present invention also proposes a video rendering system based on voxel compression, which includes:

[0019] An initialization module, configured to obtain a frame-synchronized video for training a voxel model, and use a starting video frame specified in the frame-synchronized video as a current frame;

[0020] a compression module, configured to input the current frame into the neural radiation field, construct an initial voxel model of the current frame, and perform regularization processing on the initial voxel model to eliminate voxels other than the surface of the target object in the initial voxel model, thereby obtaining a compressed voxel model of the current frame;

[0021] A loop module is used to select the next frame of the current frame as the current frame according to the frame sequence of the frame synchronization video and call the compression module again until a compressed voxel model of the specified final video frame is obtained;

[0022] The rendering module is used to treat all compressed voxel models as point cloud models or the voxels of the compressed voxel models as vertices of the mesh model, and render each compressed voxel model into an image of a specified perspective through the point cloud rendering pipeline or the mesh rendering pipeline to form the video rendering result from the starting video frame to the final video frame.

[0023] The present invention also proposes a storage medium for storing a program for executing any one of the above-mentioned voxel compression-based video rendering methods.

[0024] The present invention also proposes a client for use in the voxel compression-based video rendering system.

[0025] It can be seen from the above scheme that the advantages of the present invention are:

[0026] The 6DoF volumetric video produced based on the voxel model close to the surface of the object is rendered using a point cloud or meshing method. It is compatible with the traditional rendering pipeline, suitable for edge rendering, and can be widely promoted.

[0027] BRIEF DESCRIPTION OF THE DRAWINGS

[0028] FIG1 is a schematic diagram of the neural radiation field (NeRF) rendering and training principle of the present invention;

[0029] FIG2 is a schematic diagram of a video captured by a camera array with a frame synchronization function;

[0030] Figure 3 shows data augmentation to help voxel model training. The dotted lines in the figure represent new perspectives generated by other NeRF models (such as the Instant-ngp model).

[0031] Figure 4 shows the use of lidar or depth camera to help train high-quality voxel models;

[0032] FIG5 is a schematic diagram of valid voxels in adjacent frames, where solid-line voxels represent voxels that have not changed between adjacent frames, and dashed-line voxels represent voxels that have changed between adjacent frames.

[0033] Best Mode for Carrying Out the Invention

[0034] After research, the inventors found that this defect can be improved by cutting out useless voxels during the training process. Based on some algorithms for compressing voxels, effective voxels can be distributed close to the surface of the object. At this time, the voxel model can be regarded as a 3D point cloud or each voxel point and its neighboring voxels can be combined into a triangular face. Then, a shader can be written for rendering, and the edge end can be adapted for calculation. Therefore, when we can train a voxel model close to the surface of a real object and obtain a voxel-based 3D dynamic model, we can then regard it as a point cloud or directly form a mesh to write a shader for edge rendering. The core invention of this application is to train a high-quality voxel model close to the surface of an object, regard the voxel model as a dense point cloud or directly form a mesh model to achieve edge rendering of 6DoF (Degree of Freedom) volumetric video at the edge.

[0035] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0036] Step S1, as shown in FIG2, uses a camera array with a frame synchronization function to shoot a video to obtain a frame-synchronized video.

[0037] Step S2: Extract the first frame of the video captured by each camera and train a voxel model. When training voxels, the present invention uses a regularization method to remove voxels that are of little importance to the model expression, and can also merge some voxels of little importance. Specifically, the following method can be used to train a voxel model that is close to the surface of the real object:

[0038] a) Set a volume density threshold. During training, if a voxel's volume density is less than the threshold, the voxel is considered invalid and can be deleted. For example, when capturing the scene shown in Figure 2, voxels with a volume density less than the threshold may actually be objects such as air that do not significantly block light. Therefore, deleting them will not affect the final viewing effect. This reduces the amount of data in the voxel model and allows for a voxel model that is closer to the surface of the real object.

[0039] b) During training, set the importance of a parameter, such as the volume density parameter of the voxel model, or the importance of the color information parameter. For example, for the parameter w and the given loss function L(θ; w), if If the value is less than a certain threshold, the parameter is deleted. In the formula, θ in L(θ;w) represents all parameters except w. Represents partial derivative.

[0040] c) For forward-facing scenes, photo inputs with sparse viewpoints can hinder voxel model training. Data augmentation can be used to aid training. Since the network-based NeRF model is relatively easy to train and achieve good results, we can use a pre-trained implicit representation model (NeRF or Instant-ngp) to generate new viewpoints, improving voxel model training performance, as shown in Figure 3.

[0041] d) In the camera array, depth cameras, lidars, and other sensors can be added to provide depth information, effectively providing prior information for voxels. For example, when we obtain depth information at a certain location, we know that there were no valid voxels before this point, but there are valid voxels at the corresponding depth, as shown in Figure 4.

[0042] Step S3: Considering real dynamic scenes, where the model changes very little when the time is close, the present invention compresses the representation of the voxel dynamic model. The voxel model of the subsequent frame is fine-tuned from the voxel model of the previous frame, preserving only the key voxels that have changed, achieving approximately 200 times model compression.

[0043] Specifically, the second frame of each camera's entire column is extracted to train the corresponding voxel model. Fine-tuning is then performed directly based on the voxel model of the first frame. Due to the short time interval, the dynamic changes between the two frames should be relatively small, so only a small number of grids will change, as shown in Figure 5. At this point, the second voxel model will have a small amount of voxel movement relative to the first, with new valid voxels generated or eliminated. The voxel model of the second frame only needs to store these changes, significantly reducing the model capacity.

[0044] Next, follow the previous step and train each frame in turn.

[0045] Step S4: Treat the trained voxel model as a point cloud model and write a shader to adapt it to the traditional rendering pipeline. Alternatively, treat the voxels as vertices in a mesh model and find neighboring voxels for each voxel to form a triangle mesh. This allows you to write a shader for the triangle mesh and make it compatible with the traditional rendering pipeline. The rendering pipeline specifies the rendering process for 3D models. However, there is a certain degree of freedom within the pipeline, and some steps can be customized.

[0046] The model is transmitted to the edge (such as mobile phones, VR headsets, etc.) for edge rendering.

[0047] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0048] The present invention also proposes a video rendering system based on voxel compression, which includes:

[0049] An initialization module, configured to obtain a frame-synchronized video for training a voxel model, and use a starting video frame specified in the frame-synchronized video as a current frame;

[0050] a compression module, configured to input the current frame into the neural radiation field, construct an initial voxel model of the current frame, and perform regularization processing on the initial voxel model to eliminate voxels other than the surface of the target object in the initial voxel model, thereby obtaining a compressed voxel model of the current frame;

[0051] A loop module is used to select the next frame of the current frame as the current frame according to the frame sequence of the frame synchronization video and call the compression module again until a compressed voxel model of the specified final video frame is obtained;

[0052] The rendering module is used to treat all compressed voxel models as point cloud models or the voxels of the compressed voxel models as vertices of the mesh model, and render each compressed voxel model into an image of a specified perspective through the point cloud rendering pipeline or the mesh rendering pipeline to form the video rendering result from the starting video frame to the final video frame.

[0053] The regularization process includes: when the volume density of a voxel in the initial voxel model is less than a preset volume density threshold, determining the voxel as an invalid voxel and deleting the voxel.

[0054] The training process of the neural radiation field includes: setting the importance of the specified parameter w in the neural radiation field, using the loss function L(θ; w) when training the neural radiation field, and If θ is less than a certain threshold, the specified parameter w is deleted; where θ represents all parameters other than the specified parameter w in the neural radiation field.

[0055] The process of generating the frame-synchronized video includes:

[0056] A camera array is used to shoot the target object to obtain an initial video that includes the video stream of each physical camera after frame synchronization. A pre-trained implicit expression-based model is used to generate a new virtual camera video stream for the initial video, and the virtual camera video stream is integrated into the initial video in a frame-synchronized manner to obtain the frame-synchronized video.

[0057] The camera array includes a depth camera, and the regularization process includes: determining and deleting invalid voxels between the target object and the depth camera in the initial voxel model based on depth information collected by the depth camera.

[0058] The loop module is used to save all compressed voxel models in a manner of saving model variations between frames after obtaining the compressed voxel model of the final video frame.

[0059] The initial module, the compression module and the loop module are located in the cloud device, and the rendering module is located in the edge device.

[0060] The present invention also proposes a storage medium for storing a program for executing any one of the above-mentioned voxel compression-based video rendering methods.

[0061] The present invention also proposes a client for use in the voxel compression-based video rendering system.

[0062] Industrial Applicability

[0063] The present invention proposes a video rendering method based on voxel compression, comprising: inputting the current frame of a frame-synchronized video into a neural radiation field, constructing an initial voxel model of the current frame, and performing regularization processing on the initial voxel model to eliminate voxels outside the target object surface in the initial voxel model to obtain a compressed voxel model of the current frame; selecting the next frame of the current frame as the current frame according to the frame sequence of the frame-synchronized video, until a compressed voxel model of a specified final video frame is obtained; treating all compressed voxel models as point cloud models or treating voxels of the compressed voxel models as vertices of a mesh model, rendering each compressed voxel model into an image of a specified perspective through a point cloud rendering pipeline or a mesh rendering pipeline, thereby forming a video rendering result from the starting video frame to the final video frame. The present invention crops the voxel model so that valid voxels can be distributed near the surface of the object. At this time, the voxel model can be regarded as a 3D point cloud or each voxel point and its neighboring voxels form a triangular face. Then, a shader can be written for rendering, and the edge end can be adapted for calculation.

Claims

1. A video rendering method based on voxel compression, characterized in that: include: Step 1: Obtain a frame-synchronized video for training a voxel model, and use a specified starting video frame in the frame-synchronized video as the current frame; Step 2: Input the current frame into the neural radiation field, construct an initial voxel model of the current frame, and perform regularization on the initial voxel model to remove voxels other than the surface of the target object in the initial voxel model to obtain a compressed voxel model of the current frame; Step 3: According to the frame sequence of the frame synchronization video, select the next frame of the current frame as the current frame and execute step 2 again until the compressed voxel model of the specified final video frame is obtained; Step 4: Treat all compressed voxel models as point cloud models or treat the voxels of the compressed voxel models as vertices of the mesh model. Render each compressed voxel model into an image of a specified perspective through the point cloud rendering pipeline or the mesh rendering pipeline to form the video rendering result from the starting video frame to the final video frame.

2. The video rendering method based on voxel compression according to claim 1, characterized in that: The regularization process includes: when the volume density of a voxel in the initial voxel model is less than a preset volume density threshold, determining the voxel as an invalid voxel and deleting the voxel.

3. The video rendering method based on voxel compression according to claim 1, wherein: The training process of the neural radiation field includes: setting the importance of the specified parameter w in the neural radiation field, the loss function used in training the neural radiation field is L(θ; w), and when training the neural radiation field, if If θ is less than a certain threshold, the specified parameter w is deleted; where θ represents all parameters other than the specified parameter w in the neural radiation field.

4. The video rendering method based on voxel compression according to claim 1, wherein: The process of generating this frame-synchronized video includes: A camera array is used to shoot the target object to obtain an initial video that includes the video stream of each physical camera after frame synchronization. A pre-trained implicit expression-based model is used to generate a new virtual camera video stream for the initial video, and the virtual camera video stream is integrated into the initial video in a frame-synchronized manner to obtain the frame-synchronized video.

5. The video rendering method based on voxel compression according to claim 4, characterized in that: The camera array includes a depth camera, and the regularization process includes: determining and deleting invalid voxels between the target object and the depth camera in the initial voxel model based on depth information collected by the depth camera.

6. The video rendering method based on voxel compression according to claim 4, characterized in that: The step 3 includes: after obtaining the compressed voxel model of the final video frame, saving all compressed voxel models in a manner of saving the model variation between frames.

7. The video rendering method based on voxel compression according to claim 4, characterized in that: After the cloud device executes steps 1 to 3, it sends the compressed voxel model to the edge device to execute step 4.

8. A video rendering system based on voxel compression, characterized in that: include: An initialization module, configured to obtain a frame-synchronized video for training a voxel model, and use a starting video frame specified in the frame-synchronized video as a current frame; a compression module, configured to input the current frame into the neural radiation field, construct an initial voxel model of the current frame, and perform regularization processing on the initial voxel model to eliminate voxels other than the surface of the target object in the initial voxel model, thereby obtaining a compressed voxel model of the current frame; A loop module is used to select the next frame of the current frame as the current frame according to the frame sequence of the frame synchronization video and call the compression module again until a compressed voxel model of the specified final video frame is obtained; The rendering module is used to treat all compressed voxel models as point cloud models or the voxels of the compressed voxel models as vertices of the mesh model, and render each compressed voxel model into an image of a specified perspective through the point cloud rendering pipeline or the mesh rendering pipeline to form the video rendering result from the starting video frame to the final video frame.

9. A storage medium for storing a program for executing any one of the voxel compression-based video rendering methods according to claims 1 to 7.

10. A client, used in the voxel compression-based video rendering system according to claim 8.

Images