A real-time rendering method and device based on a dynamic perceptron graph
By predicting frame-by-frame perceptron graphs based on a two-dimensional deep convolutional network and using three orthogonal perceptron graphs to represent the 3D scene, the problem of slow dynamic scene rendering speed and high storage cost is solved, and efficient dynamic scene rendering with low storage cost is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2022-12-27
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies are slow and costly to store in dynamic scene rendering, making it difficult to achieve efficient rendering.
A perceptron graph based on a two-dimensional deep convolutional network is used to predict frame by frame. The three-dimensional scene is represented by three orthogonal perceptron graphs to achieve real-time rendering of dynamic scenes.
It achieves efficient rendering of dynamic scenes with low storage costs and improves the modeling capabilities of 3D scenes.
Smart Images
Figure CN116188682B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of rendering, and in particular relates to a real-time rendering method and apparatus based on dynamic perceptron graphs. Background Technology
[0002] This invention proposes a dynamic perceptron graph representation of dynamic 3D scenes, enabling real-time rendering of dynamic scenes. In related technologies, most methods focus on rendering static scenes. Some neural network-based rendering methods achieve dynamic scene rendering by introducing time-dimensional variables, but these methods are slow. Although these methods can be accelerated through pre-computation techniques, this also leads to increased storage costs. This invention aims to represent 3D scenes using a small set of neural networks, achieving efficient rendering of dynamic scenes with low storage costs. Summary of the Invention
[0003] The purpose of this invention is to address the shortcomings of existing technologies by proposing a real-time rendering method and apparatus based on dynamic perceptron graphs. The method uses a two-dimensional deep convolutional network to predict frame-by-frame perceptron graphs to represent the geometry and appearance of a three-dimensional scene frame by frame, thereby achieving real-time rendering of volumetric videos.
[0004] The objective of this invention is achieved through the following technical solution: Firstly, this invention provides a real-time rendering method based on a dynamic perceptron graph, the method comprising the following steps:
[0005] (1) For dynamic scenes of the target, use a synchronous multi-view camera to acquire multi-view video;
[0006] (2) For video frames of multi-view videos, predict the dynamic perceptron graph based on a two-dimensional deep convolutional network. The dynamic perceptron graph is a two-dimensional image on each video frame, and each image pixel records the parameters of the perceptron network;
[0007] (3) Real-time rendering of dynamic scenes is achieved by using perceptron graph representation to represent each frame of the three-dimensional scene in the neural volume video.
[0008] Further, in step (2), the prediction of the dynamic perceptron graph based on the two-dimensional deep convolutional network is specifically as follows: In order to obtain the volumetric video from the multi-view video, for any video frame, a set of images is selected from the input viewpoint, a two-dimensional convolutional encoding network is used to obtain the feature vector, and then the feature vector is input into the two-dimensional convolutional decoding network to output the perceptron graph of the video frame.
[0009] Further, in step (3), the three-dimensional scene in each frame of the neural volume video based on the perceptron graph representation is specifically: for any three-dimensional point in the three-dimensional scene, the corresponding network parameters are selected from the perceptron graph to predict the voxel density and color of the three-dimensional point.
[0010] Furthermore, the 3D scene is represented by three mutually orthogonal perceptron graphs. Specifically, orthogonality means that the two-dimensional planes containing the three perceptron graphs are perpendicular to each other. For any 3D point, it is projected onto the three orthogonal perceptron graph planes respectively, and the voxel density and color are regressed respectively. Finally, they are added together to obtain the voxel density and color of the 3D point.
[0011] Secondly, the present invention provides a real-time rendering apparatus based on a dynamic perceptron graph, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the aforementioned real-time rendering method based on a dynamic perceptron graph.
[0012] Thirdly, the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned real-time rendering method based on a dynamic perceptron graph.
[0013] The beneficial effects of this invention are as follows: This invention proposes a dynamic perceptron graph, thereby realizing real-time rendering of dynamic scenes. To obtain the perceptron graph for each frame, this invention predicts the perceptron graph based on a two-dimensional deep convolutional network. This invention also proposes mutually orthogonal perceptron graphs, which improve the modeling capability of three-dimensional scenes by representing the three-dimensional scene as the sum of signals in three orthogonal directions. Attached Figure Description
[0014] Figure 1 This is a schematic diagram of the present invention generating a single perceptron image from a multi-view two-dimensional image.
[0015] Figure 2 This invention proposes a perceptron graph based on mutual orthogonality, used to regress the density and color diagram of continuous voxels.
[0016] Figure 3 This is a structural diagram of a real-time rendering device based on a dynamic perceptron graph proposed in this invention. Detailed Implementation
[0017] The technical details and principles of the present invention will be further described below with reference to the accompanying drawings:
[0018] This invention proposes a real-time rendering method based on dynamic perceptron graphs. The invention optimizes and reconstructs a neural volumetric video from an input multi-view video. The reconstructed neural volumetric video can be used for real-time rendering of dynamic scenes.
[0019] like Figure 1 As shown, the real-time rendering method based on dynamic perceptron graphs proposed in this invention uses a two-dimensional deep convolutional network, i.e., an autoencoder, to predict the dynamic perceptron graph. The specific steps are as follows:
[0020] 1. For example Figure 1 As shown in (a), for a given video frame, this invention uses a two-dimensional deep convolutional network as an encoder to regress the input fixed-viewpoint image into a feature vector. Then, the feature vector z is mapped into a high-resolution feature map through a backbone network. This invention uses two-dimensional deep convolutional networks as decoders for color and voxel density respectively, decoding the feature map into corresponding dynamic perceptron maps.
[0021] 2. For example Figure 1 As shown in (b), a perceptron graph is a two-dimensional feature grid map. Each pixel in the grid map stores the network parameters of a multilayer perceptron in the form of a parameter vector. When the parameter vector needs to be queried, it is reconstructed into a multilayer perceptron to regress the corresponding scene information. This invention uses a single-layer voxel density perceptron and a three-layer color perceptron to model the 3D scene. Because the number of parameters in a voxel density multilayer perceptron is much smaller than that in a color multilayer perceptron, this invention predicts a higher resolution voxel density perceptron graph.
[0022] like Figure 2 As shown, in the real-time rendering method based on dynamic perceptron graph proposed in this invention, the three-dimensional scene of each frame in the volumetric video is represented by the perceptron graph. The specific steps are as follows:
[0023] 1. This invention generates continuous voxel density and color based on a dynamic perceptron graph, serving as a 3D representation of volumetric video. For a point p in 3D space and a specific frame t in a multi-view video, this invention first aligns the 3D point with the perceptron graph using the scene bounding box at time t, and then orthogonally projects it onto the 2D perceptron graph to obtain the corresponding multilayer perceptron parameters M. This invention represents the voxel density and color functions as functions of the 3D point p, the specific time frame t, and the view direction d. To enable the network to learn high-frequency functions better, this invention also performs positional encoding on the view direction d and multi-resolution hash encoding on the 3D point p and the specific time frame t. The voxel density and color functions are defined as:
[0024] (σ,c)=M(γ p (p),γ d (d))
[0025] Where M is Figure 1 The corresponding multilayer perceptron network in (b) is γ. p (p) and γ d (d) are the multi-resolution HAZ coding function applied to a 3D point p and a specific time frame t, and the position coding function applied to the view direction d, respectively.
[0026] 2. This invention regresses complex signals in space by predicting three orthogonal perceptron maps, where orthogonality means that the two-dimensional planes containing the perceptron maps are perpendicular to each other. For example... Figure 2 As shown, for any 3D point at any time frame, this invention projects it onto three orthogonal perceptron graph planes to obtain voxel density and color {(σ i ,c i )|i=1,2,3}, where σ i ,c i This represents the voxel density and color obtained from the i-th perceptron image, which are then summed to obtain the voxel density and color of the 3D point at a specific time frame.
[0027] The real-time rendering method based on dynamic perceptron graphs proposed in this invention includes the following specific steps for achieving real-time rendering:
[0028] 1. For example Figure 1 As shown in (a), after the autoencoder training is completed, the present invention calculates the feature vector z at each specific time frame t. t During rendering, the feature vectors z of all specific time frames are used. t Replace autoencoders to reduce network forwarding time.
[0029] 2. This invention extracts a low-resolution occupancy grid from the trained model for each specific time frame. Each voxel in the occupancy grid is marked as occupied when its voxel density exceeds a certain threshold, and the remaining voxels are marked as empty regions. During rendering, this invention only performs network forwarding on the 3D points of the occupied regions within the occupancy grid.
[0030] Corresponding to the aforementioned embodiments of the real-time rendering method based on dynamic perceptron graphs, the present invention also provides embodiments of a real-time rendering apparatus based on dynamic perceptron graphs.
[0031] See Figure 3 The present invention provides a real-time rendering device based on a dynamic perceptron graph, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement the real-time rendering method based on a dynamic perceptron graph in the above embodiments.
[0032] The embodiments of the real-time rendering device based on dynamic perceptron graphs of this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware structure diagram of any device with data processing capabilities, including the real-time rendering device based on a dynamic perceptron graph according to the present invention. (Except for...) Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.
[0033] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0034] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0035] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the real-time rendering method based on a dynamic perceptron graph as described in the above embodiments.
[0036] The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.
[0037] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.
Claims
1. A real-time rendering method based on dynamic perceptron graphs, characterized in that, The method includes the following steps: (1) For dynamic scenes of the target, use a synchronized multi-view camera to acquire multi-view videos; (2) For video frames of multi-view videos, predict dynamic perceptron graphs based on two-dimensional deep convolutional networks; the perceptron graph is a two-dimensional feature grid map, corresponding to the two-dimensional image on each video frame. Each image pixel in the grid map stores the network parameters of a multilayer perceptron in the form of a parameter vector. When the parameter vector needs to be queried, it will be reshaped into a multilayer perceptron to regress the corresponding scene information. (3) Real-time rendering of dynamic scenes is achieved by representing each frame of the three-dimensional scene in the neural volume video based on the perceptron graph representation. Specifically, the three-dimensional scene is represented by three mutually orthogonal perceptron graphs: orthogonality means that the two-dimensional planes in which the three perceptron graphs are located are perpendicular to each other; for any three-dimensional point, it is projected onto the three mutually orthogonal perceptron graph planes respectively, and the voxel density and color are regressed respectively, and finally added together to obtain the voxel density and color of the three-dimensional point; the corresponding network parameters are selected through the perceptron graph to predict the voxel density and color of the three-dimensional point.
2. The real-time rendering method based on a dynamic perceptron graph according to claim 1, characterized in that, In step (2), the prediction of the dynamic perceptron graph based on the two-dimensional deep convolutional network is specifically as follows: In order to obtain the volumetric video from the multi-view video, for any video frame, a set of images is selected from the input viewpoint, a two-dimensional convolutional encoding network is used to obtain the feature vector, and then the feature vector is input into the two-dimensional convolutional decoding network to output the perceptron graph of the video frame.
3. A real-time rendering apparatus based on a dynamic perceptron graph, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a real-time rendering method based on a dynamic perceptron graph as described in any one of claims 1-2.
4. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a real-time rendering method based on a dynamic perceptron graph as described in any one of claims 1-2.