Volume video figure rendering method, system and device based on double-layer Gaussian splashing, chip and medium

Abstract

The invention discloses a volume video figure rendering method, system and equipment based on double-layer Gaussian splashing, a chip and a medium. The method comprises the following steps: S1, acquiring human body pictures with actions from a plurality of visual angles; s2, capturing global motion by using joint Gauss, capturing appearance details by using skin Gauss, and establishing mapping between the skin Gauss and the joint Gauss; s3, performing double-Gaussian optimization by using a coarse-to-fine optimization strategy, and ensuring time consistency; and S4, respectively providing a residual quantization compression method, a picture coding and decoding compression method and a persistent codebook compression method for different Gaussian attributes. According to the invention, through multi-view RGB video input, a double-layer Gaussian splashing and coarse-to-fine optimization strategy is used, so that the rendering quality with extremely high competitiveness is achieved; different Gaussian point cloud attributes are compressed through residual quantization compression, picture coding and decoding compression and a persistent codebook, and the method is superior to the prior art in the aspects of rendering quality, efficiency and storage efficiency.

Description

Technical Field

[0001] The invention relates to computer graphics processing technology, in particular to a volumetric video character rendering method, system, equipment, chip and medium based on double-layer Gaussian splashing. Background Art

[0002] As the boundaries between the digital world and the real world blur, 3D and 4D content is rapidly emerging, reshaping society's expectations and applications of the digital landscape. Among these innovations, volumetric video, as an innovative video format, represents a revolutionary advancement in visual media: it allows viewers to watch from any perspective, enabling viewers to have an immersive virtual experience. Specifically, users can immerse themselves in a virtual musical journey, watch musicians perform up close, and feel the rhythm of the music as if they were standing next to them.

[0003] Over the past two decades, studios have built multi-view domes around the world, from the West to the East, to capture volumetric video. However, the main workflow for producing human-centric volumetric video still relies on explicit reconstruction and tracking of textured meshes. This approach is susceptible to occlusion, often resulting in holes and noise that degrade texture quality. Even producing the most basic immersive footage requires significant computing resources and meticulous cleanup by skilled artists. In addition, volumetric video assets are often too large to store and integrate into immersive devices. As a result, volumetric video has not yet been widely adopted.

[0004] In the field of neural rendering, most methods select a canonical frame and use multi-layer perceptrons (MLPs) to learn a deformation field, deform subsequent frames to the canonical frame, and produce temporally consistent volumetric videos. However, this method often has difficulty handling fast motion changes and topological changes, and may produce some undesirable results; these adverse effects mainly include the loss of motion details, unnatural morphological deformation, and rendering errors or incoherence when the topological structure changes. Recently, there are also methods that combine explicit point cloud rendering technology with linear hybrid skinning / multi-layer perceptrons to model volumetric videos, but this sacrifices the GPU-friendly nature of point cloud rendering and reduces rendering speed. Similarly, these methods have difficulty handling complex motions and require a lot of storage space. Summary of the invention

[0005] The present invention aims to solve the existing problems and provide a volumetric video character rendering method, system, device, chip and medium based on double-layer Gaussian splashing, which realizes robust human performance tracking and high-fidelity rendering. In order to achieve the above-mentioned purpose, the volumetric video character rendering method based on double-layer Gaussian splashing provided by the present invention includes:

[0006] S1, collects human body pictures with actions from multiple perspectives;

[0007] S2, uses a small number of joint Gaussians for motion perception to capture global motion, a large number of skin Gaussians for appearance perception to capture appearance details, and establishes a mapping between skin Gaussians and joint Gaussians;

[0008] S3, uses a coarse-to-fine optimization strategy for dual Gaussian optimization and ensures temporal consistency;

[0009] S4, applying residual vector quantization to the motion properties of the joint Gaussians and then losslessly compressing them; and / or embedding the opacity and scaling of the skin Gaussians into an independent lookup table and then applying an image codec for compression; and / or compressing the spherical harmonic properties of the skin Gaussians into persistent spherical harmonic indices through a persistent codebook and combining them with length encoding.

[0010] Furthermore, in S1, a dome multi-camera system is used to capture human body images, and the captured object performs actions in the center of the dome.

[0011] Furthermore, in S2, the properties of Gaussian are divided into two groups: motion perception parameters include position p and rotation q; appearance parameters include spherical harmonic coefficients C, opacity σ and scale s.

[0012] Furthermore, in S2, a small number of joint Gaussians are initialized to represent the dynamics of the global motion; the initialized joint Gaussian kernel is used as input for training to initialize the skin Gaussian; and each skin Gaussian is anchored to several nearest neighbor joint Gaussians.

[0013] Furthermore, in order to constrain the scaling of the joint Gaussian to be too thin and long, an isotropic loss is used:

[0014]

[0015] s i represents the scaling parameter of the Gaussian of the i-th joint, e is the activation function; and enforces the constraint that the length ratio of the major axis and the minor axis does not exceed r.

[0016] Furthermore, in order to prevent local over-reconstruction, another term is proposed to constrain the Gaussian that is too large:

[0017]

[0018] sg stands for stop gradient operator, E size Penalizes Gaussians whose size exceeds the mean by α times.

[0019] Furthermore, for each skin Gaussian position Find the k nearest neighbor KNN joint Gaussian within the containing ellipsoid As its anchor joint, and define the blend weights:

[0020]

[0021] Where l is the influence radius.

[0022] Furthermore, the loss function of the joint Gaussian is:

[0023] Enit=λisoEiso+λsizeEsize+Eeolor

[0024] Among them, Ecolo r is the luminosity loss, λ iso , size as a parameter.

[0025] Furthermore, in S3, the coarse-to-fine optimization strategy starts from a separate coarse alignment and proceeds to an integrated fine-grained optimization.

[0026] Furthermore, in the rough alignment, a smoothing regularizer is used to rigidly constrain the motion of the joint Gaussians in the local range:

[0027]

[0028]

[0029] where N(i) is a set of neighbors of the joint Gaussian, wi,k represents the influence weights, and R(·) converts the quaternion back to a rotation matrix.

[0030] Furthermore, the photometric loss is added in the rough alignment process, and the energy formula is:

[0031]

[0032] is the mixed color after joint Gaussian rasterization.

[0033] Furthermore, in S3, a velocity attribute is maintained for each Gaussian and weighted updates are performed using the position change between the last two frames; before rough alignment, the Gaussian position of the new frame is estimated based on the last frame and the velocity, and then non-rigid constraints are applied to limit unreasonable motion.

[0034] Furthermore, in fine-grained optimization, the joint Gaussian motion from the coarse alignment stage is used to interpolate the position and rotation of the skin Gaussian to balance rendering quality and temporal consistency:

[0035]

[0036]

[0037] in and Represent the KNN graph and mixed weights pre-calculated in the initialization phase; during the back propagation process, the gradient of the skin Gaussian Further propagation along the computation graph to the joint Gaussian and

[0038] Furthermore, the overall energy formula is as follows:

[0039]

[0040] E temp is a time term, λ temp is a hyperparameter.

[0041] Furthermore, in S4, residual vector quantization is applied to the joint Gaussian motion, and the position R of the first frame is retained in the current segment. i,1 =p i,1 , and then perform time quantization:

[0042]

[0043] Where Q and Q -1 They represent quantization and dequantization respectively.

[0044] Furthermore, in S4, the opacity and scaling attributes are arranged into a two-dimensional LUT, whose height corresponds to the number of skin Gaussians and whose width corresponds to the length of the segment frame; the LUT is sorted by the average value of each row; and the two-dimensional LUT is quantized and compressed using an image codec.

[0045] Furthermore, in S4, K-Means clustering is applied to the d-order (d=0, 1, 2, 3) spherical harmonic coefficients of all frames in the segment; the code book Z d Initialized to uniform distribution, and iteratively updated by randomly selecting a batch of d-order coefficients; after optimization, four codebooks of length L are obtained;

[0046] The spherical harmonic properties of the skin Gaussian are encoded into a spherical harmonic index via a codebook:

[0047]

[0048] is the SH index of order d of skin Gaussian i in frame t.

[0049] The present invention provides a high-fidelity dynamic character rendering system, comprising:

[0050] A collection module, used to collect multi-view human body images;

[0051] The tracking module uses relatively sparse motion-aware joint Gaussians to track the motion trajectory of the character;

[0052] The optimization module uses the motion-aware joint Gaussian obtained by the tracking module as the action prior, and further optimizes the appearance-aware skin Gaussian to generate a temporally consistent high-fidelity volumetric video;

[0053] A compression module is used to compress the Gaussian representation of high-fidelity volumetric video through residual quantization compression, and / or picture codec compression, and / or persistent code.

[0054] The present invention provides a high-fidelity dynamic character rendering device, comprising an acquisition device, a memory and a processor. The acquisition device is used to acquire multi-view human body images. The memory includes a computer program that can be run thereon. When the processor executes the computer program, the above-mentioned rendering method is implemented.

[0055] The present invention also provides a chip, comprising one or more processors, for calling and running a computer program from a memory, so that a device equipped with the chip executes the above-mentioned rendering method.

[0056] The present invention also provides a storage medium containing computer executable instructions, and the computer executable instructions are used to execute the above rendering method when executed by a computer processor.

[0057] Compared with the prior art, the present invention uses multi-view RGB video input, double-layer Gaussian splashing and coarse-to-fine optimization strategy to achieve highly competitive rendering quality, and combines a motion prediction module to ensure temporal consistency.

[0058] At the same time, the present invention compresses different Gaussian point cloud attributes through residual quantization compression, image codec compression and persistent code, achieving a compression rate of 120 times compared with the existing method, realizing high-fidelity rendering results, and comprehensively outperforming the existing technology in terms of quality, efficiency and storage efficiency.

[0059] The present invention overcomes the problem of lack of consistency in timing of human scenes with fast-moving changes, which is difficult to solve in existing volumetric video generation methods, while achieving fast optimization speed, high-fidelity rendering quality and small storage space; the results can also be seamlessly integrated into a GPU-based rasterization pipeline, and can support an immersive experience of human performance with low storage on various platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flowchart of the method of the first embodiment of the present invention;

[0061] Figure 2 It is a structural schematic diagram of the system of the second embodiment of the present invention;

[0062] Figure 3 This is a rendering of an embodiment of the present invention;

[0063] Figure 4 This is a real machine demonstration diagram of the present invention in a VR helmet. DETAILED DESCRIPTION

[0064] The present invention will now be further described with reference to the accompanying drawings. Figures 1 to 4 The present invention provides a volumetric video character rendering method (DualGS) based on dual-layer Gaussian splatting (3DGS), which is a fully explicit and dual-layer Gaussian representation method for high-fidelity 4D human-machine performance rendering from dense footage; see Figure 1 , mainly includes the following steps:

[0065] S1, collects human body images with actions from multiple perspectives.

[0066] See also Figure 2 In this embodiment, a dome multi-camera system with a diameter of 8 meters and a height of 5 meters is used; 81 cameras (Z-CAM) are distributed in the dome and all cameras point to the center of the dome. The resolution of the cameras is above 2000×2000, and human body data with rich texture is used. The multi-camera system is calibrated using traditional algorithms.

[0067] During the data collection process, the subject makes free movements in the center of the dome, capturing complex human performances at a resolution of 3840×2160 at 30 frames per second, and subsequent algorithm operations are performed after the frames are cut. The aforementioned human body data covers a variety of clothing, from traditional Chinese clothing to casual wear and role-playing; it also includes a wide range of activities such as entertainment, fitness, and various interactive activities.

[0068] S2 uses a small number of joint gaussians for motion perception to capture global motion and a large number of skin gaussians for appearance perception to capture appearance details.

[0069] Initialize DualGS to establish a mapping between skin Gaussian and joint Gaussian. To simplify the description, the properties of Gaussian are first divided into two groups: motion-aware parameters include position p and rotation q, and skin-aware parameters include spherical harmonic coefficients C, opacity σ, and scale s.

[0070] Inspired by a parametric model of the human body, where skin vertices are represented by interpolating a small number of joints, the method of this embodiment uses a dual Gaussian scheme to encode motion and appearance separately. Preferably, a compact set of Gaussians (about 15,000) is used in S2 to capture the overall motion of the performer, while a larger set of Gaussians (about 180,000) is used to capture subtle appearance details. Once established, the number of these Gaussians will remain constant over time, and only the attributes will be continuously updated. This has two advantages:

[0071] 1. The relative positions of the skin Gaussians in local space remain stable, driven by the same joint Gaussian motion, thus enforcing spatiotemporal consistency.

[0072] 2. The reduced motion parameters are very beneficial to the subsequent compression process.

[0073] Similar to the original 3DGS, we start by initializing a small number of joint Gaussians to represent the global motion dynamics. Specifically, the Gaussian model is trained with the first frame initialized randomly. The number of Gaussians is controlled during training to achieve the best balance between efficient motion representation and compact storage.

[0074] Furthermore, during the optimization process, it is possible to generate a relatively thin and long Gaussian kernel to effectively adapt to the local appearance details, but the lack of geometric information leads to unexpected fluffy artifacts in subsequent optimization. Therefore, an isotropic loss is proposed here to constrain the overly thin and long scaling:

[0075]

[0076] Among them, s i represents the scaling parameter of the Gaussian of the i-th joint, e is the activation function; and enforces the constraint that the length ratio of the major axis and the minor axis does not exceed r.

[0077] Furthermore, this embodiment also proposes another penalty term to constrain the excessively large Gaussian to prevent local over-reconstruction:

[0078]

[0079] Among them, sg represents the stop gradient operator, Esize Penalizes Gaussians whose size exceeds the mean by α times.

[0080] For the initialization of the skin Gaussian, this embodiment uses the initialized joint Gaussian kernel as input for training to achieve high fidelity quality. During the training process, the position of the skin Gaussian is updated by differentiable rasterization. According to the human body parameterized model SMPL, where the skin vertices are driven by predefined joint motions and skinning weights, each skin Gaussian is bound to the k nearest neighbor joint Gaussians, which facilitates the interpolation of position and rotation for sequential optimization.

[0081] Specifically, for each skin Gaussian position Find the k nearest neighbors (KNN, k=8) of the joint Gaussian in the containing ellipsoid As its anchor joint, and define the blend weights:

[0082]

[0083] Where l is the influence radius. The KNN graph and the blending weights are key to the subsequent optimization process and remain unchanged. It is worth noting that the method of this embodiment provides greater flexibility than parameterized models that rely on predefined joints, skin weights, or fixed topology. Experimental results show that the above method can handle a wide range of dynamic sequences.

[0084] For the joint Gaussian, the complete loss function is as follows:

[0085] Enit=isoEiso+λsizeEsize+Ecolor

[0086] Ecolo r is the photometric loss, and uses the following empirically determined parameters: r = 4, α = 3, l = 0.001, λ iso =0.005,λ size =1.

[0087] S3, uses a coarse-to-fine optimization strategy for dual Gaussian optimization and ensures temporal consistency.

[0088] In sequence training, this embodiment fixes the number of DualGS and optimizes the motion of the joint Gaussian and the appearance of the skin Gaussian. It can be observed that the Gaussian tends to change the appearance rather than update the position to adapt to the photometric loss. To solve this problem, S3 of this embodiment adopts a coarse-to-fine training strategy, starting from a single rough alignment and gradually entering a comprehensive refinement optimization to achieve robust human motion tracking and high-fidelity rendering.

[0089] After initializing the joint Gaussian of the first frame, the color, opacity and scale attributes are fixed to focus on capturing the dynamic movement of the human body. At this stage, only the movement of the joint Gaussian is fine-tuned, and a smooth regularizer is used to constrain the movement of the joint Gaussian as rigidly as possible in the local range:

[0090]

[0091]

[0092] where N(i) is a set of neighbors of the joint Gaussian, wi,k represents the influence weights, and R(·) converts the quaternion back to a rotation matrix.

[0093] Furthermore, the photometric loss is also added in the rough alignment process, and the complete energy formula is as follows:

[0094]

[0095] is the mixed color after joint Gaussian rasterization.

[0096] To cope with complex motion, a velocity attribute is maintained for each Gaussian and weighted updates are made using the position change between the last two frames. Before coarse alignment, the Gaussian positions for the new frame are first estimated based on the last frame and the velocity, and then non-rigid constraints (ARAP) are applied to restrict unreasonable motion.

[0097] This embodiment optimizes the motion of the joint Gaussian and the appearance of the skin Gaussian through a differentiable tracking and rendering process. The joint Gaussian motion in the rough alignment stage is used to interpolate the position and rotation of the skin Gaussian to balance rendering quality and temporal consistency:

[0098]

[0099]

[0100] in and Represent the KNN graph and mixed weights pre-calculated in the initialization phase; during the back propagation process, the gradient of the skin Gaussian Further propagation along the computation graph to the joint Gaussian and

[0101] The overall energy formula is as follows:

[0102]

[0103] E tempis a temporal term that promotes a consistent appearance by directly constraining scale, opacity, and color properties to not change significantly between adjacent frames.

[0104] In the implementation process of this embodiment, the motion is first initialized using velocity prediction, and then 10,000 training iterations are performed at each stage. The hyperparameters are set as follows:

[0105] λ temp =0.00003.

[0106] S4, using the decoupling ability of the dual-layer Gaussian expression for color and motion, applies residual vector quantization to the motion attributes of the joint Gaussian, and then compresses losslessly; and / or embeds the opacity and scaling of the skin Gaussian into an independent lookup table, and then applies image codec compression; and / or compresses the spherical harmonic attributes of the skin Gaussian into a persistent spherical harmonic index through a persistent codebook, and combines it with length encoding. The goal of this embodiment is to seamlessly integrate the high-quality 4D assets generated by DualGS into low-end devices with limited memory, see Figure 4 , such as head-mounted displays. For example, users can immersively experience a music journey in a VR environment. Thanks to the effective decoupling provided by DualGS, motion and appearance can be organically compressed from joint Gaussians and skin Gaussians respectively. The strategy of this embodiment achieves a compression ratio of up to 120 times while still being able to decode high-fidelity rendering results in real time.

[0107] The accuracy of Gaussian positions plays a crucial role in the quality of the scene, and even a tiny error can severely affect the rendering quality. To address this problem, we first apply residual vector quantization (R-VQ) to the joint Gaussian motion. The position R of the first frame is retained in the current segment. i,1 =p i,1 , and then perform time quantization:

[0108]

[0109] Where Q and Q -1 Denote quantization and inverse quantization respectively. In addition, R-VQ is applied to the rotation q. Compared with only quantizing the residuals of adjacent frames, the above scheme effectively prevents error accumulation. Furthermore, Range Arithmetic Number System (RANS) coding is also used for lossless compression.

[0110] Although residual compression can also be applied to opacity and scaling parameters in this embodiment, the compressed opacity and scaling still require a lot of storage space due to the significantly larger number of skin Gaussians. In order to achieve the best balance between data accuracy and storage overhead, the spatiotemporal relationship of the two Gaussian properties is exploited, and the opacity and scaling are embedded into independent lookup tables (LUTs) by using a temporal regularizer, and then encoded using image codec compression.

[0111] Specifically, the opacity and scale attributes are arranged into a 2D LUT, whose height corresponds to the number of skin Gaussians and width corresponds to the segment frame length. To enforce 2D consistency, we further sort the LUT by the average value of each row.

[0112] We then quantize and compress the 2D LUT into an 8-bit image using an image codec (WebP / JPEG) with a quality level of 100. Color attributes take up most of the storage space, accounting for 48 of the 59 parameters, and compressing them effectively can significantly save storage space.

[0113] However, applying entropy coding or codec compression to these coefficients still requires a lot of storage overhead. To this end, this embodiment designs a persistent codebook that exploits the temporal consistency of the skin Gaussian spherical harmonic (SH) parameters to achieve up to 360 times compression. K-Means clustering is applied to the d-order (d=0,1,2,3) SH coefficients of all frames in this segment. Codebook Z d Initialized to uniform distribution, and updated iteratively by randomly selecting a batch of d-order coefficients. After optimization, four codebooks of length L (set to 8192) are obtained.

[0114] The skin Gaussian SH properties are compactly encoded as SH indices via these codebooks:

[0115]

[0116] in is the SH index of order d of skin Gaussian i in frame t.

[0117] Furthermore, the compressed SH parameters can be restored by indexing the codebook

[0118]

[0119] Using this representation, the SH attributes originally composed of n×f×48 floating-point parameters are encoded into n×f×4 integer indices and four independent codebooks, where n is the number of skin Gaussians.

[0120] In addition, after conversion to indexes, the temporally coherent SH coefficients remain highly consistent. According to calculations, on average only one percent of the skin Gaussian SH indexes change between adjacent frames. Therefore, instead of saving the spatiotemporal SH index of each frame, only the index of the first frame and the position of the index change in the adjacent frames are saved. Specifically, if Then update it to the new index And save this integer quadruple (t, d, i, k). In real-time decoding, the spatiotemporal SH index of each frame can be decoded instantly based on these quadruple. In addition, the order of the quadruple does not affect the decoding process, and it is sorted in ascending order according to the first two variables, and then the length encoding is applied.

[0121] See also Figure 3 ,This embodiment proposes a dual Gaussian representation for processing complex scenes and achieving accurate tracking and high-fidelity rendering.,This embodiment cleverly tracks complex motion without sacrificing quality by organically,combining motion-aware joint Gaussians for global motion and appearance-aware skin Gaussians for visual details.

[0122] This embodiment also proposes a coarse-to-fine training strategy to reduce the optimization difficulty. In order to integrate long-term volumetric video sequences into the VR platform, this embodiment provides a compression strategy based on DualGS, achieving a compression ratio of 120 times. Experimental results show that the present invention can vividly generate high-quality rendering effects.

[0123] See also Figure 2 The second embodiment of the present invention provides a high-fidelity dynamic character rendering system, based on the method of the above embodiment, mainly comprising:

[0124] A collection module, used to collect multi-view human body images;

[0125] The tracking module uses relatively sparse motion-aware joint Gaussians to track the motion trajectory of the character;

[0126] The optimization module uses the motion-aware joint Gaussian obtained by the tracking module as the action prior, and further optimizes the appearance-aware skin Gaussian to generate a temporally consistent high-fidelity volumetric video;

[0127] A compression module is used to compress the Gaussian representation of high-fidelity volumetric video through residual quantization compression, and / or picture codec compression, and / or persistent code.

[0128] The third embodiment of the present invention also provides a high-fidelity dynamic character rendering device, including an acquisition device, a memory and a processor. The acquisition device is used to acquire multi-perspective human body images. The memory includes a computer program that can be run thereon. When the processor executes the computer program, the above-mentioned rendering method is implemented.

[0129] The memory may include a random access memory, a flash memory, a read-only memory, a programmable read-only memory, a non-volatile memory or a register, etc. The processor may be a central processing unit (CPU), etc. or a graphic processing unit (GPU). The memory may store executable instructions. The processor may execute the executable instructions stored in the memory to implement the various processes described herein.

[0130] It can be understood that the memory in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a ROM (Read-Only Memory), a PROM (Programmable ROM), an EPROM (Erasable PROM), an EEPROM (Electrically EPROM), or a flash memory. The volatile memory can be a RAM (Random Access Memory), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as SRAM (Static RAM), DRAM (Dynamic RAM), SDRAM (Synchronous DRAM), DDRSDRAM (Double Data Rate SDRAM), ESDRAM (Enhanced SDRAM), SLDRAM (Synchlink DRAM), and DRRAM (Direct Rambus RAM). The memory described herein is intended to include, but is not limited to, these and any other suitable types of memory. In some embodiments, the memory stores the following elements, upgrade packages, executable units, or data structures, or a subset thereof, or an extended set thereof: an operating system and an application.

[0131] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program includes various application programs, which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application program.

[0132] In the embodiment of the present invention, the processor calls a program or instruction stored in the memory, specifically, a program or instruction stored in an application program, and the processor is used to execute the rendering method steps provided in the first embodiment.

[0133] The fourth embodiment of the present invention further provides a chip for executing the rendering method in the first aspect. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip is used to execute the rendering method in the first embodiment.

[0134] In addition, in a fifth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the rendering method in the first embodiment of the present invention are implemented.

[0135] For example, machine-readable storage media may include, but are not limited to, various known and unknown types of non-volatile memory.

[0136] A sixth embodiment of the present invention further provides a computer program product, including computer program instructions, which enable a computer to execute the above method.

[0137] It will be appreciated by those skilled in the art that the units and steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

Claims

1. A volumetric video character rendering method based on double-layer Gaussian splashing, characterized by: S1, collects human body pictures with actions from multiple perspectives; S2, uses a small number of joint Gaussians for motion perception to capture global motion, a large number of skin Gaussians for appearance perception to capture appearance details, and establishes a mapping between skin Gaussians and joint Gaussians; S3, uses a coarse-to-fine optimization strategy for dual Gaussian optimization and ensures temporal consistency; S4, applying residual vector quantization to the motion properties of the joint Gaussians and then losslessly compressing them; and / or embedding the opacity and scaling of the skin Gaussians into an independent lookup table and then applying an image codec for compression; and / or compressing the spherical harmonic properties of the skin Gaussians into persistent spherical harmonic indices through a persistent codebook and combining them with length encoding.

2. The method according to claim 1, characterized in that: In S2, the properties of Gaussian are divided into two groups: motion perception parameters include position p and rotation q; appearance parameters include spherical harmonic coefficients C, opacity σ and scale s.

3. The method according to claim 1, characterized in that: In S2, a small number of joint Gaussians are initialized to represent the dynamics of global motion; the initialized joint Gaussian kernel is used as input for training to initialize the skin Gaussian; each skin Gaussian is anchored to several nearest neighbor joint Gaussians.

4. The method according to claim 3, characterized in that: In order to constrain the scaling of the joint Gaussian that is too thin, an isotropic loss is used: s i represents the scaling parameter of the Gaussian of the i-th joint, e is the activation function; and enforces the constraint that the length ratio of the major axis and the minor axis does not exceed r.

5. The method according to claim 1, characterized in that: In S3, a velocity attribute is maintained for each Gaussian and weighted updates are made using the position change between the last two frames; before rough alignment, the Gaussian position of the new frame is estimated based on the last frame and velocity, and then non-rigid constraints are applied to limit unreasonable motion.

6. The method according to claim 1, characterized in that: In S4, residual vector quantization is applied to the joint Gaussian motion, retaining the position R of the first frame in the current segment. i,1 =p i,1 , and then perform time quantization: Where Q and Q -1 They represent quantization and dequantization respectively.

7. A high-fidelity dynamic character rendering system, characterized by: A collection module, used to collect multi-view human body images; The tracking module uses relatively sparse motion-aware joint Gaussians to track the motion trajectory of the character; The optimization module uses the motion-aware joint Gaussian obtained by the tracking module as the action prior, and further optimizes the appearance-aware skin Gaussian to generate a temporally consistent high-fidelity volumetric video; A compression module is used to compress the Gaussian representation of high-fidelity volumetric video through residual quantization compression, and / or picture codec compression, and / or persistent code.

8. A high-fidelity dynamic character rendering device, characterized in that: It includes an acquisition device, a memory and a processor. The acquisition device is used to acquire multi-view human body pictures. The memory includes a computer program that can be run thereon. When the processor executes the computer program, the above-mentioned rendering method is implemented.

9. A chip, characterized in that: It includes one or more processors, which are used to call and run a computer program from a memory, so that a device equipped with the chip executes the rendering method as described above.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to perform the above rendering method when executed by a computer processor.

Images