A volumetric video reconstruction method, apparatus, device and readable storage medium

Abstract

The application discloses a volumetric video reconstruction method and device, equipment and a readable storage medium in the technical field of video processing. Different motion encoders are used for different motion partitions, and deformation processing is performed according to the motion characteristics, thereby avoiding one-size-fits-all. After the deformation of each Gaussian point in the normalized space Gaussian point cloud is obtained, the appearance attributes of the deformed normalized space Gaussian point cloud are fine-tuned end-to-end, and finally, after the fine-tuning is completed, each frame of Gaussian point cloud is exported to generate a volumetric video corresponding to a target. The application no longer regards the target as a single whole, but decouples it into a combination of semantic components with different kinematic characteristics, and different motion encoders are used for processing corresponding different motion partitions, so that a high-quality volumetric video without blur, flicker and tearing caused by motion differences is finally obtained.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a volumetric video reconstruction method, apparatus, device, and readable storage medium. Background Technology

[0002] Volumetric video (also known as spatial video or volumetric video) is a technique that captures three-dimensional (3D) scenes from multiple viewpoints (usually people or objects), creating a realistic and immersive experience for viewers. It combines cameras, sensors, and software to acquire and process data from multiple angles, ultimately generating a dynamic 3D representation of the scene.

[0003] Volumetric video allows viewers to enter and interact with virtual environments, offering a more immersive and engaging experience compared to traditional videos. However, current video reconstruction technologies often produce volumetric videos that are blurry, flickering, and torn, failing to meet user expectations.

[0004] In summary, how to effectively solve problems such as improving the quality of volumetric video reconstruction is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a volumetric video reconstruction method, apparatus, device, and readable storage medium, which can effectively improve the reconstruction quality of physical examination videos through motion partitioning.

[0006] To solve the above-mentioned technical problems, this application provides the following technical solution:

[0007] A volumetric video reconstruction method includes: acquiring a set of multi-view images of the same target simultaneously captured from different perspectives; using the multi-view image set to create a canonical Gaussian point cloud corresponding to the target; wherein each Gaussian point in the canonical Gaussian point cloud has a semantic label, and the semantic label corresponds to different motion regions of the target; traversing the Gaussian points in the canonical Gaussian point cloud, determining a matching motion encoder from a motion dictionary using the semantic label of the current Gaussian point, and encoding the current Gaussian point using the matching motion encoder to obtain the deformation of the current Gaussian point; after obtaining the deformation of each Gaussian point in the canonical Gaussian point cloud, performing end-to-end fine-tuning of the appearance attributes of the deformed canonical Gaussian point cloud; after completing the fine-tuning, generating a volumetric video corresponding to the target by exporting each frame of the Gaussian point cloud.

[0008] Preferably, creating a canonical Gaussian point cloud corresponding to the target using the multi-view image set includes: generating an initial sparse point cloud corresponding to a static scene using the starting frame in the multi-view image set; performing 3DGS optimization on the initial sparse point cloud to obtain a canonical spatial geometry; rendering the canonical spatial geometry to the view index corresponding to the multi-view image set using a 3DGS rasterizer, and tracking the color and features of pixels to determine the Gaussian point with the largest contribution; traversing all pixels of the current view index and determining semantic labels from a semantic truth mask; voting on the category of the Gaussian point using the semantic labels; obtaining the voting vector for each Gaussian point after traversing all view indices; and determining the semantic label of the Gaussian point using the voting vector.

[0009] Preferably, obtaining the semantic truth mask includes: determining the semantic truth mask for each frame in the multi-view image set; the semantic truth mask corresponds to several semantic categories corresponding to motion partitions, and the semantic categories include, but are not limited to, head, body and clothing.

[0010] Preferably, determining the matching motion encoder from the motion dictionary using the semantic label of the current Gaussian point includes: reading the semantic label of the current Gaussian point and selecting the motion encoder corresponding to the semantic label from the motion dictionary; wherein the motion dictionary records the motion encoders corresponding to different semantic labels respectively.

[0011] Preferably, the current Gaussian point is encoded using a matched motion encoder to obtain the deformation of the current Gaussian point, including: encoding the position and rotation of the current Gaussian point using a matched motion encoder to obtain the deformation of the current Gaussian point.

[0012] Preferably, after obtaining the deformation of all Gaussian points, the method further includes: using a 3DGS rasterizer to render the dynamic Gaussian points to the view index to obtain a rendered image; calculating the reconstruction loss between the rendered image and the ground truth image; determining the gradient of the reconstruction loss with respect to all trainable parameters in the motion dictionary; and directing the gradient to the corresponding motion encoder to optimize the motion encoder.

[0013] Preferably, acquiring a set of multi-view images of the same target simultaneously captured from different perspectives includes: simultaneously capturing the target from different perspectives using a matrix camera to obtain an initial set of multi-view images; and extracting foreground images from each image frame in the initial set of multi-view images to obtain the set of multi-view images.

[0014] A volumetric video reconstruction device includes: an image acquisition module for acquiring a set of multi-view images of the same target simultaneously captured from different perspectives; an initialization module for creating a canonical spatial Gaussian point cloud corresponding to the target using the multi-view image set; wherein each Gaussian point in the canonical spatial Gaussian point cloud has a semantic label, and the semantic label corresponds to different motion regions of the target; a partition deformation processing module for traversing the Gaussian points in the canonical spatial Gaussian point cloud, determining a matching motion encoder from a motion dictionary using the semantic label of the current Gaussian point, and encoding the current Gaussian point using the matching motion encoder to obtain the deformation of the current Gaussian point; a fine-tuning module for performing end-to-end fine-tuning of the appearance attributes of the deformed canonical spatial Gaussian point cloud after obtaining the deformation of each Gaussian point in the canonical spatial Gaussian point cloud; and a video generation module for generating a volumetric video corresponding to the target by exporting each frame of the Gaussian point cloud after completing the fine-tuning.

[0015] An electronic device includes: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the volumetric video reconstruction method described above.

[0016] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the volumetric video reconstruction method described above.

[0017] The method provided in this application provides a multi-view image set of the same target captured simultaneously from different perspectives. Using the multi-view image set, a canonical Gaussian point cloud corresponding to the target is created. Each Gaussian point in the canonical Gaussian point cloud has a semantic label, which corresponds to different motion regions of the target. The Gaussian points in the canonical Gaussian point cloud are traversed, and a matching motion encoder is determined from the motion dictionary using the semantic label of the current Gaussian point. The matching motion encoder is then used to encode the current Gaussian point to obtain its deformation. After obtaining the deformation of each Gaussian point in the canonical Gaussian point cloud, the appearance attributes of the deformed canonical Gaussian point cloud are fine-tuned end-to-end. After fine-tuning, each frame of the Gaussian point cloud is exported to generate a volumetric video corresponding to the target.

[0018] In this application, through analysis of volumetric video reconstruction, it was found that the root cause of blurring, flickering, and tearing in the reconstructed volumetric video lies in the fact that a unified or holistic mathematical model is typically used to describe the motion of all parts of the target during the reconstruction process. However, the physical characteristics of different locations of the same target, such as the motion of a dancer's torso (nearly rigid) and the motion of the skirt (highly non-rigid, high-frequency, fluid), are drastically different. To maintain the stability of the torso, this one-size-fits-all model must average or blur the high-frequency details of the skirt, resulting in a blurred reconstructed skirt or flickering and tearing during rapid motion.

[0019] Based on this, this application proposes using different motion encoders for deformation processing of different motion regions during the generation of volumetric video. This allows Gaussian points corresponding to different motion regions to have different deformations. Specifically, this application first acquires a set of multi-view images of the same target simultaneously captured from different perspectives. Then, using the multi-view image set, a canonical Gaussian point cloud corresponding to the target is created. It is important to note that in this application, each Gaussian point in the canonical Gaussian point cloud not only has standard optimizable attributes but also semantic labels, which correspond to different motion regions of the target. By traversing the Gaussian points in the canonical Gaussian point cloud, the semantic label of the current Gaussian point can be used to determine the matching motion encoder from the motion dictionary, and the matching motion encoder is used to encode the current Gaussian point to obtain its deformation. That is, different motion regions can use different motion encoders, and deformation processing can be performed according to their motion characteristics, avoiding a one-size-fits-all approach. Thus, after obtaining the deformation of each Gaussian point in the normalized space Gaussian point cloud, the appearance attributes of the deformed normalized space Gaussian point cloud are fine-tuned end-to-end. Finally, after completing the fine-tuning, each frame of the Gaussian point cloud is exported to generate a volumetric video corresponding to the target.

[0020] This application no longer treats the target as a single whole, but decouples it into a combination of semantic components with different kinematic properties, and uses different motion encoders to process different motion partitions, ultimately obtaining high-quality volumetric video without blurring, flickering and tearing caused by motion differences.

[0021] Accordingly, embodiments of this application also provide a volumetric video reconstruction apparatus, device, and readable storage medium corresponding to the above-described volumetric video reconstruction method, which have the aforementioned technical effects, and will not be elaborated further here. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the implementation of a volumetric video reconstruction method in this application.

[0024] Figure 2 This is a schematic diagram of the structure of a volumetric video reconstruction device according to an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of the specific structure of an electronic device in an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] Please refer to Figure 1 , Figure 1 This is a flowchart of a volumetric video reconstruction method according to an embodiment of this application. The method includes the following steps.

[0029] S101. Obtain a set of multi-view images of the same target obtained by simultaneously shooting from different perspectives.

[0030] The multi-view image set refers to the set of image frames required to construct the volumetric video corresponding to the target. This image frame set includes a sequence of images simultaneously captured from different viewpoints. Furthermore, since the volumetric video is reconstructed from the target, the images in this image sequence can be images that have undergone image processing, retaining only the foreground content corresponding to the target, i.e., no background content.

[0031] In this embodiment, a set of multi-view images can be obtained by directly downloading from the network, reading from a storage medium, or receiving data from other devices.

[0032] In one specific embodiment of this application, obtaining a multi-view image set obtained by simultaneously capturing the same target from different perspectives includes: simultaneously capturing the target from different perspectives using a matrix camera to obtain an initial multi-view image set; and extracting foreground images from each image frame in the initial multi-view image set to obtain the multi-view image set.

[0033] In this context, a matrix camera refers to multiple cameras arranged like a matrix. That is, a matrix camera can simultaneously capture images of the target from different perspectives, thus obtaining an initial multi-view image set. Then, image extraction is performed on each image frame in the initial multi-temporal image set, using only the foreground images from each frame to obtain the final multi-view image set.

[0034] This multi-view image set can be represented as... ,in, For perspective indexing, For timestamps.

[0035] S102. Using a collection of multi-view images, create a Gaussian point cloud in a normalized spatial space corresponding to the target.

[0036] In the standard space Gaussian point cloud, each Gaussian point has a semantic label, and the semantic label corresponds to different motion regions of the target.

[0037] After obtaining the multi-view image set, an initial model corresponding to the target can be created, or a canonical Gaussian point cloud corresponding to the target can be obtained by sampling based on an initial model. Each Gaussian point in the canonical Gaussian point cloud has a semantic label, and the semantic label corresponds to different motion regions of the target. That is, in this embodiment, a 3D Gaussian point cloud G can be maintained in the canonical space (i.e., the static pose at t=0).

[0038] In other words, in the embodiments of this application, each Gaussian point in the standard spatial Gaussian point cloud not only has standard optimizable attributes, such as, but also has semantic labels corresponding to the motion region.

[0039] That is, in the embodiments of this application, a core extension is performed on standard 3DGS: each Gaussian point It not only has standard attributes, but also a semantic tag. ,in, To standardize the location, For scaling, For rotation, For opacity, is the spherical harmonic coefficient.

[0040] In one specific embodiment of this application, a canonical Gaussian point cloud corresponding to a target is created using a multi-view image set, including: generating an initial sparse point cloud corresponding to a static scene using the starting frame in the multi-view image set; performing 3DGS optimization on the initial sparse point cloud to obtain a canonical spatial geometry; using a 3DGS rasterizer to render the canonical spatial geometry to the view index corresponding to the multi-view image set, and tracking the color and features of pixels to determine the Gaussian point that contributes the most; traversing all pixels of the current view index and determining semantic labels from a semantic truth mask; voting on the category of the Gaussian point using the semantic labels; obtaining the voting vector for each Gaussian point after traversing all view indices; and determining the semantic label of the Gaussian point using the voting vector.

[0041] The process of obtaining the semantic truth mask includes: determining a semantic truth mask for each frame in the multi-view image set; the semantic truth mask corresponds to several semantic categories corresponding to motion partitions, and the semantic categories include, but are not limited to, head, body and clothing.

[0042] Specifically, it can be for each frame Pre-computed 2D semantic truth mask These masks categorize pixels into N semantic categories (e.g., k=1: head, k=2: body, k=3: clothing, etc.). They can be generated using existing 2D segmentation models (such as SAM2 or SAM3) combined with labels.

[0043] Then, the semantic Gaussian is initialized, i.e., the canonical Gaussian point cloud G at time t=0 is obtained. The specific implementation process includes the following steps.

[0044] Step 1.1, Geometric Initialization: First, a static 3DGS model in a rest pose or canonical pose is required. This can be achieved in standard ways: such as using Structure from Motion (SfM) to process the t=0 frame of the input video to generate an initial sparse point cloud, or using a predefined human template (such as SMPL) and sampling points on its surface to initialize the positions of Gaussian points. .

[0045] right A high-quality canonical spatial geometry is obtained by performing standard 3DGS optimization on the static scene (using only the multi-view images at t=0, i.e., the starting frame in the multi-view image set). .

[0046] Step 1.2, Semantic Enhancement and Tag Assignment: Its goal is to... Each Gaussian point in Assigning semantic tags .

[0047] Specifically, a tag fusion strategy based on multi-view rendering and voting can be adopted. The detailed process is as follows.

[0048] Initialize the voting counter for each Gaussian point. Initialize an N-dimensional vector , is used to count its votes in each semantic category k.

[0049] Traverse all views (v): Perform rendering and association and 2D semantic supervision under each view.

[0050] Rendering and correlation: A standard 3DGS rasterizer can be used to render the standard point cloud. Render to view index v. During the rendering process, track which Gaussian points contribute the color and alpha value of each pixel. Record the indices of the M Gaussian points that contribute the most to the alpha value of the pixel (where M can be set to a value such as 1 or 3).

[0051] 2D Semantic Supervision: Load the 2D semantic truth mask corresponding to the viewpoint v at time t=0. .

[0052] Tag voting: Traverse all pixels in viewpoint v :from Search semantic tags (For example, (representing clothing). Find the record in step 3 for... The Gaussian point that contributes the most Let this be the Gaussian point. Vote for the corresponding category: .

[0053] Label determination: After traversing all views, for each Gaussian point To determine its final voting vector .

[0054] distribute : The final semantic tag identified as Index of the category with the most votes: .

[0055] After the above processing, a complete, semantically labeled, canonical Gaussian point cloud can be obtained. In the subsequent processing, It will remain unchanged.

[0056] S103. Traverse the Gaussian points in the normal space Gaussian point cloud, use the semantic label of the current Gaussian point to determine the matching motion encoder from the motion dictionary, and use the matching motion encoder to encode the current Gaussian point to obtain the deformation of the current Gaussian point.

[0057] To obtain a volumetric video, a temporally normalized Gaussian point cloud is required. This normalized Gaussian point cloud can be obtained by deforming and fine-tuning the normalized Gaussian point cloud at t=0.

[0058] Specifically, considering that different motion zones of the target have different characteristics, in this embodiment, different motion encoders will be used to encode the Gaussian points for different motion zones, so as to obtain deformations that conform to the characteristics of different motion zones.

[0059] In this implementation, a motion encoder dictionary E can be pre-built: ,in, This is a standalone motion encoder for semantic category k.

[0060] Each It can be encoded by a multi-resolution hash grid. And a small multilayer perceptron (MLP) Composition 3.

[0061] The input is a standard position. Given time t, the output is the deformation at that point. .

[0062] Furthermore, for different entries in the motion dictionary They can have different configurations. For example, (Clothing) can use more (Body) Higher resolution hash grid and deeper / wider MLP to give it a stronger ability to fit high-frequency dynamics.

[0063] For example, The optimization goal of the (clothing encoder) is simply to reconstruct the movement of the clothing. It does not need to concern itself with the stability of the body. Therefore, it can be designed with extremely high hash table resolution and deeper network layers, giving it a powerful representation capacity to fit high-frequency, non-rigid clothing dynamics.

[0064] at the same time, The optimization goal of the (body encoder) is simply to maintain the smoothness and stability of the torso. It will not be... High-frequency optimization may be contaminated or interfered with.

[0065] The output data of the motion encoder dictionary is a high-fidelity 4D character performance model that can be rendered in real time (defined by the canonical point cloud G and the motion encoder dictionary E).

[0066] In one specific embodiment of this application, determining a matching motion encoder from a motion dictionary using the semantic label of the current Gaussian point includes:

[0067] Read the semantic label of the current Gaussian point and select the motion encoder corresponding to the semantic label from the motion dictionary; the motion dictionary records the motion encoders corresponding to different semantic labels.

[0068] Specifically, the current Gaussian point is encoded using a matched motion encoder to obtain its deformation, including:

[0069] The position and rotation of the current Gaussian point are encoded using a matched motion encoder to obtain the deformation of the current Gaussian point.

[0070] Furthermore, after obtaining the deformation of all Gaussian points, the process also includes: using a 3DGS rasterizer to render the dynamic Gaussian points to the view index to obtain a rendered image; calculating the reconstruction loss between the rendered image and the ground truth image; determining the gradient of the reconstruction loss with respect to all trainable parameters in the motion dictionary; and directing the gradients to the corresponding motion encoder to optimize the motion encoder.

[0071] By training the motion encoder dictionary All trainable parameters ( and ), enabling it to be based on Accurately predict the motion of all Gaussian points at all times t.

[0072] The detailed process for obtaining deformation and training the motion encoder (frame-by-frame training loop) includes the following steps.

[0073] A batch is randomly sampled from a multi-view image collection, containing the timestamp t and the viewpoint. .

[0074] Obtain the ground truth image of this frame. .

[0075] Differentiated Deformation: Initializes an empty dynamic list of Gaussian points. Traverse every Gaussian point in the normalized point cloud G. : Read its fixed semantic tags Select the corresponding motion encoder from dictionary E. Deformation calculation: Calculate the dynamic Gaussian point at time t. Attributes: ;

[0076] The appearance and geometry (which remain unchanged at this point) are as follows: Will Add to List.

[0077] Rendering: Using the standard 3DGS rasterizer, all dynamic Gaussian points are rendered. Rendered to viewpoint v, resulting in a rendered image. .

[0078] Loss Calculation: Calculates the reconstruction loss between the rendered image and the ground truth image. (For example, a combination of L1 loss and SSIM loss): .

[0079] Backpropagation and gradient isolation: computation The gradient is relative to the gradients of all trainable parameters in dictionary E. Due to semantic gating, the backpropagation paths of the gradients are automatically isolated. For example: labeled as clothing ( Gaussian point The resulting loss gradient will only flow to and update of The parameters. Marked as header ( Gaussian point The resulting loss ladder will only flow to and update The parameters. This gradient isolation optimization fundamentally solves the averaging problem. The optimization is driven only by clothing reconstruction errors, allowing it to freely develop high-capacity, high-frequency representations without worrying about whether this will compromise the stability of the body.

[0080] S104. After obtaining the deformation of each Gaussian point in the normalized space Gaussian point cloud, perform end-to-end fine-tuning of the appearance attributes of the deformed normalized space Gaussian point cloud.

[0081] After step S103, based on the accurate reconstruction of kinematics (position, rotation), the appearance attributes (scaling, spherical harmonics, and opacity) of the canonical point cloud G are finely tuned end-to-end to capture subtle appearance changes caused by motion and viewpoint. That is, motion (e.g., clothing folds) causes complex self-shadowing and lighting changes, which can be globally optimized after the motion model is fixed, i.e., a stable decoupled optimization strategy.

[0082] The detailed implementation process includes the following steps.

[0083] Freeze Motion: Fixes all motion encoders trained in step S103. All parameters.

[0084] Fine-tune Appearance: Only optimizes the appearance parameters in the normalized Gaussian point cloud G, i.e. The optimization type of the motion encoder in the training loop and step S103 will not be described in detail here.

[0085] Ultimately, a high-fidelity 4D model can be obtained in terms of geometry, motion, and appearance.

[0086] S105. After fine-tuning, export each frame of Gaussian point cloud to generate a volumetric video corresponding to the target.

[0087] In other words, exporting the gigabyte (G) of each frame yields the final video file. In practical applications, further compression can be performed to reduce the file size.

[0088] The method provided in this application provides a multi-view image set of the same target captured simultaneously from different perspectives. Using the multi-view image set, a canonical Gaussian point cloud corresponding to the target is created. Each Gaussian point in the canonical Gaussian point cloud has a semantic label, which corresponds to different motion regions of the target. The Gaussian points in the canonical Gaussian point cloud are traversed, and a matching motion encoder is determined from the motion dictionary using the semantic label of the current Gaussian point. The matching motion encoder is then used to encode the current Gaussian point to obtain its deformation. After obtaining the deformation of each Gaussian point in the canonical Gaussian point cloud, the appearance attributes of the deformed canonical Gaussian point cloud are fine-tuned end-to-end. After fine-tuning, each frame of the Gaussian point cloud is exported to generate a volumetric video corresponding to the target.

[0089] In this application, through analysis of volumetric video reconstruction, it was found that the root cause of blurring, flickering, and tearing in the reconstructed volumetric video lies in the fact that a unified or holistic mathematical model is typically used to describe the motion of all parts of the target during the reconstruction process. However, the physical characteristics of different locations of the same target, such as the motion of a dancer's torso (nearly rigid) and the motion of the skirt (highly non-rigid, high-frequency, fluid), are drastically different. To maintain the stability of the torso, this one-size-fits-all model must average or blur the high-frequency details of the skirt, resulting in a blurred reconstructed skirt or flickering and tearing during rapid motion.

[0090] Based on this, this application proposes using different motion encoders for deformation processing of different motion regions during the generation of volumetric video. This allows Gaussian points corresponding to different motion regions to have different deformations. Specifically, this application first acquires a set of multi-view images of the same target simultaneously captured from different perspectives. Then, using the multi-view image set, a canonical Gaussian point cloud corresponding to the target is created. It is important to note that in this application, each Gaussian point in the canonical Gaussian point cloud not only has standard optimizable attributes but also semantic labels, which correspond to different motion regions of the target. By traversing the Gaussian points in the canonical Gaussian point cloud, the semantic label of the current Gaussian point can be used to determine the matching motion encoder from the motion dictionary, and the matching motion encoder is used to encode the current Gaussian point to obtain its deformation. That is, different motion regions can use different motion encoders, and deformation processing can be performed according to their motion characteristics, avoiding a one-size-fits-all approach. Thus, after obtaining the deformation of each Gaussian point in the normalized space Gaussian point cloud, the appearance attributes of the deformed normalized space Gaussian point cloud are fine-tuned end-to-end. Finally, after completing the fine-tuning, each frame of the Gaussian point cloud is exported to generate a volumetric video corresponding to the target.

[0091] This application no longer treats the target as a single whole, but decouples it into a combination of semantic components with different kinematic properties, and uses different motion encoders to process different motion partitions, ultimately obtaining high-quality volumetric video without blurring, flickering and tearing caused by motion differences.

[0092] In other words, this application no longer treats the target, such as the human body, as a single whole, but rather decouples it into a combination of semantic components with different kinematic properties. Specifically, in the semantic attribution of Gaussians, all 3D Gaussian points are no longer treated as undifferentiated geometric primitives. Using a 2D semantic segmentation mask synchronized with the input 2D video (e.g., generated by a 2D segmentation model such as SAM2) as a supervision signal, a semantic lifting process is employed to assign a persistent, categorical semantic label to each Gaussian point during the initialization of the 3D Gaussian point cloud.

[0093] Simultaneously, semantic-based motion model gating and differentiated encoding are employed. Instead of a single, globally shared motion encoder, a dictionary of motion encoders is designed and maintained, where each encoder is an independent, differentially configurable motion coding network.

[0094] This new architecture completely solves the problem of averaging from a single model. Semantic attributes act as a router or gating signal.

[0095] During the motion reconstruction phase, when the system needs to calculate the motion of the Gaussian point: the system first reads the Gaussian electrical signal. (For example, clothing).

[0096] The system then uses this semantic label as a key to query and select the unique corresponding motion encoder from the motion encoder dictionary. The motion of the Gaussian point is calculated only by the selected unique motion encoder.

[0097] Semantic gating was used to decouple the optimization objective and isolate the gradient.

[0098] In this way, this method allows different body parts with vastly different motion characteristics to be reconstructed in parallel, without interference, and in a manner best suited to their characteristics. This fundamentally solves the artifacts and loss of detail problems caused by averaging in existing technologies.

[0099] Corresponding to the above method embodiments, this application also provides a volumetric video reconstruction apparatus. The volumetric video reconstruction apparatus described below can be referred to in correspondence with the volumetric video reconstruction method described above.

[0100] See Figure 2 As shown, the device includes the following modules.

[0101] The image acquisition module 101 is used to acquire a set of multi-view images of the same target simultaneously captured from different perspectives.

[0102] The initialization module 102 is used to create a normalized spatial Gaussian point cloud corresponding to the target using a multi-view image set; wherein each Gaussian point in the normalized spatial Gaussian point cloud has a semantic label, and the semantic label corresponds to different motion regions of the target.

[0103] The partition deformation processing module 103 is used to traverse the Gaussian points in the normal space Gaussian point cloud, determine the matching motion encoder from the motion dictionary using the semantic label of the current Gaussian point, and use the matching motion encoder to encode the current Gaussian point to obtain the deformation of the current Gaussian point.

[0104] The fine-tuning module 104 is used to perform end-to-end fine-tuning of the appearance attributes of the deformed Gaussian point cloud after obtaining the deformation of each Gaussian point in the normalized space Gaussian point cloud.

[0105] The video generation module 105 is used to generate a volumetric video corresponding to the target by exporting each frame of Gaussian point cloud after fine-tuning.

[0106] Using the apparatus provided in the embodiments of this application, a multi-view image set of the same target simultaneously captured from different perspectives is acquired; using the multi-view image set, a canonical spatial Gaussian point cloud corresponding to the target is created; wherein, each Gaussian point in the canonical spatial Gaussian point cloud has a semantic label, and the semantic label corresponds to different motion regions of the target; traversing the Gaussian points in the canonical spatial Gaussian point cloud, using the semantic label of the current Gaussian point to determine the matching motion encoder from the motion dictionary, and using the matching motion encoder to encode the current Gaussian point to obtain the deformation of the current Gaussian point; after obtaining the deformation of each Gaussian point in the canonical spatial Gaussian point cloud, the appearance attributes of the deformed canonical spatial Gaussian point cloud are fine-tuned end-to-end; after completing the fine-tuning, each frame of the Gaussian point cloud is exported to generate a volumetric video corresponding to the target.

[0107] In this application, through analysis of volumetric video reconstruction, it was found that the root cause of blurring, flickering, and tearing in the reconstructed volumetric video lies in the fact that a unified or holistic mathematical model is typically used to describe the motion of all parts of the target during the reconstruction process. However, the physical characteristics of different locations of the same target, such as the motion of a dancer's torso (nearly rigid) and the motion of the skirt (highly non-rigid, high-frequency, fluid), are drastically different. To maintain the stability of the torso, this one-size-fits-all model must average or blur the high-frequency details of the skirt, resulting in a blurred reconstructed skirt or flickering and tearing during rapid motion.

[0108] Based on this, this application proposes using different motion encoders for deformation processing of different motion regions during the generation of volumetric video. This allows Gaussian points corresponding to different motion regions to have different deformations. Specifically, this application first acquires a set of multi-view images of the same target simultaneously captured from different perspectives. Then, using the multi-view image set, a canonical Gaussian point cloud corresponding to the target is created. It is important to note that in this application, each Gaussian point in the canonical Gaussian point cloud not only has standard optimizable attributes but also semantic labels, which correspond to different motion regions of the target. By traversing the Gaussian points in the canonical Gaussian point cloud, the semantic label of the current Gaussian point can be used to determine the matching motion encoder from the motion dictionary, and the matching motion encoder is used to encode the current Gaussian point to obtain its deformation. That is, different motion regions can use different motion encoders, and deformation processing can be performed according to their motion characteristics, avoiding a one-size-fits-all approach. Thus, after obtaining the deformation of each Gaussian point in the normalized space Gaussian point cloud, the appearance attributes of the deformed normalized space Gaussian point cloud are fine-tuned end-to-end. Finally, after completing the fine-tuning, each frame of the Gaussian point cloud is exported to generate a volumetric video corresponding to the target.

[0109] This application no longer treats the target as a single whole, but decouples it into a combination of semantic components with different kinematic properties, and uses different motion encoders to process different motion partitions, ultimately obtaining high-quality volumetric video without blurring, flickering and tearing caused by motion differences.

[0110] In one specific embodiment of this application, the initialization module is specifically used to generate an initial sparse point cloud corresponding to a static scene using the starting frame in the multi-view image set.

[0111] The initial sparse point cloud is optimized using 3DGS to obtain a normalized spatial geometry.

[0112] Using the 3DGS rasterizer, the normalized spatial geometry is rendered to the view index corresponding to the multi-view image set, and the color and features of the pixels are tracked to determine the Gaussian point that contributes the most.

[0113] Traverse all pixels at the current view index and determine the semantic label from the semantic truth mask;

[0114] Use semantic tags to vote on the category of Gaussian points;

[0115] After traversing all view indices, obtain the voting vector for each Gaussian point;

[0116] The semantic labels of Gaussian points are determined using voting vectors.

[0117] In one specific embodiment of this application, the initialization module is specifically used to determine a semantic truth mask for each frame in the multi-view image set; the semantic truth mask corresponds to several semantic categories corresponding to motion partitions, and the semantic categories include, but are not limited to, head, body and clothing.

[0118] In one specific embodiment of this application, the partition deformation processing module is specifically used to read the semantic label of the current Gaussian point and select the motion encoder corresponding to the semantic label from the motion dictionary; wherein, the motion dictionary records the motion encoders corresponding to different semantic labels respectively.

[0119] In one specific embodiment of this application, the partition deformation processing module is specifically used to encode the position and rotation of the current Gaussian point using a matched motion encoder to obtain the deformation of the current Gaussian point.

[0120] In one specific embodiment of this application, the partition deformation processing module has the following functions: after obtaining the deformation of all Gaussian points, it uses a 3DGS rasterizer to render the dynamic Gaussian points to the view index to obtain a rendered image; calculates the reconstruction loss between the rendered image and the ground truth image; determines the gradient of the reconstruction loss with respect to all trainable parameters in the motion dictionary; and directs the gradient to the corresponding motion encoder to optimize the motion encoder.

[0121] In one specific embodiment of this application, the image acquisition module is specifically used to simultaneously capture the target from different perspectives using a matrix camera to obtain an initial multi-view image set; and to extract the foreground image from each image frame in the initial multi-view image set to obtain the multi-view image set.

[0122] Corresponding to the above method embodiments, this application also provides an electronic device. The electronic device described below and the volumetric video reconstruction method described above can be referred to in correspondence.

[0123] See Figure 3 As shown, the electronic device includes: a memory 332 for storing a computer program; and a processor 322 for executing the computer program to implement the steps of the volumetric video reconstruction method of the above-described method embodiments.

[0124] For details, please refer to Figure 4 , Figure 4This is a schematic diagram of the specific structure of an electronic device provided in this embodiment. The electronic device can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) (e.g., one or more processors) and a memory 332. The memory 332 stores one or more computer programs 342 or data 344. The memory 332 can be temporary or permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 and execute the series of instruction operations stored in the memory 332 on the electronic device 301.

[0125] Electronic device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0126] The steps in the volumetric video reconstruction method described above can be implemented by the structure of an electronic device.

[0127] Corresponding to the above method embodiments, this application also provides a readable storage medium. The readable storage medium described below can be referred to in conjunction with the volumetric video reconstruction method described above.

[0128] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the volumetric video reconstruction method described in the above method embodiments.

[0129] The readable storage medium can specifically be a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or any other readable storage medium capable of storing program code.

[0130] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0131] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0132] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0133] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "include," "contain," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0134] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A volumetric video reconstruction method, characterized by, The method comprises the following steps: acquiring a multi-view image set obtained by synchronously photographing a same target from different perspectives; wherein the target is a person target; creating a canonical space Gaussian point cloud corresponding to the target by using the multi-view image set; wherein each Gaussian point in the canonical space Gaussian point cloud has a semantic label, and the semantic label corresponds to a different motion area of the target; traversing the Gaussian points in the canonical space Gaussian point cloud, determining a matched motion encoder from a motion dictionary by using the semantic label of a current Gaussian point, and performing encoding processing on the current Gaussian point by using the matched motion encoder to obtain a deformation of the current Gaussian point; after obtaining the deformations of the Gaussian points in the canonical space Gaussian point cloud, performing end-to-end fine tuning on the appearance attributes of the deformed canonical space Gaussian point cloud; after completing the fine tuning, exporting each frame of Gaussian point cloud to generate a volumetric video corresponding to the target; wherein the motion dictionary contains a plurality of independent motion encoders respectively corresponding to different semantic categories; each independent motion encoder comprises a multi-resolution hash coding network and a multi-layer perception network; the independent motion encoder takes the canonical position and time of a Gaussian point as input and outputs the deformation of the Gaussian point; the independent motion encoders for different semantic categories have different network configurations, specifically including: the independent motion encoder corresponding to the clothing semantic category has a higher resolution of the multi-resolution hash coding network than the independent motion encoder corresponding to the body semantic category, and has a deeper or wider multi-layer perception network than the independent motion encoder corresponding to the body semantic category.

2. The method of claim 1, wherein, The method for creating a canonical space Gaussian point cloud corresponding to the target by using the multi-view image set comprises the following steps: generating an initial sparse point cloud corresponding to a static scene by using a starting frame in the multi-view image set; performing 3DGS optimization on the initial sparse point cloud to obtain a canonical space geometry; rendering the canonical space geometry to a perspective index corresponding to the multi-view image set by using a 3DGS rasterizer, and tracking the color and elements of pixels to determine the Gaussian point with the largest contribution; traversing all pixels of the current perspective index to determine a semantic label from a semantic ground truth mask; voting on the category of the Gaussian point by using the semantic label; after traversing all view indexes, obtaining a voting vector of each Gaussian point; determining the semantic label of the Gaussian point by using the voting vector.

3. The method of claim 2, wherein, The method for obtaining the semantic ground truth mask comprises the following steps: determining the semantic ground truth mask for each frame in the multi-view image set; the semantic ground truth mask corresponds to a plurality of semantic categories corresponding to motion partitions, and the semantic categories include but are not limited to head, body and clothing.

4. The method of claim 1, wherein, The method for determining a matched motion encoder from a motion dictionary by using the semantic label of a current Gaussian point comprises the following steps: reading the semantic label of the current Gaussian point, and selecting a motion encoder corresponding to the semantic label from the motion dictionary; wherein the motion dictionary records the motion encoders corresponding to different semantic labels respectively.

5. The method of claim 1, wherein, The method for performing encoding processing on the current Gaussian point by using the matched motion encoder to obtain the deformation of the current Gaussian point comprises the following steps: The position and rotation of the current Gaussian point are encoded by a matched motion encoder to obtain the deformation of the current Gaussian point.

6. The method of claim 5, wherein, After obtaining the deformation of all Gaussian points, further comprising: Using a 3DGS rasterizer, render the dynamic Gaussian points to the view index to obtain a rendered image. Calculate the reconstruction loss between the rendered image and the ground truth image. Determine the gradient of the reconstruction loss with respect to all trainable parameters in the motion dictionary. Flow the gradient to the corresponding motion encoder to optimize the motion encoder.

7. The method according to any one of claims 1 to 6, characterized in that, Obtain a multi-view image set of a same target synchronously captured from different views, comprising: Synchronously capture the target from different views using a matrix camera to obtain an initial multi-view image set; Extract foreground images from each image frame in the initial multi-view image set to obtain the multi-view image set.

8. A volumetric video reconstruction apparatus, characterized in that, Comprising: An image acquisition module for acquiring a multi-view image set of a same target synchronously captured from different views; wherein the target is a person target; An initialization module for creating a canonical space Gaussian point cloud corresponding to the target using the multi-view image set; wherein each Gaussian point in the canonical space Gaussian point cloud has a semantic label corresponding to a different motion region of the target; A partition deformation processing module for traversing the Gaussian points in the canonical space Gaussian point cloud, determining a matched motion encoder from a motion dictionary using the semantic label of a current Gaussian point, and encoding the current Gaussian point using the matched motion encoder to obtain the deformation of the current Gaussian point; A fine-tuning module for fine-tuning the appearance attributes of the deformed canonical space Gaussian point cloud in an end-to-end manner after obtaining the deformation of each Gaussian point in the canonical space Gaussian point cloud; A video generation module for generating a volumetric video corresponding to the target by exporting each frame of Gaussian point cloud after fine-tuning. The motion dictionary includes multiple independent motion encoders corresponding to different semantic categories; each independent motion encoder includes a multi-resolution hash coding network and a multi-layer perception network; the independent motion encoder takes the canonical position and time of a Gaussian point as input and outputs the deformation of the Gaussian point; the independent motion encoders for different semantic categories have different network configurations, specifically including: the resolution of the multi-resolution hash coding network used by the independent motion encoder corresponding to the clothing semantic category is higher than that of the independent motion encoder corresponding to the body semantic category, and the multi-layer perception network used by the former is deeper or wider than that of the latter.

9. An electronic device, comprising: Comprising: A memory for storing a computer program; A processor for executing the computer program to implement the steps of the volumetric video reconstruction method of any one of claims 1 to 7.

10. A readable storage medium, characterized by, The readable storage medium stores a computer program, which is executed by a processor to implement the steps of the volumetric video reconstruction method of any one of claims 1 to 7.

Images