Space-time continuous dynamic human body reconstruction method and device, electronic equipment and storage medium

Abstract

The invention provides a space-time continuous dynamic human body reconstruction method and device, electronic equipment and a storage medium, and the method comprises the steps: collecting and processing multi-view video data, and obtaining a human body initial position and time sequence information corresponding to an initial frame picture; inputting a pre-trained time continuity module to obtain a target frame position parameter rough value; inputting the position parameter rough value into a polynomial compensation module to obtain a target frame position parameter fine value; inputting the target frame position parameter rough value and the time sequence information into a spatial-temporal feature coding network, and decoding to generate a target frame spherical three-dimensional Gaussian scale parameter value and an opacity parameter value; the method comprises the following steps: acquiring a spherical three-dimensional Gaussian color parameter value of a target frame, inputting a position parameter fine value, a scale parameter value, an opacity parameter value and a color parameter value into a renderer, rendering to obtain a picture of a target visual angle, and through a time continuous module and parameterized representation, ensuring that a model parameter quantity does not linearly increase along with a frame number, space-time continuity exists between frames, and the rendering effect is improved.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and artificial intelligence, and in particular to a spatiotemporal continuous dynamic human body reconstruction method, apparatus, electronic device and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence and computer graphics technologies, dynamic human body reconstruction technology has been widely applied in fields such as virtual digital human content creation, film and television special effects, and virtual reality. Traditional human body reconstruction methods are usually based on three-dimensional Gaussian representation, using an ellipsoidal three-dimensional Gaussian group to represent the target scene. However, this method has significant limitations when applied to dynamic human body rendering. First, the parameters in the dynamic human body rendering model increase linearly with the number of frames, leading to a sharp increase in storage space overhead and making it difficult to efficiently process long sequences of dynamic scenes. Second, existing dynamic representation methods typically decompose dynamic scenes into frames of static scenes, resulting in a lack of spatiotemporal continuity between frames and causing problems such as color jumps in the rendering results. Summary of the Invention

[0003] This invention provides a spatiotemporally continuous dynamic human body reconstruction method, device, electronic device, and storage medium to solve the defects of traditional human body reconstruction methods, such as the linear increase of parameters in the dynamic human body rendering model with the number of frames, which leads to a sharp increase in storage space consumption, and the lack of spatiotemporal continuity between frames, which makes the rendering results prone to color jumps.

[0004] This invention provides a spatiotemporal continuous dynamic human body reconstruction method, comprising: Collect and process multi-view video data to obtain the initial position and temporal information of the human body corresponding to the initial frame image; The initial human body position and target frame time corresponding to the initial frame image are input into a pre-trained time-continuous module to obtain the rough value of the position parameter of the target frame spherical three-dimensional Gaussian. The rough values ​​of the position parameters of the target frame spherical three-dimensional Gaussian are input into the polynomial compensation module to obtain the fine values ​​of the position parameters of the target frame spherical three-dimensional Gaussian. The position parameter coarseness value of the target frame spherical three-dimensional Gaussian and the temporal information are input into the spatiotemporal feature encoding network to decode and generate the scale parameter value and opacity parameter value of the target frame spherical three-dimensional Gaussian; A learnable color feature vector is established, and the learnable color feature vector is superimposed with the temporal information and directional information to obtain the color parameter value of the target frame spherical three-dimensional Gaussian. The fine values ​​of the position parameters, scale parameters, opacity parameters, and color parameters of the target frame spherical 3D Gaussian are input into the renderer to render an image from the target viewpoint.

[0005] According to the spatiotemporal continuous dynamic human body reconstruction method provided by the present invention, the processing of multi-view video data includes: The open-source pose estimation method is used to identify the SMPL-X model parameters of the objects collected in each frame, and the mesh vertices of the SMPL-X model are calculated based on the SMPL-X model parameters. According to the spatiotemporal continuous dynamic human body reconstruction method provided by the present invention, the training method of the time continuous module includes: Construct a multilayer perceptron network; The coarse point cloud of frame 0 Mesh vertices of the SMPL-X model The input to the multilayer perceptron network is a deformed point cloud. and vertex Using the Chamfer distance loss function To measure and Overall similarity between them, and through the isometric loss function To maintain the relative position of the point cloud during the deformation process; Using the mesh vertex pairing information of the SMPL-X model as auxiliary supervision, through the vertex loss function... minimize and Differences are identified to ensure that the model can accurately capture large-scale limb movements of the human body; The final loss function is calculated as follows: : The network parameters are optimized using the backpropagation algorithm until the final loss function converges.

[0006] According to the spatiotemporal continuous dynamic human body reconstruction method provided by the present invention, the step of inputting the initial human body position corresponding to the initial frame image and the target frame time into a pre-trained temporal continuous module to obtain the coarse value of the spherical three-dimensional Gaussian position parameters of the target frame includes: Construct a 4-dimensional input vector, where 3 dimensions are the spherical 3D Gaussian position coordinates of the initial frame and 1 dimension is the time parameter of the target frame; The input vector is input into a pre-trained time-continuous module, the network structure of which includes a 4-dimensional input layer and a 12-dimensional output layer. The output vector of the output layer is analyzed, where the 9-dimensional vector is recombined into a 3×3 affine transformation weight matrix, and the 3-dimensional vector is used as the affine transformation bias vector. Calculate the coarse position parameters of the target frame using the affine transformation formula: in, and This represents the initial point position and time parameters. The point position parameter represents the target frame. and These represent the weights and biases of the affine transformation.

[0007] According to the spatiotemporal continuous dynamic human body reconstruction method provided by the present invention, the step of inputting the coarse value of the position parameter of the target frame spherical three-dimensional Gaussian into the polynomial compensation module to obtain the fine value of the position parameter of the target frame spherical three-dimensional Gaussian includes: Construct a learnable polynomial compensation function based on time variables: in, and Let represent the learnable coefficients and bias parameters of the polynomial, respectively. express The number of midpoints, K is the polynomial order, and t∈[0,1] is the normalized time parameter; The polynomial compensation value is calculated based on the time-variable learnable polynomial compensation function; The rough value of the position parameter is added to the polynomial compensation value to obtain the fine value of the position parameter of the spherical three-dimensional Gaussian of the target frame.

[0008] According to the spatiotemporal continuous dynamic human body reconstruction method provided by the present invention, the step of inputting the coarseness value of the position parameter of the spherical three-dimensional Gaussian of the target frame and the temporal information into a spatiotemporal feature encoding network, and decoding to generate the scale parameter value and opacity parameter value of the spherical three-dimensional Gaussian of the target frame, includes: Construct a four-dimensional spatiotemporal encoded input vector, which includes the coarse value of the position parameter of the target frame's spherical three-dimensional Gaussian and a one-dimensional time parameter; The four-dimensional spatiotemporal encoded input vector is feature encoded using a K-Planes structure, establishing six feature planes. Each plane learns a 16-dimensional feature vector, and the six plane features are concatenated into a 96-dimensional fusion feature. The fused features are input into a single-layer MLP decoder, which includes a two-dimensional output layer. The first-dimensional output layer is used to output the scale parameter, and the second-dimensional output layer is used to output the opacity parameter.

[0009] According to the spatiotemporal continuous dynamic human body reconstruction method provided by the present invention, the step of establishing a learnable color feature vector and superimposing the learnable color feature vector with the temporal information and directional information to obtain the color parameter values ​​of the target frame in a spherical three-dimensional Gaussian shape includes: The color parameters of a spherical three-dimensional Gaussian are represented as a learnable color feature vector with respect to time and viewpoint. The learnable color feature vector includes the base color, viewpoint color information, and time color information. The time color information is superimposed with the time sequence information, and the viewpoint color information is superimposed with the direction information. Based on the base color, the color parameter value of the target frame spherical three-dimensional Gaussian is obtained.

[0010] This invention provides a dynamic human body reconstruction device, comprising: The acquisition module is used to acquire and process multi-view video data, and obtain the initial position and timing information of the human body corresponding to the initial frame image; The first input module is used to input the initial human body position and target frame time corresponding to the initial frame image into the pre-trained time continuity module to obtain the rough value of the position parameter of the target frame spherical three-dimensional Gaussian. The second input module is used to input the coarse value of the position parameter of the target frame spherical three-dimensional Gaussian into the polynomial compensation module to obtain the fine value of the position parameter of the target frame spherical three-dimensional Gaussian; The decoding module is used to input the position parameter coarseness value of the target frame spherical three-dimensional Gaussian and the temporal information into the spatiotemporal feature coding network, and decode to generate the scale parameter value and opacity parameter value of the target frame spherical three-dimensional Gaussian; The overlay module is used to establish a learnable color feature vector, and overlay the learnable color feature vector with the temporal information and directional information to obtain the color parameter value of the target frame spherical three-dimensional Gaussian. The rendering module is used to input the fine values ​​of the position parameters, scale parameters, opacity parameters, and color parameters of the target frame's spherical 3D Gaussian shape into the renderer to render an image from the target viewpoint.

[0011] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the spatiotemporal continuous dynamic human body reconstruction method as described in any of the preceding claims.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the spatiotemporal continuous dynamic human body reconstruction method described above.

[0013] The spatiotemporal continuous dynamic human body reconstruction method, apparatus, electronic device, and storage medium provided by this invention acquire and process multi-view video data to obtain the initial human body position and temporal information corresponding to the initial frame image; input the initial human body position and target frame time corresponding to the initial frame image into a pre-trained temporal continuity module to obtain a coarse value of the target frame's spherical three-dimensional Gaussian position parameters; input the coarse value of the target frame's spherical three-dimensional Gaussian position parameters into a polynomial compensation module to obtain a fine value of the target frame's spherical three-dimensional Gaussian position parameters; input the coarse value of the target frame's spherical three-dimensional Gaussian position parameters and the temporal information into a spatiotemporal feature encoding network to decode and generate the target... The scale and opacity parameters of the target frame's spherical 3D Gaussian are obtained. A learnable color feature vector is established, and the learnable color feature vector is superimposed with the temporal and directional information to obtain the color parameter value of the target frame's spherical 3D Gaussian. The fine-grained position parameter value, scale parameter value, opacity parameter value, and color parameter value of the target frame's spherical 3D Gaussian are input into the renderer to render an image from the target viewpoint. Through the temporal continuity module and parameterized representation, it can be ensured that the number of model parameters does not increase linearly with the number of frames, and there is spatiotemporal continuity between frames, thus improving the rendering effect. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0015] Figure 1 This is one of the flowcharts of the spatiotemporal continuous dynamic human body reconstruction method provided in the embodiments of the present invention; Figure 2 This is a schematic diagram of the time continuity module structure provided in an embodiment of the present invention; Figure 3 This is the second flowchart of the spatiotemporal continuous dynamic human body reconstruction method provided in the embodiments of the present invention; Figure 4 This is a schematic diagram of the functional structure of the dynamic human body reconstruction device provided in an embodiment of the present invention; Figure 5 This is a functional structure diagram of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0017] Figure 1 The flowchart of the spatiotemporal continuous dynamic human body reconstruction method provided in the embodiments of the present invention is as follows: Figure 1 As shown, the spatiotemporal continuous dynamic human body reconstruction method provided in this embodiment of the invention includes: Step 101: Collect and process multi-view video data to obtain the initial position and temporal information of the human body corresponding to the initial frame image; In this embodiment of the invention, video data of a specific person is acquired using a camera array. All cameras are arranged at equal intervals around the subject on a spherical or cylindrical surface and uniformly calibrated to obtain the camera's intrinsic and extrinsic parameters. The number of cameras... The camera angle should be 48-160mm to ensure all details of the subject's entire body are captured. The camera's field of view should remain constant throughout the capture process. The video resolution is 2448 x 2048 pixels, with a frame rate of 15 frames per second. Subjects can wear various clothing, including everyday clothes, traditional ethnic costumes, and ancient costumes, and can perform large-scale body movements such as dancing, martial arts, and sports. Data capture duration should be 10 seconds or more (frame count N>=150).

[0018] In this embodiment of the invention, processing multi-view video data includes using an open-source pose estimation method to identify the SMPL-X model parameters of the acquired objects in each frame, and calculating the mesh vertices of the SMPL-X model based on the SMPL-X model parameters, specifically including: Acquire motion video sequences, and extract the image corresponding to the video frame t of the i-th camera frame by frame. The open-source object segmentation model SAM2 is used to obtain the mask of the captured object corresponding to each video frame image. The mask dataset is obtained by organizing the data frame by frame. . use Will Process into images containing only the captured objects The image dataset is obtained by organizing the images frame by frame. .

[0019] For video frame image data The open-source pose estimation method SMPLify-X was used to identify the SMPL-X model parameters of the captured objects in each video frame. It contains three parameters, among which... , , These represent the pose parameters, shape parameters, and facial expression parameters of the corresponding video frame of the captured object, respectively. The pose parameters can be further divided into overall pose parameters. Head posture parameters Hand posture parameters and root node parameters .

[0020] Multi-view video data of the action sequences of the acquired objects can be collected and processed to obtain images, masks, mesh vertices of the SMPL-X model, coarse point clouds, and other related data for each video frame.

[0021] Step 102: Input the initial human body position and target frame time corresponding to the initial frame image into the pre-trained time continuity module to obtain the rough value of the position parameters of the target frame spherical three-dimensional Gaussian. In this embodiment of the invention, the time continuity module is used to capture the changes in the movement of the entire body of the subject over time.

[0022] Step 103: Input the rough value of the position parameter of the target frame spherical three-dimensional Gaussian into the polynomial compensation module to obtain the fine value of the position parameter of the target frame spherical three-dimensional Gaussian; Step 104: Input the position parameter coarseness value of the target frame spherical three-dimensional Gaussian and the temporal information into the spatiotemporal feature coding network to decode and generate the scale parameter value and opacity parameter value of the target frame spherical three-dimensional Gaussian; Step 105: Establish a learnable color feature vector, and superimpose the learnable color feature vector with the temporal information and directional information to obtain the color parameter value of the target frame spherical three-dimensional Gaussian. Step 106: Input the fine values ​​of the position parameters, scale parameters, opacity parameters, and color parameters of the target frame spherical 3D Gaussian into the renderer to render an image from the target viewpoint.

[0023] In traditional dynamic human rendering models, parameters increase linearly with the number of frames, leading to a sharp increase in storage space overhead and making it difficult to efficiently handle long sequences of dynamic scenes. Secondly, existing dynamic representation methods typically decompose dynamic scenes into frames of static scenes, resulting in a lack of spatiotemporal continuity between frames and causing problems such as color jumps in the rendering results.

[0024] The spatiotemporal continuous dynamic human body reconstruction method provided in this invention collects and processes multi-view video data to obtain the initial human body position and temporal information corresponding to the initial frame image; inputs the initial human body position and target frame time corresponding to the initial frame image into a pre-trained temporal continuity module to obtain the coarse value of the position parameters of the target frame's spherical 3D Gaussian; inputs the coarse value of the position parameters of the target frame's spherical 3D Gaussian into a polynomial compensation module to obtain the fine value of the position parameters of the target frame's spherical 3D Gaussian; inputs the coarse value of the position parameters of the target frame's spherical 3D Gaussian and the temporal information into a spatiotemporal feature encoding network to decode and generate the scale parameter value and opacity parameter value of the target frame's spherical 3D Gaussian; obtains the color parameter value of the target frame's spherical 3D Gaussian; inputs the fine value of the position parameters, the scale parameter value, the opacity parameter value, and the color parameter value of the target frame's spherical 3D Gaussian into a renderer to render an image from the target viewpoint. Through the temporal continuity module and parameterized representation, it can be ensured that the number of model parameters does not increase linearly with the number of frames, and there is spatiotemporal continuity between frames, thus improving the rendering effect.

[0025] Based on any of the above embodiments, the time-continuous module structure provided by the embodiments of the present invention is as follows: Figure 2 As shown, the training method for the time-continuous module includes: Step 201: Construct a multilayer perceptron network; The embodiments of the present invention obtain the SMPL-X parameters of the t-th frame. Linear blending skinning is performed to obtain the mesh vertex representation of the corresponding SMPL-X model. ( (Represents the number of vertices). The Neus2 static scene reconstruction method is used to obtain a coarse point cloud of the acquired object. ( (This represents the number of point clouds).

[0026] In this embodiment of the invention, the multilayer perceptron network has a 4-dimensional input, 3 intermediate layers with a total of 128 dimensions, and a 12-dimensional output. The activation function between layers is a sinusoidal activation function. The first 3 dimensions of the input represent the initial point location parameter values, and the 4th dimension represents the time parameter values. The first 9 dimensions of the output represent the weighted output of the affine transformation, and the last 3 dimensions represent the biased output of the affine transformation.

[0027] Step 202: Extract the coarse point cloud from frame 0. Mesh vertices of the SMPL-X model The input to the multilayer perceptron network is a deformed point cloud. and vertex Step 203: Use the Chamfer distance loss function To measure and Overall similarity between them, and through the isometric loss function To maintain the relative position of the point cloud during the deformation process; Step 204: Utilize the pairing information of the mesh vertices in the SMPL-X model as auxiliary supervision, through the vertex loss function... minimize and Differences are identified to ensure that the model can accurately capture large-scale limb movements of the human body; Step 205: Calculate the final loss function. : Step 206: Optimize the network parameters using the backpropagation algorithm until the final loss function converges.

[0028] In this embodiment of the invention, to estimate the point cloud position of the target frame based on the point cloud position of the input initial frame, the training data for the constructed temporal continuous module is the mesh vertices of the acquired SMPL-X model. and coarse point clouds The module's input is the coarse point cloud of frame 0. Mesh vertices of the SMPL-X model The output is the deformed point cloud. and vertex The goal of the time-continuous module is to minimize the point cloud. and ,vertex and The differences between them are used to capture changes in human movement.

[0029] Based on point cloud and ,vertex and Using the Chamfer distance loss function To measure and Overall similarity between them, and through the isometric loss function To maintain the relative positions of the point cloud during deformation, the pairing information of the mesh vertices in the SMPL-X model is used as auxiliary supervision to further optimize the temporally continuous module. This is achieved through a vertex loss function. minimize and The differences are identified to ensure the model can accurately capture large-scale limb movements of the human body. The final loss function is... Hyperparameters are set according to the specific experimental conditions. By default, the hyperparameters... ,in Indicates the current training epoch. (The default value is 2000) represents the total number of training rounds.

[0030] Based on the final loss function Gradient backpropagation is performed to optimize the parameters of the time-continuous module, ultimately enabling the depiction of the motion changes of the entire body of the acquired object over time.

[0031] Based on any of the above embodiments, the step of inputting the initial human body position corresponding to the initial frame image and the target frame time into a pre-trained time-continuous module to obtain the rough value of the spherical three-dimensional Gaussian position parameters of the target frame includes: Step 301: Construct a 4-dimensional input vector, where 3 dimensions are the spherical 3D Gaussian position coordinates of the initial frame and 1 dimension is the time parameter of the target frame; Step 302: Input the input vector into a pre-trained time continuous module. The network structure of the time continuous module includes a 4-dimensional input layer and a 12-dimensional output layer. Step 303: Analyze the output vector of the output layer, where the 9-dimensional vector is recombined into a 3×3 affine transformation weight matrix, and the 3-dimensional vector is used as the affine transformation bias vector; Step 304: Calculate the coarse position parameters of the target frame using the affine transformation formula: in, and This represents the initial point position and time parameters. The point position parameter represents the target frame. and These represent the weights and biases of the affine transformation.

[0032] This time-continuous module predicts the point cloud position parameters of the target frame by adding the three-dimensional point cloud position parameter values ​​of the initial frame to the one-dimensional time parameter values ​​of the target frame. This change is represented as an affine transformation: in, and This represents the initial point position and time parameters. This represents the point position parameter of the target frame. and These represent the weights and biases of the affine transformation.

[0033] Based on any of the above embodiments, the step of inputting the coarse value of the position parameter of the target frame spherical three-dimensional Gaussian into the polynomial compensation module to obtain the fine value of the position parameter of the target frame spherical three-dimensional Gaussian includes: Step 401: Construct a learnable polynomial compensation function based on time variables: in, and Let represent the learnable coefficients and bias parameters of the polynomial, respectively. express The number of midpoints, K is the polynomial order, and t∈[0,1] is the normalized time parameter; Step 402: Calculate the polynomial compensation value based on the time-variable learnable polynomial compensation function; Step 403: Add the rough value of the position parameter to the polynomial compensation value to obtain the fine value of the position parameter of the target frame spherical three-dimensional Gaussian.

[0034] In this embodiment of the invention, the spherical three-dimensional Gaussian position parameters initial value Set as coarse point cloud Based on a pre-trained time-continuous module, the roughness values ​​of the position parameters of a spherical 3D Gaussian at time t are obtained. To overcome the smoothness of the implicit time-continuous module, a time-learnable polynomial compensation module is introduced for fine-tuning the Gaussian position. This polynomial compensation module takes the form: in, and Let represent the learnable coefficients and bias parameters of the polynomial, respectively. express The number of midpoints. Using a polynomial compensation module, the roughness value and the compensation value are added together to obtain the final spherical 3D Gaussian position parameters: .

[0035] Based on any of the above embodiments, the step of inputting the coarseness value of the position parameter of the spherical three-dimensional Gaussian of the target frame and the temporal information into a spatiotemporal feature encoding network to decode and generate the scale parameter value and opacity parameter value of the spherical three-dimensional Gaussian of the target frame includes: Step 501: Construct a four-dimensional spatiotemporal coding input vector, which includes the roughness value of the position parameter of the target frame's spherical three-dimensional Gaussian and a one-dimensional time parameter; Step 502: Use the K-Planes structure to perform feature encoding on the four-dimensional spatiotemporal encoded input vector, establish 6 feature planes, learn 16-dimensional feature vectors for each plane, and concatenate the 6 plane features into 96-dimensional fused features. Step 503: Input the fused features into a single-layer MLP decoder. The single-layer MLP decoder includes a two-dimensional output layer. The first-dimensional output layer is used to output the scale parameter, and the second-dimensional output layer is used to output the opacity parameter.

[0036] In this embodiment of the invention, the roughness value of the spherical three-dimensional Gaussian position parameter at time t is... The time and space information, denoted by t, are input into a four-dimensional feature grid module, which first encodes the temporal and spatial information into high-dimensional feature vectors. This four-dimensional feature grid adopts a K-Planes structure, which integrates four-dimensional spacetime. Decomposed into six planes: , , , , , The 16-dimensional feature vectors encoded by each feature plane are combined to obtain a 96-dimensional high-dimensional feature vector. The input is fed into a single-layer, 128-dimensional multilayer perceptron network, and the output is a 2-dimensional vector, where the first dimension is the scaling parameter of a spherical 3D Gaussian. The second dimension is the opacity parameter of a spherical three-dimensional Gaussian. .

[0037] Based on any of the above embodiments, the step of establishing a learnable color feature vector and superimposing the learnable color feature vector with the temporal information and directional information to obtain the color parameter value of the target frame's spherical three-dimensional Gaussian shape includes: The color parameters of a spherical three-dimensional Gaussian are represented as a learnable color feature vector with respect to time and viewpoint. The learnable color feature vector includes the base color, viewpoint color information, and time color information. The time color information is superimposed with the time sequence information, and the viewpoint color information is superimposed with the direction information. Based on the base color, the color parameter value of the target frame spherical three-dimensional Gaussian is obtained.

[0038] In this embodiment of the invention, the color parameters of the spherical three-dimensional Gaussian are... This can be represented as a learnable feature vector relating time and perspective: .

[0039] in, , , These represent the base color, viewpoint color information, and time-related color information, respectively. This is the deviation parameter.

[0040] The appearance of a traditional dynamic human body (such as clothing wrinkles, changes in lighting, skin tone, etc.) undergoes subtle changes over time. Traditional static color parameters struggle to model these time-varying characteristics. This invention provides a learnable feature vector related to time and perspective, enabling the model to adaptively learn the evolution of dynamic appearance (such as changes in clothing shadows caused by movement or the influence of ambient light).

[0041] In this embodiment of the invention, the fine values ​​of the position parameters, scale parameters, opacity parameters, and color parameters of the target frame's spherical three-dimensional Gaussian shape are input into the renderer to render an image from the target viewpoint, including: Step 601: Render the base color image, view color image, and time color image using the Gaussian renderer; Step 602: Establish a color decoding module of a 2-layer 128-dimensional multilayer perceptron network to decode the basic color image, viewpoint color image, and time color image into the final complete image.

[0042] The embodiments of the present invention will , , and The input is fed into a spherical 3D Gaussian splash renderer, where a hardware-accelerated splash algorithm renders the base color image. Viewpoint color image Time-color image And additionally obtain the rendering mask. .

[0043] Establish a color decoding module using a 2-layer, 128-dimensional multilayer perceptron network. ,Will , and Decoded into the final complete image: , where r represents viewpoint information.

[0044] Using MSE loss Perceived loss ( (representing the pre-trained VGG model) and mask loss By minimizing the rendered image With real images To address the differences between the parameters, the polynomial compensation module, the four-dimensional feature mesh module, and the color parameter module are optimized, ensuring high fidelity in the rendering results. The total loss function is defined as: Hyperparameters are set according to the specific experimental conditions. By default, the hyperparameters... , .

[0045] Based on any of the above embodiments, the dynamic human reconstruction architecture based on spherical three-dimensional Gaussian representation and time-continuous modules provided by the embodiments of the present invention is as follows: Figure 3 As shown. First, the multi-view video is input into the data processing module to obtain the relevant dataset. Second, the model training is divided into two stages. The first stage uses coarse point clouds and the mesh vertices of the SMPL-X model to train the temporal continuity module. The second stage consists of several modules: using a multinomial compensation module to obtain the position parameters of the spherical 3D Gaussian; using a four-dimensional feature mesh module to obtain the scale and opacity parameters of the spherical 3D Gaussian; the color parameter module is represented by learnable feature vectors with respect to time and viewpoint; and the obtained spherical 3D Gaussian parameter values ​​are rendered using a Gaussian renderer to obtain the image.

[0046] Compared to 4K4D (a film and television technology that combines 4K ultra-high-definition resolution with a 4D movie experience), this invention significantly reduces the number of model parameters while maintaining the same rendering quality. First, this invention eliminates the need to store the position parameters of the spherical 3D Gaussian sphere for each frame; it only needs to save the position parameters of the initial frame's spherical 3D Gaussian sphere, using a compact temporal continuity module and a polynomial compensation module to represent the position parameters of the target frame's spherical 3D Gaussian sphere. Second, this invention represents the color parameters of the spherical 3D Gaussian sphere as a learnable feature vector related to time and viewpoint, eliminating the need to load an image dataset or assign a color parameter value to each spatiotemporal spherical 3D Gaussian sphere.

[0047] Furthermore, the embodiments of the present invention also possess temporal continuity. Existing dynamic human reconstruction work based on 3D Gaussian and 4K4D both represent dynamic human scenes as static scenes frame by frame. Therefore, the rendering results inevitably exhibit color jumps. However, the embodiments of the present invention use a temporally continuous module to represent dynamic human scenes, thus possessing temporal continuity, and the rendering results do not exhibit color jumps.

[0048] The spatiotemporally continuous dynamic human body reconstruction method provided in this invention introduces a series of modules to represent the position, scale, opacity, color, and other parameters of the spherical three-dimensional Gaussian shape of the target frame. Therefore, this invention can significantly reduce the number of model parameters while maintaining the same rendering quality. Existing dynamic human body rendering models based on three-dimensional Gaussians suffer from insufficient spatiotemporal continuity; they construct a continuous representation, thus representing a dynamic human scene that is temporally continuous.

[0049] The dynamic human body reconstruction device provided by the present invention is described below. The dynamic human body reconstruction device described below can be referred to in correspondence with the spatiotemporal continuous dynamic human body reconstruction method described above.

[0050] Figure 4 This is a schematic diagram of the structure of the dynamic human body reconstruction device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the dynamic human body reconstruction device provided in this embodiment of the invention includes: The acquisition module 401 is used to acquire and process multi-view video data and obtain the initial position and timing information of the human body corresponding to the initial frame image. The first input module 402 is used to input the initial position of the human body and the target frame time corresponding to the initial frame image into the pre-trained time continuity module to obtain the rough value of the position parameter of the target frame spherical three-dimensional Gaussian. The second input module 403 is used to input the rough value of the position parameter of the target frame spherical three-dimensional Gaussian into the polynomial compensation module to obtain the fine value of the position parameter of the target frame spherical three-dimensional Gaussian. The decoding module 404 is used to input the position parameter coarseness value of the target frame spherical three-dimensional Gaussian and the temporal information into the spatiotemporal feature coding network, and decode to generate the scale parameter value and opacity parameter value of the target frame spherical three-dimensional Gaussian; The overlay module 405 is used to establish a learnable color feature vector and overlay the learnable color feature vector with the temporal information and directional information to obtain the color parameter value of the target frame spherical three-dimensional Gaussian. The rendering module 406 is used to input the fine values ​​of the position parameters, scale parameters, opacity parameters, and color parameters of the target frame spherical three-dimensional Gaussian into the renderer to render an image from the target viewpoint.

[0051] The dynamic human body reconstruction device provided in this embodiment of the invention acquires and processes multi-view video data to obtain the initial human body position and temporal information corresponding to the initial frame image; inputs the initial human body position and target frame time corresponding to the initial frame image into a pre-trained temporal continuity module to obtain a coarse value of the position parameters of the target frame's spherical three-dimensional Gaussian; inputs the coarse value of the position parameters of the target frame's spherical three-dimensional Gaussian into a polynomial compensation module to obtain a fine value of the position parameters of the target frame's spherical three-dimensional Gaussian; inputs the coarse value of the position parameters of the target frame's spherical three-dimensional Gaussian and the temporal information into a spatiotemporal feature encoding network to decode and generate the scale parameter value and opacity parameter value of the target frame's spherical three-dimensional Gaussian; obtains the color parameter value of the target frame's spherical three-dimensional Gaussian; inputs the fine value of the position parameters, the scale parameter value, the opacity parameter value, and the color parameter value of the target frame's spherical three-dimensional Gaussian into a renderer to render an image from the target viewpoint. Through the temporal continuity module and parameterized representation, it can ensure that the number of model parameters does not increase linearly with the number of frames, and that there is spatiotemporal continuity between frames, thereby improving the rendering effect.

[0052] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540. The processor 510, communication interface 520, and memory 530 communicate with each other via the communication bus 540. The memory 530 includes computer programs, an operating system, and acquired data. The processor 510 can call logical instructions in the memory 530 to execute a spatiotemporal continuous dynamic human body reconstruction method. This method includes: acquiring and processing multi-view video data to obtain the initial human body position and temporal information corresponding to the initial frame image; inputting the initial human body position and target frame time corresponding to the initial frame image into a pre-trained temporal continuity module to obtain a rough value of the target frame's spherical three-dimensional Gaussian position parameters; and inputting the rough value of the target frame's spherical three-dimensional Gaussian position parameters into a polynomial compensation module to obtain a refined value of the target frame's spherical three-dimensional Gaussian position parameters. Fine-grained values ​​are obtained by inputting the coarse position parameters of the target frame's spherical 3D Gaussian sphere and the temporal information into a spatiotemporal feature encoding network to decode and generate the scale and opacity parameters of the target frame's spherical 3D Gaussian sphere. A learnable color feature vector is established, and the learnable color feature vector is superimposed with the temporal and directional information to obtain the color parameter values ​​of the target frame's spherical 3D Gaussian sphere. The fine-grained position parameters, scale and opacity parameters, and color parameters of the target frame's spherical 3D Gaussian sphere are then input into a renderer to render an image from the target viewpoint.

[0053] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0054] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the spatiotemporal continuous dynamic human body reconstruction method provided by the above methods. This method includes: acquiring and processing multi-view video data to obtain the initial human body position and temporal information corresponding to an initial frame image; inputting the initial human body position and target frame time corresponding to the initial frame image into a pre-trained temporal continuity module to obtain a rough value of the position parameters of the target frame's spherical three-dimensional Gaussian; inputting the rough value of the position parameters of the target frame's spherical three-dimensional Gaussian into a polynomial compensation module to obtain the position parameters of the target frame's spherical three-dimensional Gaussian. The fine-grained values ​​of the position parameters of the target frame's spherical 3D Gaussian sphere and the temporal information are input into a spatiotemporal feature encoding network to decode and generate the scale parameter value and opacity parameter value of the target frame's spherical 3D Gaussian sphere. A learnable color feature vector is established, and the learnable color feature vector is superimposed with the temporal information and orientation information to obtain the color parameter value of the target frame's spherical 3D Gaussian sphere. The fine-grained values ​​of the position parameters, scale parameter value, opacity parameter value, and color parameter value of the target frame's spherical 3D Gaussian sphere are input into a renderer to render an image from the target viewpoint.

[0055] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0056] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A spatiotemporally continuous dynamic human body reconstruction method, characterized in that, include: Collect and process multi-view video data to obtain the initial position and temporal information of the human body corresponding to the initial frame image; The initial human body position and target frame time corresponding to the initial frame image are input into a pre-trained time-continuous module to obtain the rough value of the position parameter of the target frame spherical three-dimensional Gaussian. The rough values ​​of the position parameters of the target frame spherical three-dimensional Gaussian are input into the polynomial compensation module to obtain the fine values ​​of the position parameters of the target frame spherical three-dimensional Gaussian. The position parameter coarseness value of the target frame spherical three-dimensional Gaussian and the temporal information are input into the spatiotemporal feature encoding network to decode and generate the scale parameter value and opacity parameter value of the target frame spherical three-dimensional Gaussian; A learnable color feature vector is established, and the learnable color feature vector is superimposed with the temporal information and directional information to obtain the color parameter value of the target frame spherical three-dimensional Gaussian. The fine values ​​of the position parameters, scale parameters, opacity parameters, and color parameters of the target frame spherical 3D Gaussian are input into the renderer to render an image from the target viewpoint.

2. The spatiotemporal continuous dynamic human body reconstruction method according to claim 1, characterized in that, The processing of multi-view video data includes: The open-source pose estimation method is used to identify the SMPL-X model parameters of the objects collected in each frame, and the mesh vertices of the SMPL-X model are calculated based on the SMPL-X model parameters.

3. The spatiotemporal continuous dynamic human body reconstruction method according to claim 2, characterized in that, The training method for the time-continuous module includes: Construct a multilayer perceptron network; The coarse point cloud of frame 0 Mesh vertices of the SMPL-X model The input to the multilayer perceptron network is a deformed point cloud. and vertex Using the Chamfer distance loss function To measure and Overall similarity between them, and through the isometric loss function To maintain the relative position of the point cloud during the deformation process; Using the mesh vertex pairing information of the SMPL-X model as auxiliary supervision, through the vertex loss function... minimize and Differences are identified to ensure that the model can accurately capture large-scale limb movements of the human body; The final loss function is calculated as follows: ; The network parameters are optimized using the backpropagation algorithm until the final loss function output converges.

4. The spatiotemporal continuous dynamic human body reconstruction method according to claim 3, characterized in that, The step of inputting the initial human body position corresponding to the initial frame image and the target frame time into a pre-trained time-continuous module to obtain the rough value of the spherical three-dimensional Gaussian position parameters of the target frame includes: Construct a 4-dimensional input vector, where 3 dimensions are the spherical 3D Gaussian position coordinates of the initial frame and 1 dimension is the time parameter of the target frame; The input vector is input into a pre-trained time-continuous module, the network structure of which includes a 4-dimensional input layer and a 12-dimensional output layer. The output vector of the output layer is analyzed, where the 9-dimensional vector is recombined into a 3×3 affine transformation weight matrix, and the 3-dimensional vector is used as the affine transformation bias vector. Calculate the coarse position parameters of the target frame using the affine transformation formula: ； in, and This represents the initial point position and time parameters. The point position parameter represents the target frame. and These represent the weights and biases of the affine transformation.

5. The spatiotemporal continuous dynamic human body reconstruction method according to claim 1, characterized in that, The step of inputting the coarse value of the position parameter of the target frame spherical three-dimensional Gaussian into the polynomial compensation module to obtain the fine value of the position parameter of the target frame spherical three-dimensional Gaussian includes: Construct a learnable polynomial compensation function based on time variables: ； in, and Let represent the learnable coefficients and bias parameters of the polynomial, respectively. express The number of midpoints, K is the polynomial order, and t∈[0,1] is the normalized time parameter; The polynomial compensation value is calculated based on the time-variable learnable polynomial compensation function; The rough value of the position parameter is added to the polynomial compensation value to obtain the fine value of the position parameter of the spherical three-dimensional Gaussian of the target frame.

6. The spatiotemporal continuous dynamic human body reconstruction method according to claim 1, characterized in that, The step of inputting the coarseness value of the position parameter of the spherical three-dimensional Gaussian of the target frame and the temporal information into a spatiotemporal feature encoding network to decode and generate the scale parameter value and opacity parameter value of the spherical three-dimensional Gaussian of the target frame includes: Construct a four-dimensional spatiotemporal encoded input vector, which includes the coarse value of the position parameter of the target frame's spherical three-dimensional Gaussian and a one-dimensional time parameter; The four-dimensional spatiotemporal encoded input vector is feature encoded using a K-Planes structure, establishing six feature planes. Each plane learns a 16-dimensional feature vector, and the six plane features are concatenated into a 96-dimensional fusion feature. The fused features are input into a single-layer MLP decoder, which includes a two-dimensional output layer. The first-dimensional output layer is used to output the scale parameter, and the second-dimensional output layer is used to output the opacity parameter.

7. The spatiotemporal continuous dynamic human body reconstruction method according to claim 1, characterized in that, The step of establishing a learnable color feature vector and superimposing the learnable color feature vector with the temporal information and directional information to obtain the color parameter values ​​of the target frame's spherical three-dimensional Gaussian shape includes: The color parameters of a spherical three-dimensional Gaussian are represented as a learnable color feature vector with respect to time and viewpoint. The learnable color feature vector includes the base color, viewpoint color information, and time color information. The time color information is superimposed with the time sequence information, and the viewpoint color information is superimposed with the direction information. Based on the base color, the color parameter value of the target frame spherical three-dimensional Gaussian is obtained.

8. A dynamic human body reconstruction device, characterized in that, include: The acquisition module is used to acquire and process multi-view video data, and obtain the initial position and timing information of the human body corresponding to the initial frame image; The first input module is used to input the initial human body position and target frame time corresponding to the initial frame image into the pre-trained time continuity module to obtain the rough value of the position parameter of the target frame spherical three-dimensional Gaussian. The second input module is used to input the coarse value of the position parameter of the target frame spherical three-dimensional Gaussian into the polynomial compensation module to obtain the fine value of the position parameter of the target frame spherical three-dimensional Gaussian; The decoding module is used to input the position parameter coarseness value of the target frame spherical three-dimensional Gaussian and the temporal information into the spatiotemporal feature coding network, and decode to generate the scale parameter value and opacity parameter value of the target frame spherical three-dimensional Gaussian; The overlay module is used to establish a learnable color feature vector, and overlay the learnable color feature vector with the temporal information and directional information to obtain the color parameter value of the target frame spherical three-dimensional Gaussian. The rendering module is used to input the fine values ​​of the position parameters, scale parameters, opacity parameters, and color parameters of the target frame's spherical 3D Gaussian shape into the renderer to render an image from the target viewpoint.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the spatiotemporal continuous dynamic human body reconstruction method as described in any one of claims 1 to 7.

10. A non-transitory readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the spatiotemporal continuous dynamic human body reconstruction method as described in any one of claims 1 to 7.

Images