Reconstruction and driving method based on multi-view motion video of clothed human body

Abstract

Provided in the present invention is a reconstruction and driving method based on a multi-view motion video of a clothed human body. The method comprises: using a two-layer three-dimensional Gaussian representation to model a human body and corresponding clothes; defining geometric attributes of three-dimensional Gaussian ellipsoids for any face on coarse three-dimensional meshes of the human body and the clothes; at a reconstruction stage, transforming coarse triangular meshes of the clothes and the body from a canonical space to an observation space, and updating Gaussian geometric attributes; and on the basis of a camera pose and parameters of an input video, rasterizing a three-dimensional Gaussian representation of the observation space to an image space, and rendering a picture on the basis of a Gaussian splatting formula. Self-supervised training can be performed on a model by means of calculating the color difference between a rendered picture and a real picture and using same as an optimization objective; when reconstruction is completed and driving is performed, a geometric result of a current frame is used to calculate the position of the coarse triangular mesh of the clothes in the next frame; and the reconstruction-stage method and related parameters are used to render a driving picture in a new posture.

Description

A reconstruction and driving method based on multi-view clothed human motion video TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to the technical field of multi-view three-dimensional motion video processing, and more particularly to a reconstruction and driving method based on multi-view clothed human motion video. BACKGROUND

[0002] The reconstruction and driving of dynamic three-dimensional clothed human is an important research direction in computer vision. It has wide applications in many fields, such as live broadcast, video conference and VR / AR (Virtual Reality: virtual reality / Augmented Reality: augmented reality). Although this direction has been studied for a long time, it is still challenging due to the difficulty in predicting the non-rigid deformation of clothes under the driven pose. The key technical problem is how to accurately simulate loose clothes and generate clothes motion that conforms to the kinematics.

[0003] Traditional clothed human reconstruction and driving methods often need to manually model the human body and clothes in detail, and use optimization-based physical simulation methods to drive the clothes. Since manual modeling requires a lot of manpower and time, and the final result is heavily dependent on the technical ability of the three-dimensional animation artist responsible for modeling. Some existing clothed human reconstruction and driving methods ignore the non-rigid deformation of clothes, resulting in distorted driving results. In addition, although the non-rigid deformation of clothes is considered in other clothed human reconstruction and driving methods, and good driving results are achieved based on a large number of training sets, these methods only rely on single-frame driving pose information, and will generate unreasonable driving results under human poses outside the training set, and further generate motion results that do not conform to the kinematics. SUMMARY

[0004] In view of the above problems, the present application provides a reconstruction and driving method based on multi-view clothed human motion video.

[0005] According to an embodiment of the present application, a reconstruction and driving method based on multi-view clothed human motion video is provided, the method comprising:

[0006] Obtaining camera parameters and human body pose information of a target person from a multi-view clothed human motion video, and segmenting the multi-view clothed human motion video to obtain a video segmentation result;

[0007] Obtaining a rough three-dimensional triangular mesh of the human body in a reference space, and obtaining a rough three-dimensional triangular mesh of the clothes in the reference space using the video segmentation result, the camera parameters, the rough three-dimensional triangular mesh of the human body, and the human body pose information;

[0008] Anchoring the human body three-dimensional Gaussian representation on the human body coarse three-dimensional triangle mesh and anchoring the clothes three-dimensional Gaussian representation on the clothes coarse three-dimensional triangle mesh;

[0009] Transforming the human body coarse three-dimensional triangle mesh and the human body three-dimensional Gaussian representation and the clothes coarse three-dimensional triangle mesh and the clothes three-dimensional Gaussian representation from a reference space to an observation space respectively;

[0010] Projecting the human body three-dimensional Gaussian representation and the clothes three-dimensional Gaussian representation in the observation space into two-dimensional Gaussian representations, and rendering the human body two-dimensional Gaussian representation and the clothes two-dimensional Gaussian representation;

[0011] Repeating the space transformation operation, the projection operation and the rendering operation, and optimizing the coarse three-dimensional triangle mesh and the three-dimensional Gaussian representation in a self-supervised manner to obtain a reconstruction result of a target person.

[0012] According to the embodiments of the present application, the above-mentioned reconstruction and driving method based on the multi-view clothed human motion video further comprises:

[0013] Driving the human body of the target person according to the input driving pose sequence to obtain a human body coarse three-dimensional triangle mesh at a driving time;

[0014] Driving the clothes of the target person based on the human body coarse three-dimensional triangle mesh at the driving time to obtain a clothes coarse three-dimensional triangle mesh at the driving time;

[0015] Updating the three-dimensional Gaussian representation corresponding to the human body coarse three-dimensional triangle mesh at the driving time and the clothes coarse three-dimensional triangle mesh at the driving time;

[0016] Projecting and rendering the updated three-dimensional Gaussian representation using the reconstruction result and the camera parameter to obtain a driving result of the target person.

[0017] According to the embodiments of the present application, the above-mentioned segmentation of the multi-view clothed human motion video to obtain a video segmentation result comprises:

[0018] Performing frame-by-frame camera parameter acquisition operation and target person human body motion capture operation on each view of the multi-view clothed human motion video using a predefined camera calibration and human body pose capture method, to obtain camera parameters of each frame image and human body pose information of the target person, wherein the predefined camera calibration and human body pose capture method comprises an EasyMocap method;

[0019] The pre-trained image segmentation large model is used for frame-by-frame segmentation of each view of the multi-view clothed human motion video, to obtain a video segmentation result of each frame of image, wherein the pre-trained image segmentation large model comprises a pre-trained SAM model, and the video segmentation result of each frame of image comprises a human body label and a clothing label of the target person.

[0020] According to the embodiment of the present application, the above-mentioned obtaining of the clothing coarse three-dimensional triangular mesh in the reference space by using the video segmentation result, the camera parameter, the human body coarse three-dimensional triangular mesh and the human body pose information comprises:

[0021] According to the human body motion video input by the user, a human body coarse three-dimensional triangular mesh of the target person in the reference space is obtained by using a pre-trained 3D human body template model, wherein the pre-trained 3D human body template model comprises a pre-trained SMPL-X model.

[0022] The clothing coarse three-dimensional triangular mesh is segmented from the human body coarse three-dimensional triangular mesh by using the clothing label in the video segmentation result of each frame of image, the camera parameter and the human body pose information.

[0023] According to the embodiment of the present application, the above-mentioned anchoring of the human body three-dimensional Gaussian representation on the human body coarse three-dimensional triangular mesh and the anchoring of the clothing three-dimensional Gaussian representation on the clothing coarse three-dimensional triangular mesh comprise:

[0024] The human body three-dimensional Gaussian representation is anchored on the human body coarse three-dimensional triangular mesh, and the clothing three-dimensional Gaussian representation is anchored on the clothing coarse three-dimensional triangular mesh, wherein the three-dimensional Gaussian representation comprises a plurality of three-dimensional Gaussian ellipsoids.

[0025] The geometric attribute calculation method of the three-dimensional Gaussian representation is defined according to the geometric information of the three-dimensional triangular facet, and the position information, the rotation information and the scale information of the human body three-dimensional Gaussian representation and the clothing three-dimensional Gaussian representation are calculated respectively.

[0026] Based on the calculated position information, rotation information and scale information, the synchronous transformation of the three-dimensional Gaussian representation and the coarse three-dimensional triangular mesh is realized in the spatial transformation process.

[0027] According to the embodiment of the present application, the above-mentioned transformation of the human body coarse three-dimensional triangular mesh and the human body three-dimensional Gaussian representation and the clothing coarse three-dimensional triangular mesh and the clothing three-dimensional Gaussian representation from the reference space to the observation space respectively comprises:

[0028] Based on the human body pose information, a human body non-rigid deformation multi-level perception machine is used to model the non-rigid deformation of the human body coarse three-dimensional triangular mesh of each frame of image in each view of the multi-view clothed human motion video.

[0029] transforming the human coarse 3D triangle mesh with non-rigid deformation to the observation space by using a linear blend skinning based skeleton rigid deformation module in a pre-trained 3D human template model, and updating geometric attributes of the human 3D Gaussian representation by using a pre-defined attribute calculation method of the 3D Gaussian representation;

[0030] modeling non-rigid deformation of the clothing coarse 3D triangle mesh of each frame of image in each view of the multi-view clothed human motion video based on human pose information by using a clothing non-rigid deformation multi-level perception machine;

[0031] transforming the clothing coarse 3D triangle mesh with non-rigid deformation to the observation space by using a linear blend skinning based skeleton rigid deformation module, and updating geometric attributes of the clothing 3D Gaussian representation by using a pre-defined attribute calculation method of the 3D Gaussian representation.

[0032] According to the embodiments of the present application, the projection of the human 3D Gaussian representation and the clothing 3D Gaussian representation in the observation space into the 2D Gaussian representation, the rendering of the human 2D Gaussian representation and the clothing 2D Gaussian representation comprises:

[0033] calculating geometric attributes of the 3D Gaussian representation of each frame of image in the multi-view clothed human motion video by using a pre-defined attribute calculation method of the 3D Gaussian representation, wherein the 3D Gaussian representation comprises a human 3D Gaussian representation and a clothing 3D Gaussian representation;

[0034] projecting the 3D Gaussian representation in the observation space into an imaging plane to obtain a 2D Gaussian representation of each frame of image in each view of the multi-view clothed human motion video by using camera parameters of each frame of image in each view of the multi-view clothed human motion video and the geometric attributes of the 3D Gaussian representation, wherein the 2D Gaussian representation comprises a human 2D Gaussian representation and a clothing 2D Gaussian representation;

[0035] obtaining influence information of the 2D Gaussian representation on each pixel in the multi-view clothed human motion video by using a color multi-level perception machine and a pre-defined splatting method;

[0036] calculating color of each pixel in the multi-view clothed human motion video by using the obtained influence information, density and color information of the 2D Gaussian representation, a color multi-level perception machine and a pre-defined rendering method, and then completing the rendering operation.

[0037] According to the embodiments of the present application, the repeated execution of the space transformation operation, the projection operation and the rendering operation, and the optimization of the coarse 3D triangle mesh and the 3D Gaussian representation in a self-supervised manner to obtain the reconstruction result of the target person comprises:

[0038] The space transformation operation, the projection operation and the rendering operation are repeatedly performed, and a coarse three-dimensional triangular mesh, a three-dimensional Gaussian, a human non-rigid deformation multi-level perception machine, a clothes non-rigid deformation multi-level perception machine and a color multi-level perception machine are optimized by a predefined loss function in a self-supervised manner to obtain a reconstruction result of the target person.

[0039] According to the embodiment of the present application, the human body of the target person is driven according to the input driving posture sequence to obtain the human body coarse three-dimensional triangular mesh in the driving state.

[0040] According to the user input driving posture sequence, the human body of the target person is driven by using the linear mixed skin-based skeleton rigid deformation module in the pre-trained 3D human body template model to obtain the human body coarse three-dimensional triangular mesh in the driving state.

[0041] According to the embodiment of the present application, the clothes of the target person are driven based on the human body coarse three-dimensional triangular mesh in the driving state to obtain the clothes coarse three-dimensional triangular mesh in the driving state.

[0042] The clothes of the target person are driven based on the human body coarse three-dimensional triangular mesh in the driving state by using the neural physics simulation model to obtain the clothes coarse three-dimensional triangular mesh in the driving state, wherein the neural physics simulation model is a pre-trained graph neural network model.

[0043] According to the embodiment of the present application, the three-dimensional Gaussian representation corresponding to the human body coarse three-dimensional triangular mesh in the driving state and the clothes coarse three-dimensional triangular mesh in the driving state is updated.

[0044] The geometric attribute of the three-dimensional Gaussian representation in the driving state is updated by using the human body coarse three-dimensional triangular mesh in the driving state and the clothes coarse three-dimensional triangular mesh in the driving state by using a predefined three-dimensional Gaussian representation attribute calculation method.

[0045] According to the embodiment of the present application, the three-dimensional Gaussian representation in the driving state is projected and rendered by using the reconstruction result and the camera parameter to obtain the driving result of the target person.

[0046] The three-dimensional Gaussian representation in the driving state is projected and rendered by using the reconstruction result, the camera parameter and the coarse three-dimensional triangular mesh in the driving state to obtain the driving result of the target person.

[0047] The above-mentioned reconstruction and driving method based on multi-view clothed human motion video provided by the application utilizes double-layer three-dimensional Gaussian representation for respectively representing the human body and clothes of a target person in the clothed human motion video, and realizes high-quality reconstruction of the target person in the clothed human motion video by continuously optimizing the double-layer three-dimensional Gaussian representation and the constructed multi-class multi-level perception machine, and realizes human body driving conforming to the characteristics of motion mechanics by utilizing a physical simulation model. The application can not only realize efficient and high-quality human body reconstruction and human body driving conforming to the characteristics of motion mechanics, but also has wide application scenarios in the field of virtual reality or augmented reality technology, and can produce good economic and social benefits. BRIEF DESCRIPTION OF DRAWINGS

[0048] The above and other objects, features and advantages of the present application will become more apparent from the following description of embodiments of the present application taken with reference to the accompanying drawings, in which:

[0049] Fig. 1 is a flowchart of the reconstruction and driving method based on multi-view clothed human motion video according to an embodiment of the present application;

[0050] Fig. 2 is an architectural diagram of the high-quality clothed human reconstruction and driving technical solution utilizing double-layer three-dimensional Gaussian representation according to an embodiment of the present application;

[0051] Fig. 3 is a schematic diagram of the reconstruction and driving method based on multi-view clothed human motion video in a virtual fitting scene according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It should be understood, however, that the description which follows is merely exemplary and is not intended to limit the scope of the application. In the following detailed description of embodiments of the present application, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that one or more embodiments of the present application can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring the concepts of the present application.

[0053] The terms used herein are merely used to describe specific embodiments and are not intended to limit the present application. The terms "include", "comprise" and the like used herein indicate the presence of the described features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0054] All terms used herein (including technical and scientific terms) have meanings commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted in a manner consistent with the context of the present specification, and should not be interpreted in an idealized or overly formal manner.

[0055] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should generally be interpreted to include at least one of A, B, or C (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C, etc.).

[0056] In the technical solutions of the present application, the user information (including but not limited to user personal information, user image information, user equipment information, such as location information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary security measures are taken, public order and good customs are not violated, and corresponding operation entrances are provided for the user to choose authorization or refusal.

[0057] Traditional clothed human reconstruction and driving methods often require fine modeling of the human body and clothes manually, and use optimization-based physical simulation methods to drive the clothes. Since manual modeling requires a lot of manpower and time, and the final result is heavily dependent on the technical ability of the three-dimensional animation artist responsible for modeling. Recently, some methods apply three-dimensional Gaussian representation to reconstruct and drive clothed humans. They can reconstruct high-quality human geometry and appearance from multi-view videos containing humans in a few hours or even minutes by combining human priors with three-dimensional Gaussian representation. However, these methods often use a single-layer representation based on a human parametric model to model both the human body and the clothes, making them perform poorly on loose clothes. In addition, most methods directly use linear blend skinning-based skeleton deformation to drive both the human body and the clothes. This approach completely ignores the non-rigid deformation of clothes when the human body moves, making the driving result rigid and distorted. Some methods use a multi-level perception to model the non-rigid deformation of clothes, which takes the human body pose of the current frame as input. This approach can achieve good results within the poses covered by the training sequence, but may generate unreasonable results outside the training set. At the same time, even if the human body is in the same pose, the clothes may be in different states, so modeling the non-rigid deformation of clothes as a function that only depends on the single-frame human pose may produce motion results that do not conform to the laws of motion.

[0058] In order to solve the problems in the prior art, the application provides a reconstruction and driving method based on multi-view clothed human motion video, which can reconstruct the approximate geometry and high-fidelity appearance of a human body and clothes from an input video respectively, and drive the clothes using a neuromechanical simulation method under a driving pose sequence to generate a rendering result conforming to motion mechanics.

[0059] Fig. 1 is a flowchart of the reconstruction and driving method based on multi-view clothed human motion video according to an embodiment of the application.

[0060] According to a first aspect of the application, a reconstruction and driving method based on multi-view clothed human motion video is provided, and the reconstruction stage includes operations S101-S106.

[0061] In operation S101, camera parameters and human body pose information of a target person are obtained from the multi-view clothed human motion video, and the multi-view clothed human motion video is segmented to obtain a video segmentation result.

[0062] Before operation S101, the processing of the multi-view clothed human motion video obtains an authorization of the target person in the video, and the operation S101 and the subsequent related operations are performed under the authorization of the target person.

[0063] In operation S102, a rough three-dimensional triangular mesh of a human body in a reference space is obtained, and a rough three-dimensional triangular mesh of clothes in the reference space is obtained using the video segmentation result, the camera parameters, the rough three-dimensional triangular mesh of the human body, and the human body pose information.

[0064] The rough three-dimensional triangular mesh can also be referred to as a rough triangular mesh, and both represent the same technical meaning in the technical field. In the embodiments of the application, the rough three-dimensional triangular mesh is simply referred to as a rough triangular mesh.

[0065] In operation S103, the human body three-dimensional Gaussian representation is anchored on the rough three-dimensional triangular mesh of the human body, and the clothes three-dimensional Gaussian representation is anchored on the rough three-dimensional triangular mesh of the clothes.

[0066] The three-dimensional Gaussian representation includes a plurality of three-dimensional Gaussian ellipsoids, and the attribute information mainly involves geometric information (or geometric attribute, such as the position of the center of the three-dimensional Gaussian ellipsoid in the three-dimensional Gaussian representation, the orthogonal basis vector, etc.) and appearance information (or appearance attribute). The geometric information is calculated by the rough triangular mesh, and the appearance information is optimized with the entire clothed human model.

[0067] In operation S104, the rough three-dimensional triangular mesh of the human body and the human body three-dimensional Gaussian representation and the rough three-dimensional triangular mesh of the clothes and the clothes three-dimensional Gaussian representation are respectively transformed from the reference space to the observation space.

[0068] In operation S104, two types of non-rigid deformation multi-level perception machines are also utilized, which are human non-rigid deformation multi-level perception machine and clothes non-rigid deformation multi-level perception machine.

[0069] In operation S105, the human three-dimensional Gaussian representation and the clothes three-dimensional Gaussian representation in the observation space are projected into two-dimensional Gaussian representations, and the human two-dimensional Gaussian representation and the clothes two-dimensional Gaussian representation are rendered.

[0070] In the process of rendering the two-dimensional Gaussian representation, a color multi-level perception machine and a predefined sputtering and rendering method are utilized.

[0071] In operation S106, the spatial transformation operation, the projection operation and the rendering operation are repeatedly performed, and the coarse three-dimensional triangular mesh and the three-dimensional Gaussian representation are optimized in a self-supervised manner to obtain the reconstruction result of the target person.

[0072] The self-supervised training and optimization process runs through each frame image of each view in the multi-view clothed motion video.

[0073] Operations S101-S106 are the reconstruction phase of the above-mentioned multi-view clothed human motion video reconstruction and driving method. By processing each frame image under each view in the multi-view clothed human motion video, the training is continuously performed in a self-supervised manner, so that the three-dimensional Gaussian representation, the coarse three-dimensional triangular mesh and the multi-level perception machine are continuously optimized, and a high-quality target person reconstruction result can be obtained.

[0074] According to the embodiments of the present application, the above-mentioned multi-view clothed human motion video reconstruction and driving method further comprises operations S107-S110.

[0075] In operation S107, the human body of the target person is driven according to the input driving pose sequence to obtain the coarse three-dimensional triangular mesh of the human body at the driving time.

[0076] In operation S108, the clothes of the target person are driven based on the coarse three-dimensional triangular mesh of the human body at the driving time to obtain the coarse three-dimensional triangular mesh of the clothes at the driving time.

[0077] In operation S109, the three-dimensional Gaussian representation corresponding to the coarse three-dimensional triangular mesh of the human body at the driving time and the coarse three-dimensional triangular mesh of the clothes at the driving time is updated.

[0078] In operation S110, the updated three-dimensional Gaussian representation is projected and rendered using the reconstruction result and the camera parameters to obtain the driving result of the target person.

[0079] Operations S107-S110 in the above embodiments relate to the driving phase of the multi-view clothed human motion video reconstruction and driving method.

[0080] The driving stage aims to generate the driving result of the target user according to the driving sequence input by the user, using the three-dimensional Gaussian representation and the like obtained in the reconstruction stage. For example, a dancing video of the target person can be generated according to the dancing driving sequence input by the user, so that the posture change and the clothing change of the human body in the dancing video can conform to the kinematics.

[0081] The above-mentioned reconstruction and driving method for the multi-view clothed human motion video provided by the present application uses the double-layer three-dimensional Gaussian representation to represent the human body and the clothing of the target person in the clothed human motion video, respectively, and realizes the high-quality reconstruction of the target person in the clothed human motion video by continuously optimizing the double-layer three-dimensional Gaussian representation and the multi-class multi-level perception machine constructed, and realizes the human body driving conforming to the kinematics characteristics by using the physical simulation model. The present application not only can realize the high-efficiency and high-quality human body reconstruction and the human body driving conforming to the kinematics, but also has a wide application scenario in the virtual reality or augmented reality technical field, can produce good economic and social benefits, for example, can realize online fitting and the like.

[0082] The above-mentioned method provided by the present application will be further described in detail below by means of specific embodiments and in combination with the accompanying drawings 2.

[0083] Fig. 2 is an architecture diagram of the high-quality clothed human reconstruction and driving technical scheme using the double-layer three-dimensional Gaussian representation according to an embodiment of the present application.

[0084] As shown in FIG. 2, the above technical solution provided by the present application extracts the human body pose information and camera position of each frame from the initial multi-view video based on an optimized method; the initial multi-view human body motion video set is segmented frame by frame to obtain the human body and clothing labels on each picture; according to the label information in the above picture, the human body pose information of each frame is combined to extract the coarse triangular mesh of the clothes in the reference space from the human body parameterized mesh in the reference space; the three-dimensional Gaussian representation is anchored on the surface of the coarse triangular mesh of the clothes and the human body parameterized mesh, and the geometric information of the three-dimensional Gaussian representation is calculated according to the point-face relationship and position information of the mesh; in each frame, the human body pose information and two multi-level perceptrons are used to transform the human body coarse triangular mesh and the coarse triangular mesh of the clothes from the reference space to the observation space, and the geometric properties of the three-dimensional Gaussian representation are updated synchronously; the pictures under the current frame and the view are generated by using the three-dimensional Gaussian sputtering method, and the positions of the human body parameterized mesh and the coarse triangular mesh of the clothes are optimized according to the initial multi-view human body motion video to reconstruct the appearance properties of each three-dimensional Gaussian representation. After the reconstruction is completed, according to the human body pose information of the current frame and the grid position, speed and other information of the last frame, the pre-trained graph neural network is used to drive the coarse triangular mesh of the clothes; the geometric information of the three-dimensional Gaussian representation is updated according to the above mesh, and the pictures under the current view and the driven human body potential are generated from the three-dimensional Gaussian representation by using the three-dimensional Gaussian sputtering method. Since the double-layer representation is used to model the human body and the clothes respectively, and the physical simulation model is used to drive the clothes, the driving result generated by the present application has the characteristics of fitting the motion mechanics, and by replacing the coarse triangular mesh of the clothes and the corresponding three-dimensional Gaussian representation, the present application can realize the three-dimensional virtual fitting application.

[0085] According to the embodiments of the present application, the above-mentioned segmentation of the multi-view clothed human body motion video to obtain the video segmentation result comprises: performing camera parameter acquisition operation and human body motion capture operation on each view of the multi-view clothed human body motion video frame by frame by using a pre-defined camera calibration and human body pose capture method to obtain the camera parameters and the human body pose information of each frame of image, wherein the pre-defined camera calibration and human body pose capture method comprises the EasyMocap method; performing frame segmentation on each view of the multi-view clothed human body motion video by using a pre-trained image segmentation large model to obtain the video segmentation result of each frame of image, wherein the pre-trained image segmentation large model comprises a pre-trained SAM model, and wherein the video segmentation result of each frame of image comprises human body label and clothing label.

[0086] FIG. 3 is a schematic diagram of the reconstruction and driving method based on the multi-view clothed human body motion video in the virtual fitting scene according to an embodiment of the present application.

[0087] As shown in FIG. 3, a double-layer three-dimensional Gaussian representation is used to model the human body and the corresponding clothes, which is composed of a coarse triangular mesh of the human body and the clothes and a three-dimensional Gaussian representation anchored thereon. For any face on the coarse three-dimensional mesh of the human body and the clothes, the geometric properties of the three-dimensional Gaussian ellipsoid are defined according to the positions of the three vertices and the corresponding barycentric coordinates. In addition to the geometric properties, each three-dimensional Gaussian ellipsoid also records the density value and the color feature vector of the region.

[0088] In the reconstruction stage, the coarse triangular meshes of the clothes and the body are transformed from the reference space to the observation space using the linear blend skinning-based skeleton rigid deformation and the multi-level perceptron-based non-rigid deformation, and the geometric properties of the Gaussians are updated according to the transformed mesh vertex positions. According to the camera pose and parameters of the input video, the three-dimensional Gaussian representation of the observation space is rasterized to the image space, and the picture is rendered according to the Gaussian sputtering formula. By calculating the color difference between the rendered picture and the real picture as the optimization target, the model can be trained in a self-supervised manner.

[0089] After the reconstruction is completed, the coarse triangular mesh of the body is driven using the linear blend skinning-based skeleton rigid deformation according to the input driving human pose sequence, and the acceleration of the coarse triangular mesh of the clothes in the current frame is predicted using a pre-trained model based on graph neural networks according to the position and velocity of the coarse triangular mesh of the body, the coarse triangular mesh of the clothes in the current frame, and the position of the coarse triangular mesh of the body in the next frame, and then the position of the coarse triangular mesh of the clothes in the next frame is calculated.

[0090] Based on the positions of the mesh and the coarse triangular mesh of the clothes, the geometric information of the three-dimensional Gaussian representation is updated, and the driving picture in the new pose is rendered using the same method as in the reconstruction stage.

[0091] According to the embodiments of the present application, the above-mentioned obtaining of the coarse three-dimensional triangular mesh of the clothes in the reference space from the video segmentation result, the camera parameters, the coarse three-dimensional triangular mesh of the human body, and the human pose information includes: obtaining the coarse three-dimensional triangular mesh of the human body of the target person in the reference space from a human motion video including the target person input by a user, using a pre-trained 3D human template model, wherein the pre-trained 3D human template model includes a pre-trained SMPL-X model; and obtaining the coarse three-dimensional triangular mesh of the clothes from the coarse three-dimensional triangular mesh of the human body, using the clothes label in the video segmentation result of each frame image, the camera parameters, and the human pose information, wherein the human pose information includes human shape parameters and human pose parameters.

[0092] The process of obtaining the two types of coarse three-dimensional triangular meshes (i.e., coarse triangular meshes, the same below) will be further described in detail through specific embodiments.

[0093] The rough triangular mesh of the human body and clothes in the reference space, and the human body pose parameters of each frame are taken. According to the input video containing human motion, the approximate rough triangular mesh of the human body and clothes can be obtained by using the existing method. In the example of the present application, the SMPL-X human parameterized mesh is used as the rough triangular mesh M body of the human body, and the pose parameters of the human body of each frame are obtained by optimization , and the human body shape parameters β, wherein n is the total number of frames of the video. Subsequently, the rough triangular mesh of the whole body is reconstructed by the existing method, and the rough triangular mesh M clothing of the clothes is segmented from the whole body mesh according to the clothes label obtained in the picture by the foregoing operation.

[0094] According to the embodiment of the present application, the above-mentioned anchoring the human body three-dimensional Gaussian representation on the rough three-dimensional triangular mesh of the human body and the clothes three-dimensional Gaussian representation on the rough three-dimensional triangular mesh of the clothes includes: anchoring the human body three-dimensional Gaussian representation on the rough three-dimensional triangular mesh of the human body and the clothes three-dimensional Gaussian representation on the rough three-dimensional triangular mesh of the clothes, wherein the three-dimensional Gaussian representation includes a plurality of three-dimensional Gaussian ellipsoids; defining a geometric attribute calculation method of the three-dimensional Gaussian representation according to the geometric information of the three-dimensional triangular patch, and calculating the position information, rotation information and scale information of the human body three-dimensional Gaussian representation and the clothes three-dimensional Gaussian representation respectively; based on the calculated position information, rotation information and scale information, realizing the synchronous transformation of the three-dimensional Gaussian representation and the rough three-dimensional triangular mesh in the spatial transformation process.

[0095] The operation of anchoring the three-dimensional Gaussian representation on the rough triangular mesh is further described in detail below through specific embodiments.

[0096] The three-dimensional Gaussian representation is anchored on the rough triangular mesh of the human body and clothes. On each triangular patch of the rough triangular mesh, a plurality of three-dimensional Gaussian ellipsoids are anchored, and the geometric information of the three-dimensional Gaussian ellipsoid is defined according to the geometric information of the triangular patch, so that when the rough triangular mesh is geometrically transformed, the three-dimensional Gaussian representation will be synchronously transformed. In the example of the present application, the position, rotation and scale of a specific three-dimensional Gaussian ellipsoid on a triangular patch are defined and calculated using formulas (1)-(5): r1=n i (2), r3=r1×r2(4),

[0097] wherein x is the center position of the three-dimensional Gaussian ellipsoid, p ij is the position of the three vertices on the i-th triangular patch of the rough triangular mesh M body or M clothing , n i is the normal vector of the i-th triangular patch, and r1,2,3 Let S be the orthogonal basis vector of the three-dimensional Gaussian ellipsoid, and let S be the length of the three-dimensional Gaussian ellipsoid in the three orthogonal basis directions.

[0098] According to embodiments of the present invention, the above-described transformation of the rough 3D triangular mesh of the human body and the 3D Gaussian representation of the human body, as well as the rough 3D triangular mesh of clothing and the 3D Gaussian representation of clothing, from the reference space to the observation space includes: based on human posture information, using a multi-level perceptron with non-rigid deformation to model the non-rigid deformation of the rough 3D triangular mesh of the human body in each frame of each viewpoint in a multi-view video of human motion with clothing; using a rigid deformation module of the skeleton based on linear hybrid skin in a pre-trained 3D human template model to transform the rough 3D triangular mesh of the human body with non-rigid deformation to the observation space, and updating the geometric properties of the 3D Gaussian representation of the human body using a predefined attribute calculation method; based on human posture information, using a multi-level perceptron with non-rigid deformation to model the non-rigid deformation of the rough 3D triangular mesh of clothing in each frame of each viewpoint in a multi-view video of human motion with clothing; using a rigid deformation module of the skeleton based on linear hybrid skin to transform the rough 3D triangular mesh of clothing with non-rigid deformation to the observation space, and updating the geometric properties of the 3D Gaussian representation of clothing using a predefined attribute calculation method.

[0099] The above spatial transformation operation will be further explained in detail below through specific embodiments.

[0100] The rough triangular mesh of the body and clothing, along with its anchored 3D Gaussian representation, is transformed to the observation space. Based on the human pose parameters obtained in the preceding operations, the non-rigid deformation of the body or clothing in each frame is modeled using two multi-level perceptrons, and then transformed to the observation space using rigid skeleton deformation based on linear hybrid skinning. In this invention example, the rough triangular mesh of the body and clothing is transformed from the reference space to the observation space according to formulas (6) and (7): Δp=f part (p,t)(7),

[0101] Where p is M body Or M clothing The position of any point in the reference space, p ′ Let w be the location of this point in the observation space. i G represents the skinning weight of the point on the i-th bone. i (θ,β) is the transformation matrix of the i-th bone based on the human morphology parameter β and the pose parameter θ of the current frame. Δp is formed by two multi-level perceptrons f. partPredicted body or garment non-rigid deformation, part refers to body or garment, t is the time sequence number of this frame. At the same time, the geometric information of the three-dimensional Gaussian representation of the anchor on the triangular mesh is updated according to the foregoing operation S103.

[0102] According to the embodiments of the present application, the above projecting the three-dimensional Gaussian representation of the human body and the three-dimensional Gaussian representation of the clothes in the observation space into the two-dimensional Gaussian representation, rendering the two-dimensional Gaussian representation of the human body and the two-dimensional Gaussian representation of the clothes comprises: calculating the geometric properties of the three-dimensional Gaussian representation of each frame image in the multi-view clothed human motion video using a pre-defined attribute calculation method of the three-dimensional Gaussian representation, wherein the three-dimensional Gaussian representation comprises the three-dimensional Gaussian representation of the human body and the three-dimensional Gaussian representation of the clothes; projecting the three-dimensional Gaussian representation in the observation space into the imaging plane using the camera parameters of each frame image under each view in the multi-view clothed human motion video and the geometric properties of the three-dimensional Gaussian representation, to obtain the two-dimensional Gaussian representation of each frame image under each view in the multi-view clothed human motion video, wherein the two-dimensional Gaussian representation comprises the two-dimensional Gaussian representation of the human body and the two-dimensional Gaussian representation of the clothes; obtaining the influence information of the two-dimensional Gaussian representation on each pixel in the multi-view clothed human motion video through the color multi-level perception machine and a pre-defined sputtering method; and calculating the color of each pixel in the multi-view clothed human motion video through the color multi-level perception machine and a pre-defined rendering method using the obtained influence information and the density and color information of the two-dimensional Gaussian representation, to complete the rendering operation.

[0103] According to the embodiments of the present application, the above repeatedly performing the space transformation operation, the projection operation and the rendering operation, and optimizing the coarse three-dimensional triangular mesh and the three-dimensional Gaussian representation in a self-supervised manner to obtain the reconstruction result of the target person comprises: repeatedly performing the space transformation operation, the projection operation and the rendering operation, and optimizing the coarse three-dimensional triangular mesh, the three-dimensional Gaussian, the human body non-rigid deformation multi-level perception machine, the garment non-rigid deformation multi-level perception machine and the color multi-level perception machine in a self-supervised manner through a pre-defined loss function, to obtain the reconstruction result of the target person.

[0104] The rendering process is further described in detail below through specific embodiments.

[0105] The three-dimensional Gaussian representation is projected onto the imaging plane according to the camera information, the image is calculated through the sputtering formula and the volume rendering formula, and the final model is reconstructed through the optimization of the reality loss function. In the present application example, first, the three-dimensional Gaussian representation is projected onto the imaging plane to obtain the two-dimensional Gaussian representation according to formulas (8) and (9): ∑=RSS T R T (8), ∑′=JW∑W T J T (9),

[0106] where ∑ is the covariance matrix of the three-dimensional Gaussian ellipsoid, which is calculated from the geometry information rotation R and scale S of the three-dimensional Gaussian ellipsoid obtained in operations S103 and S104, ∑ ′ is the covariance matrix of the two-dimensional Gaussian ellipse, W is the projection transformation matrix calculated according to the camera information, and J is the corresponding jacobi matrix. Then, the influence of each two-dimensional Gaussian ellipse on each pixel is obtained according to the following splash formula, and the color corresponding to each pixel is calculated according to the volume rendering formula according to the density and color information of each three-dimensional Gaussian ellipsoid, as shown in formulas (10) and (11):

[0107] where x is the distance from the pixel center to the two-dimensional Gaussian ellipse, σ is the density of each three-dimensional Gaussian ellipsoid, c is the color corresponding to each three-dimensional Gaussian ellipsoid, and C is the color finally calculated for the pixel.

[0108] Finally, the picture obtained by rendering is used to calculate the fidelity loss and the geometric regularization loss with the input picture, and the entire model is trained in a self-supervised manner. The pre-defined loss function is shown in formula (12): L = L color + L mask + L seg + L geo (12),

[0109] where L geo is the fidelity loss calculated with the input picture, as shown in formula (13): L geo = L laplacian + L normal + L collision + L distance (13),

[0110] where L laplacian and L normal are the Laplacian loss function and the normal consistency loss function for smoothing the mesh, L collision is the collision loss function for making the clothes outside the body, L distance is the regularization loss function for making the position of the body mesh in the reference space close to the human parametric model mesh, L color is the fidelity loss calculated with the input picture, L mask and L seg are the fidelity loss calculated with the human and clothes segmentation labels obtained by preprocessing the previous data, and L geo is the geometric regularization loss function.

[0111] According to the embodiment of the present application, the above driving the human body of the target person according to the input driving pose sequence to obtain the rough three-dimensional triangular mesh of the human body at the driving time includes: driving the human body of the target person according to the input driving pose sequence by using the linear mixed skinning based skeleton rigid deformation module in the pre-trained 3D human body template model to obtain the rough three-dimensional triangular mesh of the human body at the driving time.

[0112] Wherein, the above driving the clothes of the target person based on the rough three-dimensional triangular mesh of the human body at the driving time to obtain the rough three-dimensional triangular mesh of the clothes at the driving time includes: driving the clothes of the target person based on the rough three-dimensional triangular mesh of the human body at the driving time by using the neural physics simulation model to obtain the rough three-dimensional triangular mesh of the clothes at the driving time, wherein the neural physics simulation model is a pre-trained graph neural network model.

[0113] According to the embodiment of the present application, the above updating the three-dimensional Gaussian representation corresponding to the rough three-dimensional triangular mesh of the human body at the driving time and the rough three-dimensional triangular mesh of the clothes at the driving time includes: updating the geometric attribute of the three-dimensional Gaussian representation at the driving time by using the rough three-dimensional triangular mesh of the human body at the driving time and the rough three-dimensional triangular mesh of the clothes at the driving time through the pre-defined attribute calculation method of the three-dimensional Gaussian representation.

[0114] Wherein, the above projecting and rendering the updated three-dimensional Gaussian representation by using the reconstruction result and the camera parameter to obtain the driving result of the target person includes: projecting and rendering the three-dimensional Gaussian representation at the driving time by using the reconstruction result, the camera parameter and the rough three-dimensional triangular mesh at the driving time to obtain the driving result of the target person.

[0115] The related operations in the driving stage will be further described in detail through specific embodiments.

[0116] In the driving stage, the clothes are driven using the neural physics simulation, and the final driving image is synthesized. After the training in operation S106 is completed, the body is driven using the linear mixed skinning based skeleton rigid deformation in operation S104 based on the input driving pose sequence, and the clothes are driven using formulas (14)-(17): t = GNN(G t-1 , p)(14), v t = v t-1 + a t (15), p t = p t + v t (16), G t = Ω(G t-1 , p t , v t )(17),

[0117] Wherein, Gt is a point-edge adjacency relationship of the M body and M clothing is a graph constructed by the point-edge adjacency relationship of the M t is an acceleration of each vertex of the coarse triangle mesh of the clothes at the t-th frame, v t is a velocity of each vertex of the coarse triangle mesh of the clothes at the t-th frame, p t is a position of each vertex of the coarse triangle mesh of the clothes at the t-th frame, and Ω() denotes an operation of updating the graph G according to the information of the t-th frame. Subsequently, the geometric information of the three-dimensional Gaussian representation is updated according to operation S103, and the final driving image is rendered according to operation S105.

[0118] The computer program product of the present application can be a computer program embodied on a computer readable medium. The computer program product can also be a propagated signal on a carrier.

[0119] Those skilled in the art can understand that the features described in various embodiments of the present application can be combined and / or integrated in various combinations, even if such combinations or integrations are not explicitly described in the present application. In particular, the features described in various embodiments of the present application can be combined and / or integrated in various combinations without departing from the spirit and teachings of the present application. All such combinations and / or integrations fall within the scope of the present application.

[0120] The above describes embodiments of the present application. However, these embodiments are only for illustrative purposes, and are not intended to limit the scope of the present application. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present application, and these substitutions and modifications should all fall within the scope of the present application.

Claims

1. A method for reconstructing and driving human motion based on multi-view clothing-wearing human motion video, characterized in that, The method includes: Camera parameters and the human posture information of the target person are obtained from the multi-view video of the human body in clothing, and the video of the human body in clothing in clothing is segmented to obtain the video segmentation result. A rough 3D triangular mesh of the human body in a reference space is obtained, and a rough 3D triangular mesh of clothing in the reference space is obtained using the video segmentation results, the camera parameters, the rough 3D triangular mesh of the human body, and the human body pose information. Anchor the three-dimensional Gaussian representation of the human body to the rough three-dimensional triangular mesh of the human body, and anchor the three-dimensional Gaussian representation of the clothing to the rough three-dimensional triangular mesh of the clothing; The rough 3D triangular mesh of the human body and the 3D Gaussian representation of the human body, as well as the rough 3D triangular mesh of the clothing and the 3D Gaussian representation of the clothing, are respectively transformed from the reference space to the observation space; The three-dimensional Gaussian representation of the human body and the three-dimensional Gaussian representation of clothing in the observation space are projected into a two-dimensional Gaussian representation, and the two-dimensional Gaussian representation of the human body and the two-dimensional Gaussian representation of clothing are rendered. The spatial transformation operation, projection operation, and rendering operation are repeatedly performed, and the coarse 3D triangular mesh and the 3D Gaussian representation are optimized in a self-supervised manner to obtain the reconstruction result of the target character.

2. The method according to claim 1, characterized in that, Also includes: Based on the input driving posture sequence, the human body of the target person is driven to obtain a rough three-dimensional triangular mesh of the human body during driving. Based on the rough 3D triangular mesh of the human body during the driving process, the clothing of the target person is driven to obtain a rough 3D triangular mesh of the clothing during the driving process. Update the 3D Gaussian representations corresponding to the rough 3D triangular meshes of the human body and the rough 3D triangular meshes of the clothing during the driving process; Using the reconstruction results and the camera parameters, the updated 3D Gaussian representation is projected and rendered to obtain the driving result of the target person.

3. The method according to claim 1, characterized in that, The multi-view video of human movement while wearing clothing is segmented to obtain the following video segmentation results: The camera parameters and human pose capture of the target person are obtained frame by frame using the predefined camera calibration and human pose capture method for each view of the multi-view clothing human motion video, so as to obtain the camera parameters and human pose information of the target person for each frame. The predefined camera calibration and human pose capture method includes the EasyMocap method. The pre-trained large-scale image segmentation model is used to segment each viewpoint of the multi-view video of a dressed human body motion frame by frame to obtain the video segmentation result of each frame. The pre-trained large-scale image segmentation model includes a pre-trained SAM model. The video segmentation result of each frame includes the human body label and clothing label of the target person.

4. The method according to claim 3, characterized in that, The process of obtaining the rough 3D triangular mesh of clothing in the reference space using the video segmentation results, camera parameters, the rough 3D triangular mesh of the human body, and the human pose information includes: Based on the human motion video of the target person input by the user, a rough three-dimensional triangular mesh of the human body of the target person in the reference space is obtained using the pre-trained 3D human body template model, wherein the pre-trained 3D human body template model includes a pre-trained SMPL-X model. The clothing rough 3D triangular mesh is obtained by segmenting the human body rough 3D triangular mesh from the video segmentation results of each frame image using the clothing tag, camera parameters, and human pose information.

5. The method according to claim 1, characterized in that, Anchoring the 3D Gaussian representation of the human body to the rough 3D triangular mesh of the human body, and anchoring the 3D Gaussian representation of clothing to the rough 3D triangular mesh of clothing, includes: The three-dimensional Gaussian representation of the human body is anchored on the rough three-dimensional triangular mesh of the human body, and the three-dimensional Gaussian representation of the clothing is anchored on the rough three-dimensional triangular mesh of the clothing, wherein the three-dimensional Gaussian representation includes multiple three-dimensional Gaussian ellipsoids; Based on the geometric information of the three-dimensional triangular facets, a method for calculating the geometric properties of the three-dimensional Gaussian representation is defined, and the position, rotation, and scale information of the three-dimensional Gaussian representation of the human body and the three-dimensional Gaussian representation of the clothing are calculated respectively. Based on the calculated position, rotation, and scale information, the synchronous transformation between the three-dimensional Gaussian representation and the coarse three-dimensional triangular mesh is achieved during the spatial transformation process.

6. The method according to claim 1, characterized in that, Transforming the rough 3D triangular mesh of the human body and its 3D Gaussian representation, as well as the rough 3D triangular mesh of the clothing and its 3D Gaussian representation, from the reference space to the observation space includes: Based on the human posture information, the non-rigid deformation of the human body rough three-dimensional triangular mesh in each frame of each viewpoint in the multi-view clothing human motion video is modeled using a multi-level perceptron with non-rigid deformation of the human body. The rigid skeletal deformation module based on linear hybrid skin in the pre-trained 3D human template model is used to transform the rough 3D triangular mesh of the human body with non-rigid deformation to the observation space, and the geometric properties of the human body 3D Gaussian representation are updated using a predefined attribute calculation method. Based on the human posture information, the non-rigid deformation of the clothing rough three-dimensional triangular mesh in each frame of each viewpoint in the multi-view clothing human motion video is modeled using a multi-level perceptron with non-rigid deformation of clothing. The rough 3D triangular mesh of the clothing with non-rigid deformation is transformed to the observation space using the rigid deformation module of the skeleton based on linear hybrid skin, and the geometric properties of the clothing in the 3D Gaussian representation are updated using the predefined property calculation method of the 3D Gaussian representation.

7. The method according to claim 6, characterized in that, Projecting the three-dimensional Gaussian representation of the human body and the three-dimensional Gaussian representation of clothing in the observation space into a two-dimensional Gaussian representation, and rendering the two-dimensional Gaussian representations of the human body and clothing includes: Using a predefined attribute calculation method based on three-dimensional Gaussian representation, the geometric attributes of the three-dimensional Gaussian representation of each frame in the multi-view video of human movement in clothing are calculated, wherein the three-dimensional Gaussian representation includes the three-dimensional Gaussian representation of the human body and the three-dimensional Gaussian representation of the clothing. Using the camera parameters and geometric properties of the three-dimensional Gaussian representation of each frame image in each viewpoint of the multi-view clothing human motion video, the three-dimensional Gaussian representation in the observation space is projected onto the imaging plane to obtain the two-dimensional Gaussian representation of each frame image in each viewpoint of the multi-view clothing human motion video, wherein the two-dimensional Gaussian representation includes the two-dimensional Gaussian representation of the human body and the two-dimensional Gaussian representation of the clothing. The influence information of the two-dimensional Gaussian representation on each pixel in the multi-view video of human movement dressed in clothing is obtained by using a color multi-level perceptron and a predefined sputtering method. Using the obtained influence information and the density and color information represented by the two-dimensional Gaussian, the color of each pixel in the multi-view clothing human motion video is calculated through the color multi-level perceptron and a predefined rendering method, thereby completing the rendering operation.

8. The method according to claim 7, characterized in that, Repeatedly performing spatial transformation, projection, and rendering operations, and self-supervisedly optimizing the coarse 3D triangular mesh and the 3D Gaussian representation, the reconstructed result of the target figure includes: The spatial transformation operation, projection operation, and rendering operation are repeatedly performed, and the coarse 3D triangular mesh, the 3D Gaussian mesh, the human body non-rigid deformation multi-level perceptron, the clothing non-rigid deformation multi-level perceptron, and the color multi-level perceptron are optimized in a self-supervised manner using a predefined loss function to obtain the reconstruction result of the target person.

9. The method according to claim 2, characterized in that, Based on the input driving posture sequence, the target person's human body is driven to obtain a rough 3D triangular mesh of the human body during driving, including: Based on the user-input driving posture sequence, the skeleton rigid deformation module based on linear hybrid skin in the pre-trained 3D human body template model is used to drive the human body of the target person, and a rough three-dimensional triangular mesh of the human body during the driving is obtained. Specifically, based on the rough 3D triangular mesh of the human body during the driving process, the clothing of the target person is driven to obtain a rough 3D triangular mesh of the clothing during the driving process, including: Based on the rough 3D triangular mesh of the human body during the driving process, the clothing of the target person is driven using a neurophysical simulation model to obtain the rough 3D triangular mesh of the clothing during the driving process, wherein the neurophysical simulation model is a pre-trained graph neural network model.

10. The method according to claim 2, characterized in that, Updating the 3D Gaussian representations corresponding to the rough 3D triangular mesh of the human body and the rough 3D triangular mesh of the clothing during the driving process includes: Using a predefined three-dimensional Gaussian representation of attribute calculation method, the geometric attributes of the three-dimensional Gaussian representation during driving are updated through the rough three-dimensional triangular mesh of the human body and the rough three-dimensional triangular mesh of clothing during driving. Specifically, the updated 3D Gaussian representation is projected and rendered using the reconstruction results and camera parameters to obtain the driving result of the target person, including: The target character's driving result is obtained by projecting and rendering the three-dimensional Gaussian representation during driving using the reconstruction result, the camera parameters, and the rough three-dimensional triangular mesh during driving.

Images