Rigging method and system using 4D volumatric capture

Abstract

A rigging method and system using 4D volumetric capture, comprising: (a) receiving a sequence of 3D mesh data consisting of a continuous series of frames; (b) estimating a 3D pose consisting of joints and bones from the sequence of 3D meshes; (c) estimating 3D mesh and pose data of an SMPL model for keyframes of the 3D mesh sequence; (d) generating a 3D base model of the keyframes by transferring the 3D mesh of the SMPL model; (e) performing muscle simulation on the 3D base model; and (f) generating a sequence of 3D poses and 3D mesh models of the estimated keyframes. and, (g) a configuration including a step of adjusting the joint by comparing the result of the muscle simulation with a three-dimensional mesh model, thereby adjusting the position of the joint by comparing the result of the muscle simulation based on motion capture information with a volumetric model, can detect a more accurate joint.

Description

Technology Field

[0001] The present invention relates to a rigging method and system using 4D volumetric capture, wherein motion capture is performed in 4D volumetric, a muscle simulator is operated using motion capture information to express changes in the surface of a 3D human body model, and the position of a joint is adjusted by comparing this with the volumetric model. Background Technology

[0002] Recently, 4D volumetric capture technology, which acquires 3D models and scenes for every frame, is being extensively researched. A 4D volumetric model is defined as a case where a 3D volumetric model exists in every frame in time. While there are various methods for producing high-quality 3D models for use in digital environments, there is a fundamental problem in that producing 3D models requires a significant amount of manpower and time.

[0003] To overcome this, various technologies for generating 3D models based on 2D images have emerged, and 4D volumetric capture is receiving attention as the latest model of such technology [Non-patent Literature 1, 2, 3, 4]. Guo et al. developed "The Relightables," a volumetric capture system for realistic and high-quality relightable full-body performance capture. They presented a new system with excessive geometric, lighting, and appearance constraints through a combination of advanced active lighting, a new high-resolution depth sensor, and a high-resolution camera array [Non-patent Literature 1]. Schreer et al. proposed 360-degree volumetric video production for an integrated capture and lighting system [Non-patent Literature 3]. Furthermore, they proposed a complete multi-view 3D processing chain for high-quality mesh sequences in terms of geometric detail and texture quality. Chen et al. enhanced a professional end-to-end volumetric video production pipeline to achieve high-fidelity human body reconstruction using only passive cameras [Non-patent Literature 3]. DynamicFusion [Non-patent Literature 5] is a technology for reconstructing a 3D model in real time using depth images captured by a single depth sensor, and proposes a methodology to improve the completeness of the model by gradually accumulating a limited amount of depth information acquired by a single RGB-Depth camera.

[0004] 4D volumetric data has the advantage of enabling high-quality 3D content services by precisely acquiring and storing the shape and motion of a 3D model for every frame. On the other hand, it has the disadvantages that the data volume is very large, the mesh structure of each frame is not consistent, and the color of the texture may differ depending on each frame. Prior art literature

[0006] Guo, Kaiwen and Lincoln, Peter and Davidson, et al, The Relightables: Volumetric Performance Capture of Humans with Realistic Relighting, December 2019, ACM Trans. Graph., vol. 38. no. 6, p.19 https: / / doi.org / 10.1145 / 3355089.3356571, doi = 10.1145 / 3355089.3356571Pietroszek, Krzysztof and Eckhardt, Christian, Volumetric Capture for Narrative Films, 26th ACM Symposium on Virtual Reality Software and Technology, 2020, https: / / doi.org / 10.1145 / 3385956.3422116, doi = 10.1145 / 3385956.3422116Schreer, Oliver and Feldmann, Ingo and Ebner, et al, Advanced Volumetric Capture and Processing, SMPTE Motion Imaging Journal, 2019, vol. 128, no. 5, 18-24, doi=10.5594 / JMI.2019.2906835Schreer, Oliver and Feldmann, Ingo and Renault, et al, Capture and 3D Video Processing of Volumetric Video, 2019 IEEE International Conference on Image Processing (ICIP), 2019, 4310-4314. doi=10.1109 / ICIP.2019.8803576R. Newcombe et al., “DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time,” in IEEE Computer Vision and Pattern Recognition (CVPR), pp.343-352, 2015.Cao, Zhe, et al. “OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields.” arXiv preprint arXiv:1812.08008 (2018). The problem to be solved

[0007] The objective of the present invention is to solve the problems described above by providing a rigging method and system using 4D volumetric capture, which performs motion capture in 4D volumetric, operates a muscle simulator using motion capture information to express changes in the surface of a 3D human body model, and adjusts the position of the joints by comparing this with the volumetric model. means of solving the problem

[0008] To achieve the above objective, the present invention relates to a rigging method using 4D volumetric capture, comprising: (a) receiving a sequence of 3D mesh data consisting of a series of consecutive frames; (b) estimating a 3D pose consisting of joints and bones from the sequence of the 3D mesh; (c) estimating 3D mesh and pose data of an SMPL model for keyframes of the 3D mesh sequence; (d) generating a 3D base model of the keyframes by transferring the 3D mesh of the SMPL model; (e) performing a muscle simulation on the 3D base model; (f) generating a sequence of 3D poses and 3D mesh models of the estimated keyframes; and (g) adjusting joints by comparing the results of the muscle simulation with the 3D mesh model.

[0009] In addition, the present invention is a rigging method using 4D volumetric capture, characterized in that, in step (g), a change in the shape of a muscle according to movement is generated using the muscle simulator, a change in the surface of a 3D human body model is expressed using the change in the shape of the muscle, the 3D human body model with the changed surface is compared with the volumetric model, and the position of the joint is adjusted and updated using the difference between the two models.

[0010] In addition, the present invention is a rigging method using 4D volumetric capture, characterized in that, in step (b), projection images of a 3D mesh viewed from at least four directions are generated, the 2D joint positions in the projection images are extracted using the OpenPose library, and the 3D joint positions are generated through the calculation of intersection points in 3D.

[0011] In addition, the present invention is a rigging method using 4D volumetric capture, characterized in that, in step (f), a sequence of 3D poses of a series of keyframes is estimated by estimating the 3D pose of each keyframe from each 3D mesh of a series of keyframes; a sequence of 3D poses of keyframes of a new 3D pose is generated by editing the estimated sequence of 3D poses of keyframes; and a sequence of 3D poses of a whole frame is generated by estimating the 3D pose of a whole frame from the new 3D pose keyframes.

[0012] In addition, the present invention is a rigging method using 4D volumetric capture, wherein in step (f), the editing of the sequence of 3D poses is characterized by editing the sequence of 3D poses of keyframes, changing the order of the 3D poses, inserting other 3D poses in the sequence, or deleting existing 3D poses.

[0013] In addition, the present invention is a rigging method using 4D volumetric capture, characterized in that, in step (f), the 3D poses of intermediate frames are estimated by interpolation from keyframes of 3D poses to generate a sequence of 3D poses of the entire frame.

[0014] In addition, the present invention is a rigging method using 4D volumetric capture, characterized in that, in step (g), a 3D editing model is created by fitting a garment to the 3D base model, the 3D editing model is rigged with the pose of the SMPL model, and the 3D editing model is animated using motion retargeting.

[0015] In addition, the present invention is a rigging method using 4D volumetric capture, characterized in that, in step (g), retargeting is performed on a 3D editing model of a specific keyframe using a 3D pose sequence from the corresponding frame to the next frame.

[0016] In addition, the present invention is a rigging method using 4D volumetric capture, characterized in that, in step (g), the order of keyframes of the 3D editing model follows the order of the pose sequence. Effects of the invention

[0017] As described above, according to the rigging method and system using 4D volumetric capture according to the present invention, by comparing the results of muscle simulation based on motion capture information with a volumetric model and adjusting the position of the joint, the effect of detecting a more accurate joint is obtained. Brief explanation of the drawing

[0018] FIG. 1 is a block diagram of the configuration of the entire system for implementing the present invention. FIG. 2 is a flowchart illustrating a rigging system using 4D volumetric capture according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating a method for estimating the pose of a three-dimensional mesh according to an embodiment of the present invention. FIG. 4 is an example screen illustrating a process for estimating the pose of a three-dimensional mesh according to an embodiment of the present invention, showing (a) a three-dimensional volumetric sequence, (b) a projected image, and (d) an example screen for a two-dimensional pose image. FIG. 5 is an exemplary diagram illustrating the process of estimating a 3D pose in a 3D mesh according to an embodiment of the present invention, (a) a projected image of an AABB box, and (b) an exemplary diagram of a pose error. FIG. 6 is a diagram showing the input (2D image) and output (3D mesh in SMPL format) of an Expose deep learning model according to an embodiment of the present invention. Specific details for implementing the invention

[0019] Hereinafter, specific details for implementing the present invention will be explained with reference to the drawings.

[0020] In addition, in describing the present invention, identical parts are denoted by the same reference numerals, and their repeated description is omitted.

[0021] First, examples of the configuration of the overall system for implementing the present invention will be described with reference to FIG. 1.

[0022] As shown in FIG. 1(a), the rigging method using 4D volumetric capture according to the present invention (hereinafter the rigging method) can be implemented as a program system on a computer terminal (10) that receives a 3D mesh sequence as input and riggs it.

[0023] That is, the rigging method can be implemented using a program system (30) on a computer terminal (10), such as a PC, smartphone, or tablet PC. In particular, the rigging method can be configured as a program system and installed and executed on the computer terminal (10). The rigging method provides a service for rigging a three-dimensional mesh sequence using hardware or software resources of the computer terminal (10).

[0024] In addition, as another embodiment, as shown in FIG. 1(b), the rigging method may be executed by a server-client system composed of a rigging client (30a) and a rigging server (30b) on a computer terminal (10).

[0025] Meanwhile, the rigging client (30a) and the rigging server (30b) can be implemented according to the usual client and server configuration method. That is, the functions of the entire system can be distributed according to the performance of the client or the amount of communication with the server. Although described below as a rigging system, it can be implemented in various forms of distribution depending on the server-client configuration method.

[0026] Meanwhile, as another embodiment, the rigging method may be implemented by being configured as a program and operating on a general-purpose computer, or by being configured as a single electronic circuit such as an ASIC (Application-Specific Integrated Circuit). Alternatively, it may be developed as a dedicated computer terminal that exclusively processes the rigging of a color-stable 3D mesh sequence. Other possible forms may also be implemented.

[0027] Next, a rigging method using 4D volumetric capture according to an embodiment of the present invention will be described with reference to FIG. 2.

[0028] As shown in FIG. 2, the rigging method using 4D volumetric capture according to the present invention comprises the steps of receiving a 3D mesh sequence (S10), estimating a pose of the 3D mesh sequence (S20), generating an SMPL model from the 3D mesh sequence (S30), generating a 3D basic model (S40), performing muscle simulation (S50), and generating a human body surface based on the results of the muscle simulation and adjusting joints in comparison with the model sequence (S70). Additionally, the adjustment step (S70) is further composed of the steps of generating a human body surface (S71), comparing models (S72), and adjusting joints (S73). Additionally, the method may further include the step of generating a sequence of 3D poses (S60).

[0029] In summary, when 3D volumetric data generated from a multi-view camera is input, an SMPL model is created from the 3D volumetric data and transferred to the SMPL model to create a body. Clothing is then applied to the created body to generate a new 3D base model with the same body shape and face as the original model. Muscle simulation is performed on this 3D base model to generate a 3D human body surface, and the results are compared with the 3D sequence model to adjust the joints.

[0030] First, 3D volumetric data, that is, a 3D mesh sequence, is received as input (S10).

[0031] A 3D mesh sequence or 3D volumetric data is 3D video data of a person and consists of a 3D mesh of multiple frames in succession over time.

[0032] In particular, for a series of consecutive frames of images captured by multi-view cameras, a 3D mesh model is generated for each frame, and the 3D mesh sequence is a sequence of frames of the generated 3D mesh model.

[0033] The 3D mesh of each frame corresponds to a model (or body model) that represents a person as a mesh.

[0034] Meanwhile, a series of multiple frames can be divided into keyframes and intermediate frames. Keyframes are set by skipping a predetermined number of frames. For example, one keyframe can be set every 10 consecutive frames. Intermediate frames represent frames that exist between keyframes.

[0035] Next, 3D skeleton information consisting of the pose of the 3D model, in particular, joints (joints) and bones (skeleton), is estimated from the 3D mesh (S20).

[0036] As shown in FIGS. 3 and 4, first, to estimate the 3D pose of a 3D mesh, projection images (multiple views) of the 3D mesh viewed from multiple directions (four directions, such as front, back, left, and right) are generated (S21). Next, the 2D joint positions in the projection images are extracted using the OpenPose library (S22), and approximate 3D joint positions are generated through the calculation of intersection points in 3D (S23). Finally, a post-processing process is performed to extract high-precision 3D joint positions (S24).

[0037] Meanwhile, the pose of the 3D mesh is estimated for each frame.

[0038] First, the step (S21) of acquiring a projection image is described.

[0039] When estimating 2D joint positions using OpenPose from projection images from multiple directions, the accuracy of the joint position estimated from the image projected from the frontal direction may be the highest. Therefore, the frontal direction of the 3D mesh is identified by analyzing the spatial distribution of 3D coordinates of the points constituting the mesh, and the mesh is rotated so that the frontal direction is parallel to the Z-axis. Principal Component Analysis (PCA) is used to find the frontal direction. PCA is used to identify the principal components of the distributed data.

[0040] Applying PCA to a 3D mesh yields 3D vectors along the x, y, and z axes that most simply represent the mesh's distribution. Since the distribution along the y-axis (the vertical direction of the object) is not necessary to identify the front, the 3D mesh is projected onto the xz plane, and PCA is performed on this 2D plane. In PCA, the covariance matrix is ​​first determined, and two eigenvectors are derived from it. Among these two eigenvectors, the vector with the smaller eigenvalue represents the front direction. Using the vector found through PCA, the 3D mesh is rotated so that the front becomes the z-axis.

[0041] After finding the front of the object, an Axis-aligned Bounding Box (AABB) is set up to determine the projection plane in space. The process of projecting from 3D to a 2D plane is transformed from the world coordinate system to coordinates on the projection plane using a 4×4 matrix called the Model View Projection (MVP) matrix.

[0042] Next, a step (S22) of estimating a 2D pose in each projected 2D image is described.

[0043] When four projection images are generated, a 2D skeleton is extracted using OpenPose [Non-patent Literature 5].

[0044] OpenPose is a project presented at CVPR (IEEE Conference on Computer Vision and Pattern Recognition) 2017 and is a method developed at Carnegie Mellon University in the United States. Based on a Convolutional Neural Network (CNN), it is a library capable of extracting the features of multiple people's bodies, hands, and faces in real-time from photographs.

[0045] A key feature of this project is its ability to quickly identify the poses of multiple people. Prior to the release of OpenPose, a top-down approach was primarily used to estimate the poses of multiple individuals, which involved repeatedly detecting each person in a photograph and identifying their pose.

[0046] OpenPose is a type of bottom-up approach that improves performance without repetitive processing. The bottom-up method estimates the joints of every person, connects the positions of each joint, and then reconstructs the joint positions corresponding to each person. Generally, bottom-up methods face the problem of having to determine which person a joint belongs to. To address this, OpenPose utilizes Part Affinity Fields, which allow for the inference of which person a body part belongs to.

[0047] The result of extracting the skeleton using OpenPose is output as an image and a JSON (JavaScript Object Notation) file.

[0048] Next, the 3D pose generation step (S23) and the correction step (S24) through 3D intersection are described.

[0049] When the process of restoring from the 2D skeleton pixel coordinate system back to the 3D coordinate system is performed, the extracted joint coordinates are calculated on four projection planes located in space. When matching coordinates on the four planes are connected, four intersecting coordinates in space are obtained. Fig. 5(a) illustrates the extraction of the 3D joint of the left shoulder of a 3D body model.

[0050] Meanwhile, 2D pose estimation necessarily involves errors, and due to these errors, projection lines that extend beyond the intersection space occur. As exemplified in Fig. 5(b), the red projection line at the back can be seen extending beyond the intersection space when viewed from the front and side. Experimentally, the diameter of the intersection space is set to 3 cm. That is, if a virtual projection line does not pass through this space after defining a 3D virtual sphere, the nodes formed by this virtual projection line are not included in the calculation that integrates the 3D nodes.

[0051] In other words, the average point of the intersection space is set as the center, and only candidate coordinates within a predetermined range from the center (e.g., a sphere of diameter l) are selected, while other coordinates are excluded.

[0052] After defining points at each viewpoint for the 3D node using the unremoved candidate coordinates, the average coordinates are calculated. The (x, z) coordinates are determined from the top, and the y coordinates are determined from the side. The calculated (x, y, z) coordinates must match the (x, y) coordinates on the front face. This process is illustrated in Fig. 5(b).

[0053] Figure 4(c) shows the stiletto results for one frame visually displayed on a 3D model.

[0054] Meanwhile, the pose is obtained from a specific frame of the 3D mesh sequence or from the 3D mesh of each keyframe.

[0055] Next, an SMPL model is generated from a 3D mesh sequence (S30).

[0056] Using signal processing technology or a deep learning network, body posture and shape, face and hands are estimated from RGB images of 3D volumetric data to generate a 3D model using the SMPL-X method.

[0057] That is, a 2D image (RGB image) is obtained from the 3D mesh of a specific frame in a 3D mesh sequence (projected onto a 2D plane). Then, using signal processing or a deep learning model, a person's body, face, and hands are captured from the 2D image to generate 3D objects in SMPL-X format. An SMPL mesh model can be generated if an algorithm or deep learning network capable of generating 2D images and SMPL meshes is used.

[0058] This process is illustrated in Fig. 6. As an example, as shown in Fig. 6, an SMPL model can be estimated from a 2D image using an expose deep learning model, etc. ExPose (EXpressive POse and Shape rEgression) captures a person's body, face, and hands from a person's RGB image to generate a 3D Human in SMPL-X format [Non-patent Literature 7]. The network [is] a body pose (θ b ), hand pose(θ h ), facial pose(θ f Predicts ), shape(β), and expression(ψ).

[0059] SMPL (Skinned Multi-Person Linear Model) is a type of data format created to precisely represent the human body as a 3D mesh and is widely used in the fields of artificial intelligence and graphics. SMPL-X is a model of SMPL that includes fingers.

[0060] The SMPL model uses a single image to locate human bodies included in the image and estimates the body poses. After estimating the poses, it applies the estimated poses to a pre-defined human body model to output a human body model assuming the corresponding pose.

[0061] The SMPL model consists of shape parameters representing the external shape of the body and pose parameters representing joints and bones. That is, the SMPL model includes a 3D mesh of the external shape of the body and pose (bones and joints) information of the corresponding 3D mesh.

[0062] In addition, after analyzing the physical features in the video, the human body model is modified to have features similar to those features, thereby finally generating a human body model in the form of a 3D mesh similar to a human included in the video.

[0063] A two-dimensional image does not contain all the three-dimensional information that a 3D object or the human body originally possesses. Therefore, errors may inevitably exist in the three-dimensional information generated through inference or prediction from a two-dimensional image.

[0064] To minimize errors in the process of converting such 2D images into 3D information (3D mesh), 3D information of the SMPL model is extracted from each of the 2D images from multiple viewpoints, and the most suitable viewpoint is selected among them. That is, the confidence is calculated for each of the 2D images from multiple viewpoints. These are sorted in order of confidence, and the 2D image with the highest confidence is converted into 3D information (3D mesh). At this time, the 3D mesh of a specific frame of the volumetric image is projected onto various viewpoints to obtain 2D images from multiple viewpoints.

[0065] Meanwhile, the SMPL model can be generated from the 3D mesh of a specific frame or each keyframe. That is, the SMPL model is generated from a specific frame and used continuously, or, if necessary, newly generated from a keyframe and used.

[0067] Next, the head part separated from the 3D mesh is transferred to the head part of the SMPL model to create a 3D basic model (S40).

[0068] Using the previously separated mesh of the head part, transfer the volumetric head to the face of SMPL-X to create a new 3D model (or 3D base model) to be edited.

[0069] The method of transferring the face uses conventional methods [Non-patent Literature 6]. For example, methods such as replacing, morphing, or deforming the face can be applied. Alternatively, the face can be deformed using deep learning technology.

[0070] Meanwhile, preferably, for each keyframe, the head portion of the corresponding keyframe is transferred to the SMPL model to generate a 3D basic model of the corresponding frame.

[0071] Next, the step (S60) of generating a pose sequence is described.

[0072] That is, it generates a 3D pose of keyframes or a sequence of those keyframes. At this time, it can be generated through editing. It generates a sequence of 3D poses (of all frames) from the 3D poses of the keyframes.

[0073] Volumetric data is a sequence of three-dimensional meshes composed of a plurality of series of frames. Through the pose estimation step (S20) mentioned above, the three-dimensional pose of a specific frame can be estimated for the three-dimensional mesh of that frame.

[0074] First, the 3D pose of each keyframe is estimated from each 3D mesh of the series of keyframes to estimate the sequence of 3D poses of the series of keyframes. That is, the sequence of 3D poses of the series of keyframes can be estimated by estimating the 3D pose only for the keyframes. In this case, the 3D pose of the intermediate frame can be estimated by an interpolation method for the 3D pose of the keyframe.

[0075] Next, a new sequence of 3D pose keyframes is generated by editing the sequence of 3D poses of the estimated keyframes. Specifically, the sequence of 3D poses of the estimated keyframes is edited. The editing of the 3D pose sequence consists of editing the 3D pose skeleton and editing the sequence.

[0076] Skeleton editing of a 3D pose is editing the joints and bones of a specific frame's 3D pose. Since a 3D pose consists of joints and bones, it can be edited by changing the position of the joints or bones.

[0077] Editing a sequence of 3D poses involves changing the order of 3D poses, inserting other 3D poses, or deleting existing 3D poses. For example, if the keyframes are kf1, kf2, kf3, kf4, ..., kfn, you can change the order of some of them to kf1, kf3, kf4, kf2, ..., etc. Alternatively, you can import and insert a sequence of 3D poses that is provided in advance as a template. For example, by deleting the existing keyframe kf3 and inserting the sample sequence ks1, ks2, ks3, you can obtain a new sequence of kf1, kf2, ks1, ks2, ks3, kf4, ..., kfn.

[0078] Next, the 3D pose of the entire frame is estimated from the keyframes of the new 3D pose to generate a sequence of the 3D poses of the entire frame (hereinafter referred to as the sequence of edited 3D poses). That is, the 3D pose of the intermediate frame between the keyframes is estimated by interpolation. Each position of the bones and joints of the intermediate frame is estimated from the bones and joints of the keyframe by interpolation.

[0079] Next, the steps for adjusting the joints through muscle simulation (S71, S72, S73) are described.

[0080] That is, motion capture information is used to operate a muscle simulator. Then, the muscle simulator is used to generate changes in muscle shape according to movement, and changes in muscle shape are used to represent changes in the surface of a 3D human body model.

[0081] Compare the 3D human body model with the changed surface with the volumetric model. Then, use the difference between the two models to adjust and update the positions of the joints.

[0082] By repeating this process, you find a more precise joint.

[0083] Specifically, as shown in FIG. 2, first, a muscle simulator is operated using motion capture information (S71). Preferably, a muscle simulation is performed using a three-dimensional basic model.

[0084] Next, based on the results of the simulation, changes in the shape of the muscles according to the movement are generated (S72). Preferably, the task of generating changes in the shape of the muscles can be performed for each keyframe.

[0085] When creating changes in muscle shape based on a 3D basic model and rigging the 3D editing model, the pose data of the SMPL model is used.

[0086] Therefore, by rigging the SMPL-X skeleton onto the newly created 3D muscle model, a 3D model of an animable SMPL-X skeleton structure is finally created.

[0087] Next, the joints are adjusted by comparing the muscle model with the 3D sequence model (S73). That is, the 3D pose sequence generated in the previous pose sequence step (S60) is compared with the muscle model, and the joints are adjusted according to the result.

[0088] Meanwhile, a 3D base model is created for each keyframe, and a 3D muscle model is created by reflecting it. Then, the joints can be adjusted by reflecting the previously mentioned 3D pose sequence into the 3D muscle model. In this case, the 3D pose sequence is the sequence from the corresponding keyframe to the next keyframe.

[0089] In addition, the order of keyframes follows the order of the pose sequence. That is, based on the keyframes of the pose sequence, animation up to the next keyframe is performed using the 3D editing model of the corresponding keyframe. Therefore, if the order of keyframes in the pose sequence is changed by editing, animation is performed using the 3D editing model of the keyframes according to the changed order.

[0090] In addition, when new keyframes are inserted between keyframe k and keyframe k+1, the sequence of all frames up to the next keyframe (keyframe k+1) is animated using the 3D editing model of keyframe k.

[0091] A 3D mesh model is output over time for such movements. Then, by using the output 3D model to replace or insert the originally synthesized 3D volumetric data, an edited 3D volumetric mesh sequence can be produced. At this time, various effects can be created by adding new movements to the already produced 3D volumetric sequence (or video) or by changing clothing, accessories, hairstyles, makeup, etc.

[0092] Although the invention made by the inventor has been specifically described above according to the embodiments, the invention is not limited to the above embodiments and can be modified in various ways without departing from the gist thereof. Explanation of the symbols

[0093] 10 : Computer terminal 30: Rigging System 30a: Rigging Client 30b : Rigging Server 50 : Database 80 : Network

Claims

Claim 1 A rigging method using 4D volumetric capture, comprising: (a) receiving a sequence of 3D mesh data consisting of a series of consecutive frames; (b) estimating a 3D pose consisting of joints and bones from the sequence of 3D meshes; (c) estimating 3D mesh and pose data of an SMPL model for keyframes of the 3D mesh sequence; (d) generating a 3D base model of the keyframes by transferring the 3D mesh of the SMPL model; (e) performing a muscle simulation on the 3D base model; (f) generating a sequence of 3D poses and 3D mesh models of the estimated keyframes; and (g) adjusting joints by comparing the results of the muscle simulation with the 3D mesh model. Claim 2 A rigging method using 4D volumetric capture according to claim 1, characterized in that, in step (g), the muscle simulator is used to generate a change in the shape of the muscle according to movement, the change in the shape of the muscle is used to represent a change in the surface of a 3D human body model, the 3D human body model with the changed surface is compared with a volumetric model, and the difference between the two models is used to adjust and update the position of the joint. Claim 3 A rigging method using 4D volumetric capture according to claim 1, characterized in that, in step (b), projection images of a 3D mesh viewed from at least 4 directions are generated, 2D joint positions in the projection images are extracted using the OpenPose library, and 3D joint positions are generated through the calculation of intersection points in 3D. Claim 4 A rigging method using 4D volumetric capture according to claim 1, characterized in that, in step (f), the 3D pose of each keyframe is estimated from each 3D mesh of the series of keyframes to estimate a sequence of 3D poses of the series of keyframes; the estimated sequence of 3D poses of the keyframes is edited to generate a sequence of keyframes of a new 3D pose; and the 3D pose of the entire frame is estimated from the keyframes of the new 3D pose to generate a sequence of 3D poses of the entire frame. Claim 5 A rigging method using 4D volumetric capture, wherein in step (f) above, the editing of the sequence of 3D poses involves editing the sequence of 3D poses of keyframes, changing the order of the 3D poses, inserting other 3D poses in the sequence, or deleting existing 3D poses. Claim 6 A rigging method using 4D volumetric capture according to claim 4, characterized in that, in step (f), the 3D poses of intermediate frames are estimated by interpolation from the keyframes of the 3D poses to generate a sequence of 3D poses of the entire frame. Claim 7 A rigging method using 4D volumetric capture according to claim 1, characterized in that, in step (g), a garment is fitted to the 3D base model to create a 3D edited model, the 3D edited model is rigged with the pose of the SMPL model, and the 3D edited model is animated using motion retargeting. Claim 8 A rigging method using 4D volumetric capture according to claim 7, characterized in that, in step (g) above, retargeting a 3D editing model of a specific keyframe using a 3D pose sequence from the corresponding frame to the next frame. Claim 9 A rigging method using 4D volumetric capture according to claim 8, characterized in that, in step (g) above, the order of keyframes of the 3D editing model follows the order of the pose sequence.

Images