Automatic blending of human facial expressions and full-body poses for dynamic digital human model creation using an integrated photo-video volumetric capture system and mesh tracking.

Abstract

An integrated photo-video volumetric capture system for 3D / 4D scanning acquires both 3D and 4D scans by simultaneously acquiring images and video. A volumetric capture system for high-quality 4D scanning and mesh tracking is used to establish topological correspondences across mesh sequences of 4D scans to generate corrected shapes for shape interpolation and skeleton-driven deformation. The volumetric capture system aids mesh tracking to maintain mesh registration (topological consistency) and facilitates extreme pose modeling. It is able to identify major upper and lower body joints, which are important for generating deformations and capturing the same deformation using a wide range of motion types across all joint categories. Topological changes are tracked using the volumetric capture system and mesh tracking. Each captured pose will have the same topological structure, making blending between multiple poses easier and more accurate.

Description

[0001] Cross-references to related applications

[0002] Pursuant to 35 U.S.SC §119(e), this application claims priority to U.S. Provisional Patent Application Serial No. 63 / 169,323, filed April 1, 2021, entitled “AUTOMATIC BLENDING OF HUMAN FACIAL EXPRESSION AND FULL-BODY POSES FOR DYNAMIC DIGITAL HUMAN MODEL CREATION USING INTEGRATED PHOTO-VIDEO VOLUMETRIC CAPTURE SYSTEM AND MESH-TRACKING,” which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] This invention relates to three-dimensional computer vision and graphics for the entertainment industry. More specifically, this invention relates to acquiring and processing three-dimensional computer vision and graphics for the creation of film, TV, music, and game content. Background Technology

[0004] Virtual human creation is highly manual, time-consuming, and expensive. A recent trend is to efficiently create realistic digital human models using multi-view camera 3D / 4D scanners, rather than handcrafting computer graphics (CG) artwork from scratch. Various 3D scanner studios (3Lateral, Avatta, TEN24, Pixel Light Effect, Eisko) and 4D scanner studios (4DViews, Microsoft, 8i, DGene) exist worldwide for camera-capture-based human digitization.

[0005] Photographic 3D scanner studios consist of multiple arrays of high-resolution photographic cameras. Existing 3D scanning techniques are typically used to create assembly models and require manual animation because they do not capture deformation. Video-based 4D scanner studios (4D = 3D + Time) consist of multiple arrays of high-frame-rate machine vision cameras. They capture natural surface dynamics, but due to the fixed video and motion, they cannot create novel facial expressions or body movements. Virtual actors would need to perform numerous sequences of movements, which translates to a tremendous workload for the actors. Summary of the Invention

[0006] An integrated photo-video volumetric capture system for 3D / 4D scanning acquires both 3D and 4D scans by simultaneously acquiring images and video. A volumetric capture system for high-quality 4D scanning and mesh tracking is used to establish topological correspondences across mesh sequences of 4D scans to generate corrected shapes for shape interpolation and skeleton-driven deformations. The volumetric capture system aids mesh tracking to maintain mesh registration (topological consistency) and facilitates extreme pose modeling. It is able to identify major upper and lower body joints, which are important for generating deformations and capturing the same deformation using a wide range of motion types across all joint categories. Topological changes are tracked using the volumetric capture system and mesh tracking. Each captured pose will have the same topological structure, making blending between multiple poses easier and more accurate.

[0007] In one aspect, a method of programming in the non-transitory memory of a device includes using a volumetric capture system configured for 3D and 4D scanning, comprising simultaneously capturing photographs and videos, wherein the 3D and 4D scans include detecting muscle deformations of an actor, and mesh generation based on the 3D and 4D scans. The 3D and 4D scans include: 3D scans for generating automated high-fidelity extreme poses and 4D scans comprising enabling mesh tracking to automatically register extreme pose meshes for blending. Generating automated high-fidelity extreme poses includes using the actor's 3D scans and the actor's muscle deformations to generate automated high-fidelity extreme poses. Topological correspondences are established across the mesh sequence of the 4D scans using the 4D scans and mesh tracking to generate corrected shapes for shape interpolation and skeleton-driven deformations. The method also includes identifying the actor's joints and muscles by the volumetric capture system used for 3D and 4D scans and targeting the actor's joints and muscles. Mesh generation includes muscle estimation or projection based on the 3D and 4D scans and machine learning. Mesh generation involves using 3D and 4D scans to generate meshes in extreme poses, including muscle deformation. The method also includes implementing mesh tracking to monitor topological changes so that each captured pose has the same topological structure for blending between poses.

[0008] In another aspect, an apparatus includes a non-transitory memory for storing an application used for: using a volumetric capture system configured for 3D and 4D scanning, including simultaneous capture of photographs and videos, wherein the 3D and 4D scans include detecting muscle deformations of an actor; and performing mesh generation based on the 3D and 4D scans; and a processor coupled to the memory, configured to process the application. The 3D and 4D scans include: 3D scans for generating automated high-fidelity extreme poses and 4D scans including enabling mesh tracking to automatically register extreme pose meshes for blending. Generating automated high-fidelity extreme poses includes using the actor's 3D scans and the actor's muscle deformations to generate automated high-fidelity extreme poses. 4D scans and mesh tracking are used to establish topological correspondences across a mesh sequence of 4D scans to generate corrected shapes for shape interpolation and skeleton-driven deformations. The application is also configured to identify and target the actor's joints and muscles using the volumetric capture system for 3D and 4D scans. Mesh generation involves muscle estimation or projection based on 3D and 4D scans and machine learning. Implementing mesh generation involves using 3D and 4D scans to generate meshes in extreme poses, including muscle deformation. The application is also configured to implement mesh tracking for tracking topology changes so that each captured pose can have the same topology for blending between poses.

[0009] In another aspect, a system includes a volumetric capture system for 3D and 4D scanning, comprising simultaneously capturing photographs and videos, wherein the 3D and 4D scanning includes detecting muscle deformations of an actor, and a computing device configured to: receive the captured photographs and videos from the volumetric capture system, and to perform mesh generation based on the 3D and 4D scans. The 3D and 4D scans include: 3D scans for generating automated high-fidelity extreme poses and 4D scans comprising enabling mesh tracking to automatically register extreme pose meshes for blending. Generating automated high-fidelity extreme poses includes using the actor's 3D scans and the actor's muscle deformations to generate automated high-fidelity extreme poses. Topological correspondences are established across the mesh sequence of the 4D scans using 4D scans and mesh tracking to generate corrected shapes for shape interpolation and skeleton-driven deformations. The volumetric capture system is also configured to identify the actor's joints and muscles using the volumetric capture system for 3D and 4D scanning and to target the actor's joints and muscles. Mesh generation includes muscle estimation or projection based on the 3D and 4D scans and machine learning. Mesh generation involves using 3D and 4D scans to generate meshes in extreme poses, including muscle deformation. The volume capture system is also configured to implement mesh tracking for tracking topology changes so that each captured pose can have the same topology for blending between poses. Attached Figure Description

[0010] Figure 1 The diagram illustrates a flowchart of a method for animate an object using a photo-video volume capture system according to some embodiments.

[0011] Figure 2 The diagram illustrates a mesh generated by combining neutral and extreme poses according to some embodiments.

[0012] Figure 3 The diagram illustrates the correlation between human anatomy and computer graphics according to some embodiments.

[0013] Figures 4A-4B The diagram illustrates muscle movement according to some embodiments.

[0014] Figure 5 Examples of major muscle groups according to some embodiments are illustrated.

[0015] Figure 6 The diagram illustrates joint-based movement types for mesh capture according to some embodiments.

[0016] Figure 7 The diagram illustrates joint-based movement types for mesh capture according to some embodiments.

[0017] Figure 8 Examples of extreme postures according to some embodiments are illustrated.

[0018] Figure 9 The illustration shows a diagram extracted according to some embodiments of automatic blendshape extraction.

[0019] Figure 10 The diagram illustrates a flowchart of mesh generation according to some embodiments.

[0020] Figure 11 The illustration shows a block diagram of an exemplary computing device configured to implement an automatic mixing method according to some embodiments. Detailed Implementation

[0021] The automated fusion system utilizes an integrated photo-video volumetric capture system for 3D / 4D scanning to acquire 3D and 4D scans by simultaneously acquiring images and video. 3D scans can be used to generate automated, high-fidelity extreme poses, while 4D scans include high temporal resolution that enables mesh tracking to automatically register extreme pose meshes for fusion.

[0022] A volume capture system (photo-video based) for high-quality 4D scanning and mesh tracking can establish topological correspondences across mesh sequences from 4D scans to generate corrected shapes that will be used for shape interpolation and skeleton-driven deformation. Unlike handcrafted shape modeling that aids in registration but involves manual shape generation, and 3D scan-based methods that aid in shape generation rather than registration, photo-video systems facilitate mesh tracking to maintain mesh registration (topological consistency) and are convenient for modeling extreme poses.

[0023] The method described herein is based on photo-video capture from a “photo-video volumetric capture system.” Photo-video capture is described in PCT / US2019 / 068151, filed December 20, 2019, entitled “PHOTO-VIDEO BASED SPATIAL-TEMPORAL VOLUMETRIC CAPTURESYSTEM FOR DYNAMIC 4D HUMAN FACE AND BODY DIGITIZATION,” which is incorporated herein by reference in its entirety for all purposes. As described, the photo-video capture system is capable of capturing high-fidelity textures sparsely over time, and between photo captures, video is captured, and the video can be used to establish correspondences (e.g., transitions) between sparse photos. This correspondence information can be used to implement mesh tracking.

[0024] The system can identify the main upper and lower body joints, which are important for generating deformabilities and capturing the same deformability using a wide range of motion across all joint categories and movement types. Joints can also be used for muscle deformability. For example, by understanding how joints move and how muscles near the joints deform, skeletal / joint information can be used for muscle deformability, which can then be used for mesh generation. Further refining the example, acquired images and videos can also be used with videos featuring muscle deformities, allowing for more accurate mesh generation of meshes with these deformed muscles.

[0025] By using a photo-video system and mesh tracking, topological changes can be tracked. Therefore, each captured pose will have the same topological structure, making blending between multiple poses easier and more accurate.

[0026] Figure 1The diagram illustrates a flowchart of a method for animate an object using a photo-video volumetric capture system according to some embodiments. In step 100, an integrated volumetric photo-video system is used to perform mesh creation / generation. Mesh generation includes extreme pose modeling and registration for blending. As described, the integrated photo-video volumetric capture system for 3D / 4D scanning acquires 3D and 4D scans by simultaneously acquiring images and videos of the object / actor. 3D scans can be used to generate automatic high-fidelity extreme poses, while 4D scans include high temporal resolution that enables mesh tracking to automatically register extreme pose meshes for blending. In step 102, skeleton fitting is performed. Skeleton fitting can be performed in any manner (e.g., based on relative marker trajectories). In step 104, skin weighting is performed. Skin weighting can be performed in any manner (e.g., determining the weight of each skin segment and drawing it accordingly). In step 104, animation is performed. Animation can be performed in any manner. Depending on the implementation, each step can be performed manually, semi-automatically, or automatically. In some embodiments, fewer or additional steps are implemented. In some embodiments, the order of steps has been modified.

[0027] Figure 2 The diagram illustrates a mesh generated by combining neutral and extreme poses according to some embodiments. A neutral pose can be any standard pose, such as standing with arms down, arms up, or arms outstretched to the side. An extreme pose is a pose between standard poses, such as when an object moves between standard poses. By capturing extreme poses as targets for specific parts of the human body's muscles, it becomes possible to generate extreme shapes for game development pipelines. Photo-video systems and mesh tracking can be used to capture and resolve the problem of maintaining mesh registration in a graphical game development pipeline by targeting all muscle groups of the human body.

[0028] When developing new video games, a capture model is used for the game. Actors typically enter the studio once to be recorded performing specific movements and / or actions. The studio uses a photo-video volumetric capture system to comprehensively capture all muscle deformations of the actors. Furthermore, by using existing types of human kinematic movements and deformations occurring in the human body, corresponding meshes can have similar deformations. Using previously captured neutral poses and additional captured poses, the system is able to deform the model to resemble human movement / deformation. Additionally, human kinematic movements, deformations, and / or other knowledge and data can be used to train the system.

[0029] Figure 3The diagram illustrates the correlation between human anatomy and computer graphics according to some embodiments. In human anatomy, musculoskeletal actuation involves receiving signals from the motor cortex of the human body. Muscle deformation then occurs, enabling joint / skeletal movement by the muscles pulling on the bones. Additionally, skin / fat movement occurs. In computer graphics meshes, motion drivers trigger the movement of an animated character, specifically by performing joint / skeletal movement. Mesh deformation (skeleton subspace deformation (SSD)) then occurs, followed by mesh deformation (pose space deformation (PSD)). A clear correlation can be seen between human anatomy and meshes generated using computer graphics.

[0030] Figures 4A-4B The diagram illustrates muscle movement according to some embodiments. As shown, body parts bend at joints, such as the head bending at the neck, the hand bending at the wrist, the fingers bending at the knuckles, the leg bending at the knee, and the foot bending at the ankle. In some embodiments, all joint movements can be categorized into 12 classes. In some embodiments, by classifying joint movements into categories, it is possible to generate correct muscle deformations based on the classified movements. For example, when a character bends at the knee, specific muscles in the leg deform, and using machine learning, it is possible to make the correct muscle deformations at the appropriate time. Muscle movement is the type of movement that the actor will perform, including the range of motion. Muscle movement is targeted for capture. Figures 4A-4B DeSaix, Peter, et al. "Anatomy & Physiology (OpenStax)" (2013). (Retrieved from https: / / openlibrary-repo.ecampusontario.ca / jspui / handle / 123456789 / 331)

[0031] Figure 5 The illustration shows examples of major muscle groups according to some embodiments. The upper and lower body each have four joints (excluding finger / toe joints). Joints in the upper body include: shoulder, elbow, neck, and hand, while joints in the lower body include: torso, hip, knee, and ankle. Each joint has a corresponding muscle group. As described, these corresponding muscle groups deform when the figure moves. When the actor is moving, the lower and upper body muscles are the primary targets for capture.

[0032] Figure 6 The diagram illustrates joint-based movement types for mesh capture according to some embodiments. Many different movement types exist, each with a different range of angular motion (0 to 180 degrees) for each major upper and lower joint. By including various movement types, it is possible to capture the desired muscle and then utilize it later when generating the mesh.

[0033] Figure 7 The diagram illustrates joint-based movement types for mesh capture according to some embodiments. Two of the 12 movement types (flexion / extension and pronation / supination) are shown. In some embodiments, the range of motion angles can be selected from 0, 90, and 180 degrees, while in some embodiments, finer adjustment of the range of motion angles can be achieved to specific degrees or even fractions of degrees.

[0034] Figure 8 Examples of extreme postures according to some embodiments are illustrated. Image 800 shows six types of movement, such as raising the arm upward to the side, raising the arm overhead from below the hip, and extending the arm forward. Image 802 shows four joints and target muscles.

[0035] Figure 9 The diagram illustrates an automated blending shape extraction according to some embodiments. Pose parameters 900, combined with facial motion units 902, result in a 4D tracking mesh 904. The automated blending shape extraction method uses a 4D scan of a moving face, which speeds up the character creation process and reduces production costs. 4D facial scanning methods are available, such as U.S. Patent Application No. 17 / 411,432, filed August 25, 2021, entitled “PRESERVING GEOMETRY DETAILS IN A SEQUENCE OF TRACKED MESHES,” which is incorporated herein by reference in its entirety for all purposes. As shown in 904, it provides a high-quality 4D tracking mesh of a moving face, and pose parameters 900 are also available from the tracked 4D mesh. Users can use control points or bones for pose representation. Figure 9 The central image is from P. Ekman, Wallace V. Friesen, Joseph C. Hager, “Facial action coding system: A technique for the measurement of facial movement >> Psychology 1978, 2002. ISBN 0-931835-01-1 1.”

[0036] Facial motion units are of interest. Given the availability of 4D-tracked meshes encompassing various expressions, it is possible to automatically generate a set of person-specific facial motion units. This can be viewed as a decomposition of the 4D mesh into dynamic pose parameters and static motion units, where only the motion units are unknown. Machine learning techniques used for this decomposition problem are applicable.

[0037] Figure 10The illustration shows a flowchart of mesh generation according to some embodiments. In step 1000, a volumetric capture system is used for high-quality 3D / 4D scanning. As described in PCT patent application PCT / US2019 / 068151, the volumetric capture system is capable of simultaneously acquiring photographs and videos for high-quality 3D / 4D scanning. High-quality 3D / 4D scanning includes denser camera views for high-quality modeling. In some embodiments, instead of utilizing a volumetric capture system, another system is used to acquire 3D content and temporal information. For example, at least two separate 3D scans are acquired. Further illustrating this, it is possible to capture and / or download individual 3D scans.

[0038] During capture time, joint and muscle movement and deformation are acquired. For example, specific muscles and their specific deformations over time can be captured. During capture time, specific joints and the muscles corresponding to those joints of an actor can be targeted. For example, a target object / actor can be requested to move, and the muscles will deform. Muscle deformation can be captured both statically and in motion. The information obtained from movement and deformation can be used to train a system that can use joint and muscle information to perform any movement of a character. This is difficult for animators to do in very complex cases. Any complex muscle deformation can be learned during the modeling phase. This enables compositing during the animation phase.

[0039] In step 1002, mesh generation is performed. Once high-quality information is captured for the scan, mesh generation is performed, including extreme pose modeling and registration for blending. 3D scan information can be used to generate automatic high-fidelity extreme poses. For example, 4D scan information, including frame information between keyframes, can be used to appropriately generate frames between keyframes. The high temporal resolution of 4D scan information enables mesh tracking to automatically register extreme pose meshes for blending. In another example, 4D scanning enables mesh generation of muscles that deform over time. Similarly, using machine learning involving joint information and corresponding muscle and muscle deformation information, meshes including muscle deformation information can be generated even when movement is not captured by the capture system. For example, although an actor is requested to perform a standing vertical jump and a run for capture, the capture system does not capture the actor performing the running jump. However, based on the acquired information of the standing vertical jump and the run (where the acquired information includes muscle deformation during these movements), and using machine learning with knowledge of joints and other physiological information, meshes for running jumps including detailed muscle deformation can be generated. In some embodiments, mesh generation includes muscle estimation or projection based on 3D and 4D scans and machine learning.

[0040] The ability to identify the main upper and lower body joints is important for generating deformities and capturing deformities using a wide range of motion across all joint categories and all movement types.

[0041] By using a volume capture system and mesh tracking, topological changes can be tracked. Therefore, each captured pose will have the same topological structure, making blending between multiple poses easier and more accurate. When generating the mesh, joints and muscles can be utilized as targets.

[0042] In some embodiments, mesh generation includes generating a static mesh based on 3D scan information and is capable of modifying / animating the mesh using 4D scan information. For example, as the mesh moves over time, additional mesh information can be established / generated from video content containing 4D scan information and / or machine learning information. As described, the transitions between each frame of the animated mesh maintain topological structure, resulting in smooth mesh tracking and blending. In other words, a topological correspondence is established across the mesh sequence from the 4D scans to generate a corrected shape that will be used for shape interpolation and skeleton-driven deformation.

[0043] In some embodiments, fewer or additional steps are implemented. In some embodiments, the order of the steps is modified.

[0044] Figure 11The illustration shows a block diagram of an exemplary computing device configured to implement an automated blending method according to some embodiments. The computing device 1100 is capable of acquiring, storing, computing, processing, transmitting, and / or displaying information such as images and videos. The computing device 1100 is capable of implementing any aspect of automated blending. Typically, a suitable hardware architecture for implementing the computing device 1100 includes a network interface 1102, memory 1104, a processor 1106, one or more I / O devices 1108, a bus 1110, and a storage device 1112. The choice of processor is not critical, provided a suitable processor with sufficient speed is selected. Memory 1104 can be any conventional computer memory known in the art. Storage device 1112 can include a hard disk drive, CD-ROM, CDRW, DVD, DVDRW, high-definition disc / drive, ultra-high-definition drive, flash memory card, or any other storage device. The computing device 1100 can include one or more network interfaces 1102. Examples of network interfaces include network interface cards (NICs) connected to Ethernet or other types of LANs. One or more I / O devices 1108 may include one or more of the following: keyboard, mouse, monitor, screen, printer, modem, touchscreen, button interface, and other devices. One or more automatic blending applications 1130 for implementing the automatic blending method are likely stored in storage device 1112 and memory 1104, and processed as applications are normally processed. Computing device 1100 may include... Figure 11 The components shown may be more or fewer. In some embodiments, automatic mixing hardware 1120 is included. Although Figure 11 The computing device 1100 includes an application 1130 and hardware 1120 for an automatic blending method, but the automatic blending method can be implemented on the computing device in hardware, firmware, software, or any combination thereof. For example, in some embodiments, the automatic blending application 1130 is programmed in memory and executed using a processor. In another example, in some embodiments, the automatic blending hardware 1120 is programmable hardware logic including gates specifically designed to implement the automatic blending method.

[0045] In some embodiments, one or more automated hybrid applications 1130 include a plurality of applications and / or modules. In some embodiments, a module further includes one or more sub-modules. In some embodiments, fewer or additional modules may be included.

[0046] Examples of suitable computing devices include personal computers, laptops, computer workstations, servers, mainframes, handheld computers, personal digital assistants, cellular / mobile phones, smart appliances, game consoles, digital cameras, digital camcorders, camera phones, smartphones, portable music players, tablets, mobile devices, video players, video disc writers / players (e.g., DVD writers / players, HD disc writers / players, UHD disc writers / players), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smartwatches), vehicles (e.g., autonomous vehicles), or any other suitable computing device.

[0047] To utilize the automatic blending method described herein, content is acquired using a device such as a digital camera / camcorder / computer, and then the same device or one or more additional devices analyze the content. The automatic blending method can be implemented with user assistance or automatically without user intervention to perform automatic blending.

[0048] In practice, the automatic blending method offers a more accurate and efficient approach to automatic blending and animation. Unlike handcrafted shape modeling, which aids in registration but involves manual shape generation, and 3D scan-based methods, which focus on shape generation rather than registration, the automatic blending method utilizes a photo-video system. This photo-video system facilitates mesh tracking to maintain mesh registration (topological consistency) and enables modeling of extreme poses. By using the photo-video system and mesh tracking, topological changes can be tracked. Therefore, each captured pose will have the same topological structure, making blending between multiple poses easier and more accurate.

[0049] Examples of automatic blending of human facial expressions and full-body poses for creating dynamic digital human models using an integrated photo-video volumetric capture system and mesh tracking.

[0050] 1. A method for programming in a non-transitory memory of a device, comprising:

[0051] A volumetric capture system configured for 3D and 4D scanning, including simultaneous capture of photographs and videos, is used, where the 3D and 4D scanning includes detecting muscle deformation in the actor; and

[0052] Mesh generation is achieved based on 3D and 4D scanning.

[0053] 2. The method according to Clause 1, wherein 3D scanning and 4D scanning include:

[0054] To be used for generating 3D scans of automatic high-fidelity extreme poses, and

[0055] This includes enabling mesh tracking to automatically register extreme pose meshes for use in blended, high temporal resolution 4D scans.

[0056] 3. The method described in Clause 2, wherein generating automatic high-fidelity extreme poses includes using 3D scans of actors and muscle deformations of actors to generate automatic high-fidelity extreme poses.

[0057] 4. The method according to Clause 2, wherein 4D scanning and mesh tracing are used to establish topological correspondences across a mesh sequence of 4D scans to generate corrected shapes for shape interpolation and skeleton-driven deformation.

[0058] 5. The method according to Clause 1 further includes using a volume capture system for 3D and 4D scanning to identify the actor's joints and muscles and to target the actor's joints and muscles.

[0059] 6. The method according to Clause 1, wherein mesh generation includes muscle estimation or projection based on 3D and 4D scans and machine learning.

[0060] 7. The method according to Clause 1, wherein mesh generation includes using 3D and 4D scanning to generate meshes in extreme poses including muscle deformation.

[0061] 8. The method according to Clause 1 further includes implementing mesh tracking for tracking topology changes so that each captured pose can have the same topology for blending between poses.

[0062] 9. An apparatus comprising:

[0063] Memory for storing non-transitory applications, which are used for:

[0064] A volumetric capture system configured for 3D and 4D scanning, including simultaneous capture of photographs and videos, is used, where the 3D and 4D scanning includes detecting muscle deformation in the actor; and

[0065] Mesh generation is achieved based on 3D and 4D scanning; and

[0066] A processor coupled to the memory, the processor being configured to process the application.

[0067] 10. The apparatus according to Clause 9, wherein 3D scanning and 4D scanning include:

[0068] To be used for generating 3D scans of automatic high-fidelity extreme poses, and

[0069] This includes enabling mesh tracking to automatically register extreme pose meshes for use in blended, high temporal resolution 4D scans.

[0070] 11. The apparatus according to Clause 10, wherein generating automatic high-fidelity extreme poses includes using 3D scans of actors and muscle deformations of actors to generate automatic high-fidelity extreme poses.

[0071] 12. The apparatus according to Clause 10, wherein 4D scanning and mesh tracking are used to establish topological correspondences across a mesh sequence of 4D scans to generate corrected shapes for shape interpolation and skeleton-driven deformation.

[0072] 13. The apparatus according to Clause 9, wherein the application is further configured to identify and target the actor's joints and muscles by a volume capture system for 3D and 4D scanning.

[0073] 14. The apparatus according to Clause 9, wherein mesh generation includes muscle estimation or projection based on 3D and 4D scanning and machine learning.

[0074] 15. The apparatus according to Clause 9, wherein mesh generation includes using 3D and 4D scanning to generate meshes in extreme poses including muscle deformation.

[0075] 16. The apparatus according to Clause 9, wherein the application is further configured to implement mesh tracking for tracking topology changes such that each captured pose can have the same topology for blending between poses.

[0076] 17. A system comprising:

[0077] A volumetric capture system for 3D and 4D scanning, which simultaneously captures photos and videos, including the detection of muscle deformation in actors; and

[0078] Computing device, the computing device being configured to:

[0079] Receive captured photos and videos from the volumetric capture system; and

[0080] Mesh generation is achieved based on 3D and 4D scanning.

[0081] 18. The system described in Clause 17, wherein 3D scanning and 4D scanning include:

[0082] To be used for generating 3D scans of automatic high-fidelity extreme poses, and

[0083] This includes enabling mesh tracking to automatically register extreme pose meshes for use in blended, high temporal resolution 4D scans.

[0084] 19. The system described in Clause 18, wherein generating automatic high-fidelity extreme poses includes using 3D scans of actors and muscle deformations of actors to generate automatic high-fidelity extreme poses.

[0085] 20. The system according to Clause 18, wherein 4D scanning and mesh tracing are used to establish topological correspondences across a sequence of meshes in the 4D scan to generate corrected shapes for shape interpolation and skeleton-driven deformation.

[0086] 21. The system according to Clause 17, wherein the volume capture system is further configured to identify the actor's joints and muscles and target the actor's joints and muscles by a volume capture system for 3D and 4D scanning.

[0087] 22. The system described in Clause 17, wherein mesh generation includes muscle estimation or projection based on 3D and 4D scans and machine learning.

[0088] 23. The system according to Clause 17, wherein mesh generation includes using 3D and 4D scanning to generate meshes in extreme poses including muscle deformation.

[0089] 24. The system according to Clause 17, wherein the volume capture system is further configured to implement mesh tracking for tracking topology changes such that each captured pose can have the same topology for blending between poses.

[0090] The invention has been described with reference to specific embodiments, including details, to facilitate an understanding of the principles of its construction and operation. Such references to specific embodiments and their details herein are not intended to limit the scope of the appended claims. It will be apparent to those skilled in the art that various other modifications can be made to the selected illustrative embodiments without departing from the spirit and scope of the invention as defined by the claims.

Claims

1. A method for programming in a non-transitory memory of a device, comprising: A volumetric capture system configured for 3D and 4D scanning is used, which includes the simultaneous capture of photos and videos, and the 3D and 4D scanning includes the detection of muscle deformation in the actor. as well as Mesh generation is achieved using 3D and 4D scanning. 3D scanning and 4D scanning include: To be used for generating 3D scans of automatic high-fidelity extreme poses, and This includes enabling mesh tracking to automatically register extreme pose meshes for blending and enabling high temporal resolution 4D scanning for generating meshes of muscles that deform over time.

2. The method of claim 1, wherein generating automatic high-fidelity extreme poses includes using 3D scans of actors and muscle deformations of actors to generate automatic high-fidelity extreme poses.

3. The method of claim 1, wherein 4D scanning and mesh tracing are used to establish topological correspondences across a mesh sequence of 4D scans to generate corrected shapes for shape interpolation and skeleton-driven deformation.

4. The method of claim 1, further comprising using a volume capture system for 3D and 4D scanning to identify the actor's joints and muscles and to target the actor's joints and muscles.

5. The method of claim 1, wherein mesh generation includes muscle estimation or projection based on 3D and 4D scans and machine learning.

6. The method of claim 1, wherein mesh generation comprises using 3D and 4D scanning to generate meshes in extreme poses including muscle deformation.

7. The method of claim 1, further comprising implementing mesh tracking for tracking topology changes such that each captured pose can have the same topology for mixing between poses.

8. An apparatus comprising: Memory for storing non-transitory applications, which are used for: A volumetric capture system configured for 3D and 4D scanning is used, which includes the simultaneous capture of photos and videos, and the 3D and 4D scanning includes the detection of muscle deformation in the actor. as well as Mesh generation is achieved using 3D and 4D scanning. 3D scanning and 4D scanning include: To be used for generating 3D scans of automatic high-fidelity extreme poses, and This includes high temporal resolution 4D scanning that enables mesh tracking to automatically register extreme pose meshes for blending and enables mesh generation of muscles that deform over time. as well as A processor coupled to the memory, the processor being configured to process the application.

9. The apparatus of claim 8, wherein generating automatic high-fidelity extreme poses comprises using 3D scans of actors and muscle deformations of actors to generate automatic high-fidelity extreme poses.

10. The apparatus of claim 8, wherein 4D scanning and mesh tracking are used to establish topological correspondences across a mesh sequence of 4D scans to generate corrected shapes for shape interpolation and skeleton-driven deformation.

11. The apparatus of claim 8, wherein the application is further configured to identify the actor's joints and muscles and target the actor's joints and muscles by a volume capture system for 3D and 4D scanning.

12. The apparatus of claim 8, wherein mesh generation comprises muscle estimation or projection based on 3D and 4D scanning and machine learning.

13. The apparatus of claim 8, wherein mesh generation comprises using 3D and 4D scanning to generate a mesh in extreme poses including muscle deformation.

14. The apparatus of claim 8, wherein the application is further configured to implement mesh tracking for tracking topology changes such that each captured pose can have the same topology for mixing between poses.

15. A system comprising: Volumetric capture systems used for 3D and 4D scanning include simultaneous capture of photos and videos, with 3D and 4D scanning including the detection of muscle deformation in actors; as well as Computing device, the computing device being configured to: Receive captured photos and videos from the volumetric capture system; as well as Mesh generation is achieved using 3D and 4D scanning. 3D scanning and 4D scanning include: To be used for generating 3D scans of automatic high-fidelity extreme poses, and This includes enabling mesh tracking to automatically register extreme pose meshes for blending and enabling high temporal resolution 4D scanning for generating meshes of muscles that deform over time.

16. The system of claim 15, wherein generating automatic high-fidelity extreme poses includes using 3D scans of actors and muscle deformations of actors to generate automatic high-fidelity extreme poses.

17. The system of claim 15, wherein 4D scanning and mesh tracking are used to establish topological correspondences across a mesh sequence of 4D scans to generate corrected shapes for shape interpolation and skeleton-driven deformation.

18. The system of claim 15, wherein the volume capture system is further configured to identify the actor's joints and muscles and target the actor's joints and muscles by the volume capture system for 3D and 4D scanning.

19. The system of claim 15, wherein mesh generation includes muscle estimation or projection based on 3D and 4D scanning and machine learning.

20. The system of claim 15, wherein mesh generation comprises using 3D and 4D scanning to generate meshes in extreme poses including muscle deformation.

21. The system of claim 15, wherein the volume capture system is further configured to implement mesh tracking for tracking topology changes such that each captured pose can have the same topology for blending between poses.

Images