Unified field representation-based image-driven three-dimensional animatable asset generation method, equipment and program product
By using a unified field representation method, the problem that existing 3D generative models cannot generate high-fidelity geometry and reasonable skeletal structures is solved. This enables the generation of high-quality animated assets from a single image, improves the consistency of the generated results and the animation quality, and adapts to complex shapes and multi-category models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING WAZIDA TECH CO LTD
- Filing Date
- 2026-02-02
- Publication Date
- 2026-05-08
AI Technical Summary
Existing 3D generation technologies cannot generate animated assets that combine high-fidelity geometry, reasonable skeletal structure, and accurate skinning weights. They are particularly inadequate in complex shapes and multi-class models. Furthermore, existing methods have failed to effectively unify modeling, resulting in limitations in the consistency and functionality of the generated results.
A unified field representation-based approach is adopted to generate a sparse three-dimensional voxel representation, including geometric shape and skeletal structure, by extracting features from a single RGB image. The confidence decay mechanism is used to handle the ambiguity of skeletal connections, and a dual skin feature field is used to associate geometric and skeletal features to achieve unified modeling of shape, bone and skin weights.
It enables the direct generation of high-quality, animable 3D assets from a single image, improving the consistency of the generated results and the animation quality, enhancing the adaptability to complex shapes and multi-class models, and supporting efficient motion capture and physical simulation.
Smart Images
Figure CN121999097A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of 3D image generation technology, and in particular relates to a method, device and program product for generating image-driven 3D animable assets based on unified field representation. Background Technology
[0002] With the development of 3D generation technology, deep learning-based 3D generation models (such as NeRF, Triplane, and SparseVoxel) have made remarkable progress in recent years. These methods can generate high-fidelity static 3D assets from text or image input. However, these generated assets are usually limited to "static statues," lacking animability and failing to meet the practical needs of interactive graphics, virtual reality, game development, and animation production.
[0003] To address the animation challenges of 3D assets, numerous studies have attempted to imbue static 3D assets with animation capabilities through post-processing techniques such as automatic bone binding and skinning weights. For example, classic methods like Pinocchio embed template bones into static 3D models, while RigNet utilizes deep learning to predict bone and skinning weights. However, these methods rely on a clean topological structure of the 3D model, while the generated 3D assets often exhibit subtle geometric irregularities (such as incomplete surfaces and blurred volume boundaries), leading to failures in the automatic binding process. Furthermore, these post-processing techniques typically fail to generate bone structures that perfectly match the geometry, resulting in bone misalignment or motion distortion in the animation.
[0004] Existing technologies also directly generate animable 3D assets. For example, methods like UniRig, Anymate, and RigAnything generate animated models directly by creating skeletons and skinning weights. However, these methods are typically only applicable to specific categories (such as humans and animals) and struggle to handle multi-category or complex-shaped models. Furthermore, they often separate geometry generation from skeleton generation, failing to unify the modeling of shape, skeleton, and skinning weights, resulting in inconsistencies and limitations in the consistency and functionality of the generated results.
[0005] Existing technologies can be mainly divided into the following categories: 1. Static 3D Generation: Methods such as DreamFusion, TRELLIS, and TriplaneGaussian generate high-quality static 3D models from images or text. These methods excel in detail preservation and geometric accuracy, but the results are limited to static geometry and lack animation capabilities.
[0006] 2. Automatic binding technology includes: (1) Template-based binding: such as the Pinocchio method, which achieves animation by embedding predefined template skeletons into the 3D model. However, this method relies heavily on the topology of the input model and is difficult to handle geometric irregularities in the generated model.
[0007] (2) Binding based on deep learning: such as RigNet and DRiVE, which use neural networks to predict skeleton and skin weights, but these methods usually require a large amount of labeled data for training and have poor generalization ability for unseen categories.
[0008] (3) Direct generation of animation models: such as UniRig and Anymate, which directly generate animable models by generating bones and skinning weights. However, these methods usually separate geometry generation and animation generation, failing to unify the modeling, resulting in poor performance in terms of consistency and functionality of the generated results.
[0009] In implementing this embodiment, the inventors discovered the following drawbacks of the prior art: 1. Static models lack practicality: Existing static 3D generation methods (such as DreamFusion and TRELLIS) can generate high-quality geometry, but they lack animation capabilities and cannot be directly applied to scenes that require skeletal structure and skinning weights, such as games, animations and interactive applications.
[0010] 2. Unreliable post-processing: Methods that automatically bind bones and skin weights (such as Pinocchio and RigNet) have high requirements for the geometric integrity and topological accuracy of the generated model. However, the generated 3D assets usually have minor geometric defects, which can lead to post-processing failure or mismatch between the generated bones and the shape.
[0011] 3. Separation of generation and binding steps: Existing methods for directly generating animation models (such as UniRig and Anymate) usually separate geometry generation from bone and skinning generation, failing to effectively utilize the inherent correlation between the three, resulting in insufficient consistency in the generation results.
[0012] 4. Insufficient generalization ability for complex categories: Existing methods have limited ability to generate complex shapes (such as cartoon characters, mechanical equipment, and multi-legged animals), making it difficult to adapt to diverse application scenarios.
[0013] 5. Difficulty in fusing multimodal data: Existing methods fail to fully utilize information in the input image (such as color, texture and shape features) to guide the generation of skeletons and skins, resulting in a lack of consistency in the details of the generated model.
[0014] 6. Limited animation quality of generated results: Due to the lack of unified modeling, the skeletal structure generated by existing methods may have redundancy or missing elements in its topology, leading to problems such as clipping and distortion during motion capture or physical simulation. Summary of the Invention
[0015] To address the problems existing in the prior art, this invention provides a method, device, and program product for generating image-driven 3D animable assets based on unified field representation.
[0016] In a first aspect, embodiments of this disclosure provide an image-driven method for generating 3D animable assets based on a unified field representation, including: Feature extraction is performed on a single acquired RGB image to obtain multi-view features and global semantic features; A sparse three-dimensional voxel representation is generated based on multi-view features and global semantic features. The sparse three-dimensional voxel representation includes sparse voxels of geometric shape and sparse voxels of skeletal structure. Based on the sparse three-dimensional voxel representation, a unified field representation is generated by a structured encoder. The unified field representation includes a shape field, a skeleton field, and a skin field. Based on the unified field representation, generate 3D animable assets with geometric shape, skeletal structure and skin weights; The skeletal field employs a confidence decay mechanism to handle the ambiguity of skeletal connections, while the skin field uses dual skin feature fields to associate geometric features and skeletal features respectively.
[0017] Optionally, the feature extraction of the acquired single RGB image includes: A visual model is used to extract global semantic features, and multi-view projection technology is combined to extract multi-view features.
[0018] Optionally, the generation of sparse 3D voxel representations based on multi-view features and global semantic features includes a two-stage generation process: The first stage generates a sparse structural skeleton, including the geometry and sparse voxels that occupy space in the skeleton; The second stage generates the data needed for 3D geometry and animation.
[0019] Optionally, the shape field is a mapping from spatial voxels to local geometric parameters, and the shape field is constructed based on signed distance, normal vector, color, and shape detail interpolation weights. The skeletal field is a connection of offset vectors from a spatial point to the nearest joint and the parent node of the next higher level bone. The skin field implicitly models geometric and skeletal features using dual skin feature fields.
[0020] Optionally, the skin field implicitly models geometric and skeletal features using dual skin feature fields, including: For any point in space, geometric feature vectors and skeletal feature vectors are obtained separately through dual skin feature fields; The geometric feature vector and the bone feature vector corresponding to the same 3D point are fused and calculated to obtain the skinning weight of the 3D point for each bone.
[0021] Optionally, the formula for calculating skin weights is: , in, Represents vertices Subject to the Skin weight influenced by root skeleton It is the predicted peak temperature. These represent the vertex skin features and keypoint skin features predicted by the dual skin feature field, respectively. Indicates indexing along the skeleton Perform the Softmax normalization function.
[0022] Optionally, generating a 3D animable asset with geometry, skeletal structure, and skinning weights based on the unified field representation includes: The unified field representation is decoded into a 3D mesh, a skeletal hierarchy, and a skin weight distribution.
[0023] Optionally, the generation of 3D animable assets with geometric shape, skeletal structure, and skinning weights based on the unified field representation further includes, Based on joint density conditions, the number of joints is dynamically controlled and adjusted to generate the skeletal complexity and animation flexibility of 3D animable assets.
[0024] Secondly, embodiments of this disclosure also provide an electronic device, the electronic device comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to perform the image-driven 3D animable asset generation method based on unified field representation as described in any of the first aspects.
[0025] Thirdly, embodiments of this disclosure also provide a computer program product, including a computer program / instruction that, when executed by a processor, implements the image-driven 3D animable asset generation method based on unified field representation as described in any of the first aspects.
[0026] This invention provides a method, device, and program product for image-driven 3D animable asset generation based on unified field representation. The method addresses redundancy issues caused by geometric ambiguity in bone prediction through a confidence-decreasing bone field, and utilizes a dual-skin feature field to support the generation of any number of joints, enhancing adaptability to complex shapes. This achieves the goal of directly generating animable 3D assets from a single image that possess high-fidelity geometry, reasonable bone structure, and accurate skinning weights.
[0027] By constructing a two-stage generation process to generate sparse skeletons, the generated assets are ensured to have high consistency and high quality in terms of shape and animation functionality. Attached Figure Description
[0028] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.
[0029] Figure 1 A flowchart of an image-driven 3D animatable asset generation method based on unified field representation provided in this disclosure embodiment; Figure 2 A process flow diagram of an image-driven 3D animable asset generation method based on unified field representation in another scenario provided by embodiments of this disclosure; Figure 3 A schematic block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0030] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.
[0031] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0032] It should be noted that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice method. Furthermore, this device and / or practice method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.
[0033] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0034] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.
[0035] The technical problems solved by this implementation include: 1. The generated 3D assets lack animation capabilities: 3D models generated by existing technologies usually only provide geometric surface details, without skeletal structure and skin weights, and cannot achieve functions such as motion capture-driven or physical simulation.
[0036] 2. The post-processing automatic assembly process is unreliable: Attempts are made to add animation capabilities to the generated 3D model through post-processing (such as automatic bone binding and skinning weights), but these methods are easily affected by geometric topological noise, resulting in inconsistent bone structures, which leads to animation distortion or motion clipping problems.
[0037] 3. Existing methods are disconnected from animation steps: Existing technologies typically employ a sequential workflow of "generate first, then bind," which fails to fully utilize the inherent correlation between geometry and animation structure, resulting in limitations in functionality and consistency of the generated results.
[0038] 4. The generated models are difficult to handle complex and diverse categories: Existing methods have poor generalization ability for multiple categories and complex shapes (such as biological, mechanical, cartoon characters, etc.), making it difficult to generate high-quality animation models suitable for different application scenarios.
[0039] like Figure 1 As shown, this embodiment discloses an image-driven method for generating 3D animable assets based on a unified field representation, including: Feature extraction is performed on a single acquired RGB image to obtain multi-view features and global semantic features; Feature extraction is performed on the acquired single RGB image, including: A visual model is used to extract global semantic features, and multi-view projection technology is combined to extract multi-view features. The visual model can be the DINOv2 model.
[0040] Extracting global semantic features using a visual model specifically includes: Image segmentation: Dividing the preprocessed image into fixed-size image blocks (e.g., 16x16 pixels); Linear embedding: Each image patch is flattened and mapped to a high-dimensional vector (called "block embedding"); Add positional encoding: Add a positional information vector to each block embedding, because the Transformer itself does not contain spatial positional information; Transformer Encoder: These block embedding sequences with positional information are fed into a multi-layer Transformer encoder. Through a self-attention mechanism, each image block can interact with all other image blocks, thereby capturing global contextual relationships.
[0041] After being processed by the Transformer encoder, each image patch becomes a feature vector rich in contextual information.
[0042] A sparse three-dimensional voxel representation is generated based on multi-view features and global semantic features. The sparse three-dimensional voxel representation includes sparse voxels of geometric shape and sparse voxels of skeletal structure. For example, generating a 3D model of a lion based on a photograph of a lion.
[0043] Multi-view features are used to observe local details such as the muscle lines and hair direction of a lion in a photograph.
[0044] Global semantic features provide knowledge about lions: such as having four legs, one head, one tail, and what their body structure is like.
[0045] Generating sparse 3D voxels involves first building the skeleton of the lion's pose using skeletal sparse voxels, and then roughly filling in the lion's body blocks using geometric sparse voxels.
[0046] Based on the sparse three-dimensional voxel representation, a unified field representation is generated by a structured encoder. The unified field representation includes a shape field, a skeleton field, and a skin field. Although sparse 3D voxel representation is not precise, it fully contains information such as the basic 3D structure, proportions, and pose of an object.
[0047] Based on the unified field representation, generate 3D animable assets with geometric shape, skeletal structure and skin weights; The Unified Field method uses a sparse generation module to synchronously encode the geometric, skeletal, and skinning attributes of each spatial point in a shared 3D spatial domain. Through the information interaction mechanism within the sparse generation module, it enforces the functional consistency between these attributes, thereby enabling the end-to-end generation of high-quality, directly animable 3D assets from a single image.
[0048] The skeletal field employs a confidence decay mechanism to handle the ambiguity of skeletal connections, while the skin field uses dual skin feature fields to associate geometric features and skeletal features respectively.
[0049] Sparse 3D voxel representations are generated based on multi-view features and global semantic features, including a two-stage generation process: The first stage generates a sparse structural skeleton, including the geometry and sparse voxels that occupy space in the skeleton; The second stage generates the data needed for 3D geometry and animation.
[0050] The shape field is a mapping from spatial voxels to local geometric parameters. The shape field is constructed based on signed distance, normal vector, color, and shape detail interpolation weights. The skeletal field is a connection of offset vectors from a spatial point to the nearest joint and the parent node of the next higher level bone. The skin field implicitly models geometric and skeletal features using dual skin feature fields.
[0051] The skin field implicitly models geometric and skeletal features using dual skin feature fields, including: For any point in space, geometric feature vectors and skeletal feature vectors are obtained separately through dual skin feature fields; The geometric feature vector and the bone feature vector corresponding to the same 3D point are fused and calculated to obtain the skinning weight of the 3D point for each bone.
[0052] Based on the unified field representation, a 3D animable asset with geometric shape, skeletal structure, and skinning weights is generated, including: The unified field representation is decoded into a 3D mesh, a skeletal hierarchy, and a skin weight distribution.
[0053] Based on the unified field representation, a 3D animable asset with geometric shape, skeletal structure, and skinning weights is generated, which also includes: Based on joint density conditions, the number of joints is dynamically controlled and adjusted to generate the skeletal complexity and animation flexibility of 3D animable assets.
[0054] In a specific implementation scenario, such as Figure 2As shown, the method for generating animated assets specifically includes: Input a single RGB image (as input for generating the 3D asset).
[0055] Unified field representation: Sparse 3D voxel representation, including shape field, bone field and skin field; The skeletal field employs a confidence decay design to handle the ambiguity of skeletal connections; The skin field is represented using dual-field features (the correlation features between the shape field and the skeleton field).
[0056] Two-stage generation framework for sparse skeleton sampling: Phase 1: Generate a sparse structural skeleton (including geometry and sparse voxels that occupy space for the skeleton).
[0057] The second stage involves generating dense 3D geometry and the details required for animation, such as skin weights and joint connections.
[0058] The generated 3D assets include: geometry, such as a complete 3D mesh model; skeletal structure (containing joint nodes and their hierarchical relationships); and skinning weights (describing how each vertex follows the movement of the skeleton).
[0059] The shape field is defined as follows: The shape field S(v) maps each voxel v∈Vv \in Vv∈V to local geometric parameters: , Where d is the signed distance of the voxel corner, and n and c are the corner normal and color, respectively. It is the interpolation weight for shape details.
[0060] The skeletal field is defined as follows: The skeletal field B(x) maps a point in space to its nearest joint j(x) and its parent node p(x): B(x)=[(j(x) x)⊕(p(x) x)], Where ⊕ represents a vector connection, and x represents a point in three-dimensional space.
[0061] The skin field is defined as follows: Through dual skin feature fields W and Implicit modeling is performed on geometric and skeletal features respectively: ; The skinning weight of each vertex is calculated using the following formula: , in, This is the predicted peak temperature by W. , Indicates the inner product. express Predicted vertex skin features and joint skin features.
[0062] The specific process of the animated asset generation method is as follows: the input single image extracts multi-view features through the DINOv2 model and aligns them with sparse voxels in three-dimensional space.
[0063] Then, a sparse generation model is used to generate voxel meshes V and skeletal voxels of geometric shapes. .
[0064] Skeletal voxels handle fuzzy regions through a confidence decay mechanism: , The confidence level of the true value. Let be the coordinates of the vertex representing the depth in three-dimensional space. for nearest neighbor node for The second nearest neighbor.
[0065] Then, sparse coding generates a structured representation that combines shape, skeleton, and skin through a structured autoencoder.
[0066] Finally, a high-resolution 3D mesh model, skeletal structure, and skin weights are generated using the Transformer decoder.
[0067] The output 3D model includes geometry, animated bones, and skin weights, and supports user-defined control over joint density.
[0068] The performance parameters for this embodiment are as follows: 1. Joint accuracy: The predicted Chamfer distance for skeletal joints is 0.143 (lower than existing technology).
[0069] 2. Overall consistency: The Gromov-Wasserstein distance is significantly reduced (approximately 15% improvement over existing technologies).
[0070] 3. Skin weight distribution: KL divergence is 2.919 (approximately 30% less than existing technologies). Based on the above performance parameters, it can be seen that the method implemented in this embodiment has significantly improved all indicators compared to existing technologies.
[0071] The generated models exhibit consistency and high fidelity across complex categories such as animals, machines, and cartoon characters.
[0072] The structured autoencoder of this embodiment includes an encoder and a decoder for compressing and reconstructing shape, bone, and skin fields.
[0073] This embodiment has the following effects: 1. Higher consistency of generated results: Through a unified S3 field representation, geometry, skeletal structure and skin weights are jointly modeled and optimized in the same spatial domain, so that the three are mutually constrained and highly coordinated from the underlying feature level, thereby ensuring the inherent unity of the final generated assets in terms of form, structure and function.
[0074] 2. Enhanced adaptability: Employing a dual-skin feature field and confidence decay mechanism, it can handle the generation of assets with multiple categories and complex shapes, exhibiting stronger generalization capabilities.
[0075] 3. Significantly improved animation quality: The generated bone and skin weights are more accurate, the joint structure is more reasonable, and it supports high-quality motion capture and physical simulation.
[0076] 4. Higher efficiency: The two-stage generation process optimizes generation efficiency, especially reducing unnecessary computation in the sparse generation stage.
[0077] 5. Enhanced user interaction capabilities: Through the joint density control function, users can generate animation models with different degrees of flexibility according to their needs, meeting diverse application scenarios.
[0078] The electronic device disclosed in this embodiment includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.
[0079] The processor may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of this disclosure, the processor is used to execute computer-readable instructions stored in the memory, causing the electronic device to perform all or part of the steps of the image-driven 3D animable asset generation method based on unified field representation of the foregoing embodiments of this disclosure.
[0080] Those skilled in the art will understand that, in order to solve the technical problem of how to achieve a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this disclosure.
[0081] like Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. It illustrates a structural schematic diagram suitable for implementing the electronic device in the embodiment of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0082] like Figure 3 As shown, an electronic device may include a processing unit (such as a central processing unit, graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) or a program loaded from a storage device into random access memory (RAM). The RAM also stores various programs and data required for the operation of the electronic device. The processing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.
[0083] Typically, the following devices can be connected to the I / O interface: input devices, such as sensors or visual information acquisition devices; output devices, such as displays; storage devices, such as magnetic tapes or hard drives; and communication devices. Communication devices allow electronic devices to exchange data wirelessly or via wired communication with other devices, such as edge computing devices. Although Figure 3 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.
[0084] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, all or part of the steps of the image-driven 3D animable asset generation method based on unified field representation according to embodiments of this disclosure are performed.
[0085] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.
[0086] A computer-readable storage medium according to embodiments of the present disclosure stores non-transitory computer-readable instructions thereon. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the image-driven 3D animable asset generation method based on unified field representation according to the foregoing embodiments of the present disclosure are performed.
[0087] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).
[0088] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.
[0089] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0090] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.
[0091] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.
[0092] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.
[0093] Various changes, substitutions, and modifications can be made to the techniques described herein without departing from the teachings defined in this embodiment. Furthermore, the scope of this embodiment is not limited to the specific aspects of the processes, machines, manufacturing processes, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufacturing processes, events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein can be utilized. Therefore, this embodiment includes such processes, machines, manufacturing processes, events, means, methods, or actions within its scope.
[0094] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.
[0095] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.
Claims
1. A method for generating image-driven 3D animable assets based on unified field representation, characterized in that, include: Feature extraction is performed on a single acquired RGB image to obtain multi-view features and global semantic features; A sparse three-dimensional voxel representation is generated based on multi-view features and global semantic features. The sparse three-dimensional voxel representation includes sparse voxels of geometric shape and sparse voxels of skeletal structure. Based on the sparse three-dimensional voxel representation, a unified field representation is generated by a structured encoder. The unified field representation includes a shape field, a skeleton field, and a skin field. Based on the unified field representation, generate 3D animable assets with geometric shape, skeletal structure and skin weights; The skeletal field employs a confidence decay mechanism to handle the ambiguity of skeletal connections, while the skin field uses dual skin feature fields to associate geometric features and skeletal features respectively.
2. The image-driven 3D animable asset generation method based on unified field representation according to claim 1, characterized in that, The feature extraction of the acquired single RGB image includes: A visual model is used to extract global semantic features, and multi-view projection technology is combined to extract multi-view features.
3. The image-driven 3D animable asset generation method based on unified field representation according to claim 1, characterized in that, The generation of sparse 3D voxel representations based on multi-view features and global semantic features includes a two-stage generation process: The first stage generates a sparse structural skeleton, including the geometry and sparse voxels that occupy space in the skeleton; The second stage generates the data needed for 3D geometry and animation.
4. The image-driven 3D animable asset generation method based on unified field representation according to claim 1, characterized in that, The shape field is a mapping from spatial voxels to local geometric parameters. The shape field is constructed based on signed distance, normal vector, color, and shape detail interpolation weights. The skeletal field is a connection of offset vectors from a spatial point to the nearest joint and the parent node of the next higher level bone. The skin field implicitly models geometric and skeletal features using dual skin feature fields.
5. The image-driven 3D animable asset generation method based on unified field representation according to claim 4, characterized in that, The skin field implicitly models geometric and skeletal features using dual skin feature fields, including: For any point in space, geometric feature vectors and skeletal feature vectors are obtained separately through dual skin feature fields; The geometric feature vector and the bone feature vector corresponding to the same 3D point are fused and calculated to obtain the skinning weight of the 3D point for each bone.
6. The image-driven 3D animable asset generation method based on unified field representation according to claim 5, characterized in that, The formula for calculating skin weight is: ， in, Represents vertices Subject to the Skin weight influenced by root skeleton It is the predicted peak temperature. These represent the vertex skin features and keypoint skin features predicted by the dual skin feature field, respectively. Indicates indexing along the skeleton Perform the Softmax normalization function.
7. The image-driven 3D animable asset generation method based on unified field representation according to claim 1, characterized in that, The generation of 3D animable assets with geometric shape, skeletal structure, and skinning weights based on the unified field representation includes: The unified field representation is decoded into a 3D mesh, a skeletal hierarchy, and a skin weight distribution.
8. The image-driven 3D animable asset generation method based on unified field representation according to claim 1, characterized in that, The generation of 3D animable assets with geometric shape, skeletal structure, and skinning weights based on the unified field representation also includes, Based on joint density conditions, the number of joints is dynamically controlled and adjusted to generate the skeletal complexity and animation flexibility of 3D animable assets.
9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the image-driven 3D animable asset generation method based on unified field representation as described in any one of claims 1-8.
10. A computer program product comprising a computer program / instructions, characterized in that, When executed by a processor, the computer program / instruction implements the image-driven 3D animable asset generation method based on unified field representation as described in any one of claims 1-8.