Three-dimensional digital human generation method and device and electronic equipment

Abstract

The invention provides a three-dimensional digital human generation method and device and electronic equipment, and relates to the technical field of virtual reality. The method comprises the steps of obtaining feature data corresponding to a target object; the target object comprises an object main body and an attachment of the object main body; based on the feature data, determining a three-dimensional Gaussian model and a three-dimensional grid model corresponding to the target object; and obtaining a three-dimensional digital human corresponding to the target object based on the three-dimensional Gaussian model and the three-dimensional grid model.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to a method, apparatus and electronic device for generating a three-dimensional digital human. Background Technology

[0002] With the continuous advancement of Virtual Reality (VR) and Augmented Reality (AR) technologies, the creation and application of three-dimensional digital humans are becoming increasingly common, especially in fields such as entertainment, social interaction, education, and training. As an important subject of virtual interaction, the demand for digital humans is showing a rapid growth trend.

[0003] Currently, 3D digital humans generated by related technologies generally suffer from distortion when driven. Summary of the Invention

[0004] Based on this, the following technical solution is disclosed in this application: This application provides a method for generating a three-dimensional digital human. The method includes: acquiring feature data corresponding to a target object; the target object includes the object body and the objects attached to the object body; determining a three-dimensional Gaussian model and a three-dimensional mesh model corresponding to the target object based on the feature data; and obtaining a three-dimensional digital human corresponding to the target object based on the three-dimensional Gaussian model and the three-dimensional mesh model.

[0005] In some embodiments, obtaining feature data corresponding to a target object includes: obtaining a preset point cloud template corresponding to the object body and two-dimensional image data corresponding to the target object; wherein the preset point cloud template includes each coordinate point corresponding to the object body; determining weight information corresponding to each coordinate point in the preset point cloud template based on the two-dimensional image data; wherein the weight information is used to characterize the mapping relationship between each coordinate point in the preset point cloud template and the two-dimensional image data; and determining feature data corresponding to the target object based on the weight information.

[0006] In some embodiments, obtaining a preset point cloud template corresponding to the main body of an object includes: dividing the main body of an object into various human body parts; obtaining at least one sub-image corresponding to each human body part; determining a subset of point clouds corresponding to each human body part based on the sub-images corresponding to each human body part; and determining a preset point cloud template corresponding to the main body of the object based on the subset of point clouds corresponding to each human body part.

[0007] In some embodiments, determining the three-dimensional mesh model corresponding to the target object based on feature data includes: determining each coordinate point corresponding to the target object based on feature data; determining each mesh vertex corresponding to the target object based on the distance information from each coordinate point corresponding to the target object to the surface of the target object; and constructing the three-dimensional mesh model corresponding to the target object based on each mesh vertex.

[0008] In some embodiments, determining the three-dimensional Gaussian model corresponding to the target object based on feature data includes: inputting the feature data into a Gaussian decoder to obtain multiple Gaussian points corresponding to the target object and attribute information associated with each Gaussian point; wherein the attribute information is used to characterize the spatial relationship and appearance features of the multiple Gaussian points; and obtaining the three-dimensional Gaussian model corresponding to the target object based on the multiple Gaussian points and the attribute information.

[0009] In some embodiments, obtaining a 3D digital human corresponding to a target object based on a 3D Gaussian model and a 3D mesh model includes: determining the attribute information associated with each mesh vertex in the 3D mesh model based on multiple Gaussian points corresponding to the target object and the attribute information associated with each Gaussian point; and obtaining the 3D digital human corresponding to the target object based on the 3D mesh model and the attribute information associated with each mesh vertex.

[0010] In some embodiments, the method further includes: determining the skin weights associated with the 3D digital human based on a preset point cloud template; receiving a driving signal and generating a 3D digital human that matches the driving signal based on the driving signal and the skin weights.

[0011] In some embodiments, the method further includes: acquiring multi-view images of a 3D digital human corresponding to the target object; generating a texture map corresponding to the 3D digital human based on the multi-view images; wherein generating a texture map corresponding to the 3D digital human based on the multi-view images includes at least one of the following: generating a texture map corresponding to the 3D digital human based on the viewpoint information of the multi-view images and the visibility of the 3D digital human in the multi-view images; acquiring feature information corresponding to the multi-view images, reprojecting the feature information onto a 3D mesh model to obtain a multi-view feature set, and inputting the multi-view feature set into an image generation network to generate a texture map corresponding to the 3D digital human.

[0012] This application provides a three-dimensional digital human generation device, which includes: an acquisition module for acquiring feature data corresponding to a target object; the target object includes a main body and attachments to the main body; a determination module for determining a three-dimensional Gaussian model and a three-dimensional mesh model corresponding to the target object based on the feature data; and a generation module for obtaining a three-dimensional digital human corresponding to the target object based on the three-dimensional Gaussian model and the three-dimensional mesh model.

[0013] This application provides an electronic device, including a display device, at least one processor, and a target application running on the processor. The target application is capable of independently executing or invoking at least one artificial intelligence model to perform the following operations: Obtain the feature data corresponding to the target object; the target object includes the object body and the objects attached to the object body; based on the feature data, determine the three-dimensional Gaussian model and three-dimensional mesh model corresponding to the target object; based on the three-dimensional Gaussian model and three-dimensional mesh model, obtain the three-dimensional digital human corresponding to the target object; A display device for displaying a three-dimensional digital human corresponding to a target object.

[0014] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the steps of the three-dimensional digital human generation method provided in the above embodiments when executed by a processor.

[0015] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implement the steps of the three-dimensional digital human generation method provided in the above embodiments.

[0016] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of a three-dimensional Gaussian representation method in related technologies; Figure 2 It demonstrates the artifact phenomenon after digital human reconstruction; Figure 3 A schematic diagram showing the facial expressions of the digital human after it was powered on; Figure 4 This is a flowchart illustrating a three-dimensional digital human generation method provided in an embodiment of this application. Figure 1 ; Figure 5 This is a flowchart illustrating a method for obtaining feature data corresponding to a target object, as provided in an embodiment of this application. Figure 6 This is a flowchart illustrating a method for constructing a three-dimensional Gaussian model provided in an embodiment of this application; Figure 7 This is a flowchart illustrating a method for constructing a three-dimensional mesh model provided in an embodiment of this application; Figure 8 This is a flowchart illustrating a three-dimensional digital human generation method provided in an embodiment of this application. Figure 2 ; Figure 9 This is a flowchart illustrating a method for determining a preset point cloud template provided in an embodiment of this application; Figure 10 This is a flowchart illustrating a local partitioning method provided in an embodiment of this application; Figure 11 This is a flowchart illustrating a driving method for a three-dimensional digital human provided in an embodiment of this application; Figure 12 This is a flowchart illustrating a model optimization method provided in an embodiment of this application; Figure 13 This is an architecture diagram of a three-dimensional digital human generation method provided in an embodiment of this application; Figure 14 This is a logic block diagram of a three-dimensional digital human generation device provided in an embodiment of this application; Figure 15 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0019] Unless otherwise defined, the technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the application. In the following description, references to "some embodiments" describe a subset of all possible embodiments; however, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0020] It should also be noted that the terms "first, second, and third" used in the embodiments of this application are only used to distinguish similar objects and do not represent a specific order of objects. It can be understood that "first, second, and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0021] Furthermore, the reference to "embodiment" herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0022] With the continuous advancement of Virtual Reality (VR) and Augmented Reality (AR) technologies, 3D digital human technology has become a hot topic. Currently, 3D digital human generation methods include multi-view generation and single-view generation methods. Multi-view generation methods utilize images from multiple perspectives to generate 3D digital humans; however, these images often require multiple cameras to capture the same object, making widespread application difficult. Single-view generation methods use a single image to generate a 3D digital human, eliminating the need for multiple cameras and thus offering higher efficiency. However, the lack of information inherent in a single image results in a lack of detail in the generated 3D digital human, leading to poor quality.

[0023] In related technologies, large-scale 3D digital human reconstruction models require a large amount of data to support training, but currently, the number of high-quality digital human datasets is slightly insufficient compared to general 3D reconstruction datasets, both in terms of quantity and quality.

[0024] Currently, the mainstream 3D digital human representation in related technologies typically uses a 3D Gaussian representation method. Please refer to... Figure 1 , Figure 1 This is a schematic diagram of a three-dimensional Gaussian representation method in related technologies. Figure 1 The ellipse on the left side of the middle section represents a three-dimensional Gaussian point. Figure 1 The ellipsoid corresponding to each 3D Gaussian point on the right side contains appearance information such as color and texture, used to explicitly represent the geometric shape and appearance information of the digital human. It can be seen that the 3D Gaussian-based representation can represent the appearance of the digital human, but it cannot accurately represent the shape and geometric structure, thus easily leading to artifacts and other phenomena. Figure 2 As shown, Figure 2 It demonstrates the artifact phenomenon after digital human reconstruction.

[0025] Other related technologies have proposed schemes for generating 3D digital humans based on face-shaping technology. The algorithm of this scheme is as follows: a facial model with shape transformation parameters is established; a depth model is used to regress the facial shape features of a single image; and the regression results are used to change the shape parameters of the facial model, thus achieving face shaping. However, this method generates 3D digital humans that distinguish facial shape and expression using methods such as 3D Morphable Models (3DMMs). While the 3D digital humans perform reasonably well in static situations, a noticeable uncanny valley effect appears when using expression-driven data. For example... Figure 3 As shown, Figure 3 A schematic diagram showing the facial expressions of the digital human after it was powered up.

[0026] In summary, the 3D digital humans generated based on these technologies cannot accurately represent the shape and geometric structure of the digital human, are prone to artifacts, and cannot support dynamic driving. Furthermore, the docking engine for 3D digital humans based on 3D Gaussian representation is difficult to develop and thus hinders widespread application.

[0027] To address the aforementioned technical issues, this application provides a method for generating 3D digital humans. By combining the advantages of 3D Gaussian models in texture detail with the advantages of 3D mesh models in geometric structure, it is possible to generate 3D digital humans with high-quality surface details and accurate geometric structures. This improves reconstruction accuracy and facilitates integration with rendering engines and dynamic driving, thereby enhancing the accuracy and realism of the digital human.

[0028] The main technical concept of this solution is as follows: acquire a single 2D image, extract point cloud subsets corresponding to each human body part from a point cloud template, form a point cloud template based on the point cloud subsets, then find the required image features for each coordinate point in the point cloud template from the single 2D image to obtain feature data, then decode a 3D Gaussian model and a 3D mesh model from the feature data, wherein the 3D Gaussian model contains appearance information such as Gaussian points and the corresponding Gaussian attributes, and the 3D mesh model contains the main body of the object and the objects attached to the main body of the object. Finally, use Gaussian rendering to generate high-quality multi-view images, and map these multi-view images from 2D onto the 3D mesh model to obtain a 3D digital human.

[0029] The 3D digital humans generated using this solution are reconstructed through a combination of Gaussian and mesh models. This enhances the shape and structural information of the 3D digital humans while maintaining good texture, thus resolving the artifact problem inherent in traditional Gaussian models. Furthermore, driving the digital humans based on 3D mesh models results in more natural movements and expressions. Moreover, 3D digital humans constructed using 3D mesh models are easier to integrate with rendering engines and for wider application deployment.

[0030] The various embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0031] The execution entity of the three-dimensional digital human generation method provided in this application embodiment can be a three-dimensional digital human generation device. For example, the three-dimensional digital human generation method can be executed by a terminal device, a server, or other electronic devices. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, or wearable device, etc. In some possible implementations, the three-dimensional digital human generation method can be implemented by a processor calling computer-readable instructions stored in memory.

[0032] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating a three-dimensional digital human generation method provided in an embodiment of this application. Figure 1 The method includes: Step 401: Obtain the feature data corresponding to the target object.

[0033] The target object includes the object body and the objects attached to the object body.

[0034] In this embodiment, the target object refers to the entity object that needs to be reconstructed into a three-dimensional digital human, such as a human body, animal, or virtual character. The attachments to the target object can be elements such as clothing, accessories, or hairstyles attached to the main body of the object. The feature data corresponding to the target object typically includes image data, point cloud data, pose parameters, and other information used to describe the appearance and structure of the target object. The feature data corresponding to the target object can be extracted from a single two-dimensional image, or obtained from multi-view images or a depth sensor.

[0035] In this embodiment of the application, the feature data corresponding to the target object can be collected by a camera, the point cloud data can be obtained by a LiDAR, and the pose parameters can be extracted from the image by a pre-trained pose estimation model.

[0036] This application embodiment, by acquiring the data features corresponding to the target object, can more comprehensively describe the shape, texture, and dynamic characteristics of the target object, thereby improving the accuracy and realism of subsequent 3D modeling.

[0037] In one possible implementation, please refer to Figure 5 , Figure 5 This is a flowchart illustrating a method for obtaining feature data corresponding to a target object, provided in an embodiment of this application. The method includes: Step 501: Obtain the preset point cloud template corresponding to the main body of the object and the two-dimensional image data corresponding to the target object.

[0038] The preset point cloud template includes the coordinate points corresponding to the main body of the object.

[0039] Two-dimensional image data provides visual information about the subject, such as color, texture, and pose.

[0040] In some embodiments, two-dimensional image data can refer to one or more two-dimensional images, which can be static images or frame images extracted from video. Two-dimensional image data provides visual information about the object, such as color, texture, and pose. In embodiments of this application, two-dimensional image data includes, but is not limited to, visual information about the object corresponding to the target object and the objects attached to the object.

[0041] In this embodiment, the preset point cloud template refers to a predefined set of three-dimensional coordinates corresponding to each part of the target object, used as the basic structure for subsequent processing. Taking the human body as an example, the preset point cloud template can be a standardized human body model, such as the SMPLX model. The SMPLX model is a parametric human body model that can be used to describe changes in human posture, expression, and shape. The SMPLX model provides a standard human body topology and deformation rules, which is the key foundation for driving the digital human's movements and expressions in this solution. It should be noted that each vertex in the SMPLX model is configured with skinning weights.

[0042] In this embodiment, the SMPLX model can also be locally optimized to obtain a preset point cloud template according to the specific application scenario. The preset point cloud template not only contains the complete geometric structure, but also the topological relationship of each coordinate point corresponding to the main body of the object.

[0043] In some embodiments, the preset point cloud template can be obtained from a standard human body model (e.g., an SMPLX model). The preset point cloud template contains coordinate points of multiple key parts, thereby reflecting the overall shape and geometric details of the object.

[0044] In this embodiment of the application, by combining two-dimensional image data with a preset point cloud template, it is possible to assign more accurate spatial positioning and appearance information to each coordinate point in the preset point cloud template while maintaining the rationality of the main structure of the object, thereby improving the accuracy of the final modeling.

[0045] Step 502: Based on the two-dimensional image data, determine the weight information corresponding to each coordinate point in the preset point cloud template.

[0046] Step 503: Based on the weight information, determine the feature data corresponding to the target object.

[0047] The weight information is used to characterize the mapping relationship between each coordinate point in the preset point cloud template and the two-dimensional image data.

[0048] The feature data is a dataset generated by integrating the spatial structure information of a preset point cloud template and the image features of two-dimensional image data. It is used to represent the three-dimensional spatial features and appearance features of the object. In this embodiment, the feature data includes, but is not limited to, shape features, surface attribute features, and pose features. Shape features include, for example, vertex coordinates and normal vectors; surface attribute features include, for example, color, material, and texture; and pose features include, for example, joint angles, bone length, and skinning information.

[0049] In one implementation, the process of determining the feature data corresponding to the target object includes: matching two-dimensional image data with a preset point cloud template; finding one or more pixels in the two-dimensional image data that correspond to each coordinate point in the preset point cloud template; and establishing a correspondence between each coordinate point and its corresponding one or more pixels, thereby determining the weight information of each coordinate point. Based on this, image features corresponding to each coordinate point are obtained from the two-dimensional image data based on the weight information of each coordinate point. The image features corresponding to each coordinate point include information such as color, texture, and pose contained in one or more pixels. Finally, the feature data corresponding to the target object is obtained based on the image features corresponding to each coordinate point.

[0050] In this embodiment, the introduction of weight information enables the preset point cloud template to adapt more flexibly to two-dimensional images with different poses and perspectives, thereby improving the realism and accuracy of the three-dimensional digital human.

[0051] In another implementation, feature data corresponding to the target object can be obtained based on a deep learning model.

[0052] For example, two-dimensional image data and a preset point cloud template can be input into a deep learning model. The deep learning model can then be used to obtain the weight information corresponding to each coordinate point in the preset point cloud template, and the feature data corresponding to the target object can be obtained based on the weight information. The deep learning model can be, for example, a Transformer network model.

[0053] First, as shown in formula (1), image features can be extracted from two-dimensional image data based on the DINOv2 algorithm. Formula (1) Where I represents two-dimensional image data. ; For image encoders; These are image features.

[0054] Meanwhile, as shown in formula (2), the point cloud template can also be upgraded based on the multilayer perceptron (MLP) model.

[0055] Formula (2) in, It is a multi-layer sensing model; For masked point cloud data, ; This is the template for the upgraded point cloud.

[0056] Then, the Transformer network model can first use a self-attention network to extract the spatial information of each coordinate point in the point cloud template from its neighboring points, establishing correlation links between points. The attributes of neighboring points in 3D space have strong correlations. Therefore, the Transformer network model can learn these strong correlations to avoid isolated points or outliers with attributes significantly different from their neighbors, thus smoothing the image features corresponding to each coordinate point in the point cloud template and reducing the impact of noise.

[0057] Finally, the Transformer network model can interact with the point cloud template and the two-dimensional image data based on the cross-attention network to obtain the weight information corresponding to each coordinate point in the preset point cloud template, and determine the feature data corresponding to the target object based on the weight information. In this embodiment, the feature data corresponding to the target object can be represented by an implicit vector. For example, the implicit vector Z can be as shown in formula (3).

[0058] Formula (3) in, For image features, This is the template for the upgraded point cloud.

[0059] The implicit vector Z (feature data) encodes the local spatial geometry, color, and global semantic information of the point cloud template and 2D image data. This provides highly integrated information for the subsequent generation of 3D digital humans.

[0060] This application's embodiments map two-dimensional image data to a preset point cloud template and extract the weight information corresponding to each coordinate point in the preset point cloud template. This allows for the assignment of richer and more accurate appearance and three-dimensional spatial information to each coordinate point, enabling the feature data corresponding to the target object to more accurately reflect the object's morphological characteristics. Then, the weight information is used to obtain the target object's feature data, providing more accurate feature input for subsequent three-dimensional model construction. This facilitates refined modeling, completes the feature expression of the object's main body, effectively improves the expressive power of the object's main body's feature data, enhances the geometric rationality of the three-dimensional reconstruction, and improves the model's driving effect and realism. Therefore, it can be widely applied in fields such as games, film and television, and virtual reality. This provides more accurate feature input for subsequent three-dimensional model construction and improves reconstruction quality.

[0061] Step 402: Based on the feature data, determine the 3D Gaussian model and 3D mesh model corresponding to the target object.

[0062] In this embodiment, a three-dimensional Gaussian model is used to characterize the appearance features of the target object. The three-dimensional Gaussian model includes multiple Gaussian points, each containing attribute information such as position, scale, rotation, and opacity, effectively representing the surface texture and appearance details of the three-dimensional digital human.

[0063] In one possible implementation, the process of determining the 3D Gaussian model corresponding to the target object based on feature data includes: first, obtaining the image features corresponding to each coordinate point and the association relationship between each coordinate point and its neighboring points based on the feature data, where the association relationship is, for example, skinning weights. Then, initializing the positions of Gaussian points with the positions of each coordinate point to determine the Gaussian point cloud, which includes multiple Gaussian points. Based on the mapping relationship between each coordinate point and Gaussian point in the preset point cloud template and the image features corresponding to each coordinate point, determining the Gaussian attributes of each Gaussian point in the Gaussian point cloud. Finally, constructing a 3D Gaussian model based on the Gaussian attributes of each Gaussian point in the Gaussian point cloud.

[0064] In another possible implementation, a Gaussian decoder can be used to decode the feature data to construct a 3D Gaussian model. Please refer to [reference needed]. Figure 6 , Figure 6 This is a flowchart illustrating a method for constructing a three-dimensional Gaussian model provided in an embodiment of this application. Figure 6 As shown, the method includes steps 601 to 602.

[0065] Step 601: Input the feature data into the Gaussian decoder to obtain multiple Gaussian points corresponding to the target object and the attribute information associated with each Gaussian point.

[0066] Step 602: Based on multiple Gaussian points and attribute information, obtain the three-dimensional Gaussian model corresponding to the target object.

[0067] Among them, the Gaussian decoder is a neural network structure. In the embodiments of this application, feature data is input into the Gaussian decoder, and the Gaussian decoder is used to decode multiple Gaussian points corresponding to the target object from the feature data, and each Gaussian point is assigned associated attribute information.

[0068] Among them, the attribute information associated with Gaussian points is used to characterize the spatial relationship and appearance features of multiple Gaussian points.

[0069] In this embodiment, the attribute information associated with Gaussian points includes, but is not limited to, color, position (xyz), scale, rotation, and opacity. Each Gaussian point's attribute determines its representation in three-dimensional space; for example, opacity affects its visibility, while scale and rotation affect its shape and orientation. By combining multiple Gaussian points with their corresponding attribute information, a three-dimensional Gaussian model with rich detail and natural transitions can be constructed.

[0070] In this embodiment, the attribute information associated with Gaussian points also includes skin weights. The spatial relationship of Gaussian points is characterized based on the skin weights corresponding to each Gaussian point. Skin weights describe the degree of association between each Gaussian point and the skeleton in the 3D Gaussian model and are an important parameter for realizing dynamic actuation of the digital human.

[0071] It should be noted that, in this embodiment, the preset point cloud template is obtained based on a standard human body model (such as the SMSLX model), where each vertex carries skinning weights. Therefore, the preset point cloud template can obtain the skinning weights corresponding to each coordinate point based on the standard human body model. These skinning weights are typically generated from the SMSLX template using methods such as the k-Nearest Neighbor (KNN) algorithm, interpolation, and smoothing. Correspondingly, the feature data corresponding to the target object is obtained based on the preset point cloud template and two-dimensional image data. Therefore, the feature data corresponding to the target object carries skinning weight information corresponding to each coordinate point. Based on the mapping relationship between each coordinate point and Gaussian point in the preset point cloud template, the skinning weights corresponding to the Gaussian points can be obtained from the feature data.

[0072] In this embodiment, the 3D Gaussian model is a complete 3D representation constructed by combining all Gaussian points and their attribute information. It not only preserves the spatial relationships between multiple Gaussian points but also integrates appearance features such as color and material. Furthermore, to support dynamic actuation, the 3D Gaussian model incorporates skeletal rigging and skinning weight calculation, enabling the target object to respond to external control signals and perform operations such as facial expression changes and posture adjustments. This results in a 3D Gaussian model with realism and dynamic actuation capabilities.

[0073] It should be noted that the three-dimensional Gaussian model generated in this application embodiment, due to the introduction of the spatial relationship of Gaussian points, can perform bone binding and skin weight calculation, thereby more accurately capturing the surface texture and geometric structure of the target object, while avoiding the artifact problem common in traditional Gaussian models.

[0074] In this embodiment, a three-dimensional mesh model is used to characterize the three-dimensional spatial features of the target object. The three-dimensional mesh model is a discrete geometric structure composed of vertices, edges, and faces, which can be used to accurately describe the shape and spatial structure of a digital human. In this embodiment, the three-dimensional mesh model can also be called a digital human white model, which includes not only the main body of the object but also its attachments, but lacks visual features such as color and texture.

[0075] Based on the aforementioned techniques, a Gaussian point set containing attributes such as position, scale, rotation, and opacity is output through a Gaussian decoder, which can efficiently represent the appearance information of the target object. This method reduces computational complexity while maintaining visual quality, which is beneficial for real-time rendering and dynamic driving.

[0076] In one implementation, a Signed Distance Field (SDF) decoder can be used to decode the feature data to construct a 3D mesh model. Please refer to [reference needed]. Figure 7 , Figure 7 This is a flowchart illustrating a method for constructing a three-dimensional mesh model provided in an embodiment of this application. Figure 7 As shown, the method includes steps 701 to 703.

[0077] Step 701: Based on the feature data, determine the coordinate points corresponding to the target object.

[0078] Step 702: Based on the distance information from each coordinate point of the target object to the surface of the target object, determine each mesh vertex corresponding to the target object.

[0079] Step 703: Construct a 3D mesh model corresponding to the target object based on each mesh vertex.

[0080] In this embodiment, high-dimensional features in the feature data can be mapped to specific locations in three-dimensional space to determine the coordinate points corresponding to the target object. After identifying each coordinate point from the feature data, the distance information from each coordinate point to the surface of the target object can be obtained.

[0081] Among them, the distance information of each coordinate point to the surface of the target object is used to determine which coordinate points should be selected as the final mesh vertices, as well as the distance and position of each coordinate point relative to the surface of the target object.

[0082] As an example, coordinate points closer to the surface can be considered as mesh vertices; for coordinate points farther from the surface, they may be ignored or their positions adjusted to conform to the surface shape of the target object. This application's embodiments, by calculating distance information, can filter out the key points that best represent the geometry of the target object, thereby constructing a more accurate mesh structure.

[0083] In this embodiment, the distance information is a directed distance, wherein the distance information is negative when the point is inside the object, and positive when the point is outside the object's surface. The distance information is 0 when the point is on the object's surface.

[0084] For example, feature data can be input into an SDF decoder. The SDF decoder learns the spatial structure information implicit in the feature data corresponding to the target object, identifies each coordinate point in a preset point cloud template, and converts the feature data into SDF values ​​to represent the distance from each coordinate point to the object surface. The sign of the SDF value determines whether a vertex is inside or outside the target object's surface, while the numerical value represents the distance from the vertex to the model surface. When a point is inside the object, the SDF value is negative; when a point is outside the object's surface, the SDF value is positive. When a point is on the object's surface, the SDF value is 0. The surface of the target object corresponds to a surface with SDF=0.

[0085] Then, the FlexiCubes algorithm can be used to determine the individual mesh vertices corresponding to the target object based on the SDF values ​​of each coordinate point. The FlexiCubes algorithm is an algorithm for generating high-quality meshes from an SDF field. It can quickly construct a mesh structure that meets geometric requirements based on continuous implicit functions and is suitable for modeling and optimizing complex shapes.

[0086] For example, for each coordinate point, the FlexiCubes algorithm can first determine the sign of the coordinate point and its neighboring points. If the SDF values ​​of two coordinate points on an edge are one positive and one negative, it means that this edge intersects with the surface (isosurface) of the target object, and this intersection point is a potential edge point of the mesh. Then, the intersection point position is calculated through weighted linear interpolation to obtain the mesh vertex.

[0087] This application embodiment combines the SDF decoder with the FlexiCubes algorithm, and achieves high-quality 3D reconstruction of digital humans by extracting coordinate points based on feature data, determining mesh vertices using surface distance information, and constructing a 3D mesh model.

[0088] This application's embodiments locate mesh vertices by combining distance information, enabling a more accurate reconstruction of the target object's surface structure. Compared to traditional fixed mesh structure modeling methods, this method can adaptively adjust mesh density and distribution, thereby improving the geometric accuracy of the 3D model.

[0089] This application embodiment targets the feature data corresponding to the target object, obtaining Gaussian points and associated attribute information through a Gaussian decoder. A high-quality mesh structure is obtained by combining an SDF decoder with the FlexiCubes algorithm, generating a 3D mesh model, which prepares for the subsequent construction of a 3D digital human with high-fidelity geometry and rich texture details.

[0090] Step 403: Based on the three-dimensional Gaussian model and the three-dimensional mesh model, obtain the three-dimensional digital human corresponding to the target object.

[0091] In one implementation, appearance information can be obtained from a 3D Gaussian model, and the 3D mesh model can be rendered based on the appearance information to obtain a 3D digital human corresponding to the target object.

[0092] In another implementation, a 3D Gaussian model and a 3D mesh model can be fused. During the fusion process, Gaussian points and their associated attribute information, as well as the mapping relationship between Gaussian points and each mesh vertex of the 3D mesh model, are obtained from the Gaussian model. Based on this mapping relationship, the Gaussian attributes corresponding to each Gaussian point are used to render the 3D mesh model to obtain the 3D digital human corresponding to the target object.

[0093] For another implementation, please refer to Figure 8 , Figure 8 This is a flowchart illustrating a three-dimensional digital human generation method provided in an embodiment of this application. Figure 2 The method includes: Step 801: Based on the multiple Gaussian points corresponding to the target object and the attribute information associated with each Gaussian point, determine the attribute information associated with each mesh vertex in the 3D mesh model.

[0094] In the embodiments of this application, a Gaussian point is a three-dimensional Gaussian distribution point in a three-dimensional Gaussian model, and the attribute information associated with the Gaussian point is used to characterize the spatial relationship and appearance features of multiple Gaussian points.

[0095] As described above, there is a mapping relationship between each coordinate point in the preset point cloud template and the Gaussian points (hereinafter referred to as the first mapping relationship), and there is a mapping relationship between each coordinate point in the preset point cloud template and each grid vertex in the 3D mesh model (hereinafter referred to as the second mapping relationship). Based on the first and second mapping relationships, each Gaussian point in the 3D Gaussian model can be associated with each grid vertex in the 3D mesh model, thereby enabling the determination of the associated attribute information of the grid vertices based on the attribute information associated with the Gaussian points.

[0096] This application's embodiments achieve high-precision texture transfer by accurately mapping the attribute information of Gaussian points to mesh vertices. This ensures that the reconstructed 3D digital human is visually more realistic, improving the performance quality of the 3D digital human and thus better meeting the application needs of film, games, virtual reality, and other fields.

[0097] Step 802: Based on the 3D mesh model and the attribute information associated with each mesh vertex, obtain the 3D digital human corresponding to the target object.

[0098] In this embodiment of the application, after obtaining the attribute information associated with each grid vertex of the three-dimensional mesh model, the three-dimensional mesh model can be rendered based on the attribute information associated with each vertex, so that the digital human white model (three-dimensional mesh model) can carry appearance features and present the required three-dimensional digital human.

[0099] Among them, a 3D digital human refers to a drivable 3D human body model that combines high-quality geometric structure and rich texture information. It not only possesses a detailed appearance suitable for static display but can also be dynamically driven through driving signals. During dynamic driving, the 3D digital human can utilize skin weights to deform the 3D mesh model, allowing each vertex in the model to establish a connection with the skeleton. This maintains consistency between texture and shape during pose or facial expression changes, achieving natural facial expressions and posture adjustments. Furthermore, digital humans can generate multi-view images through differentiable rendering technology, further enhancing their application capabilities in virtual scenes.

[0100] This application embodiment integrates the attribute information associated with Gaussian points into a 3D mesh model, thereby enriching the visual details of the model while preserving its geometric structure. Compared to methods using Gaussian models or mesh models alone, this integration strategy effectively avoids artifacts and improves the visual quality and driving effect of the final digital human.

[0101] The 3D digital human generation method provided in this application involves inputting feature data into a Gaussian decoder to generate multiple Gaussian points and their attribute information corresponding to a target object, thereby constructing a 3D Gaussian model corresponding to the target object. Simultaneously, by inputting feature data into an SDF decoder and combining it with the FlexiCubes algorithm to construct a 3D mesh model, and finally combining the 3D Gaussian model with the 3D mesh model, the accuracy of the shape characteristics and geometric structure of the 3D digital human can be improved while maintaining rich texture representation. On the one hand, the 3D Gaussian model can obtain richer and more detailed appearance features; on the other hand, the 3D mesh model allows for fine-grained management of the shape and geometric structure of the 3D digital human, avoiding artifacts and other problems, and facilitating integration with common 3D mesh rendering engines, thereby enhancing the realism and dynamism of the 3D digital human. The 3D digital human provided in this application can be better applied to fields such as games, film and television, and virtual reality.

[0102] Based on the above embodiments, the process of obtaining a preset point cloud template will be described below. Please refer to... Figure 9 , Figure 9 This is a flowchart illustrating a method for determining a preset point cloud template according to an embodiment of this application. The method includes steps 901 to 904: Step 901: Divide the human body parts from the main body of the object.

[0103] Human body parts are the basic regions that make up a human body model, such as the head, torso, arms, and legs. By dividing the main body of the object into multiple human body parts, independent optimization of each part can be achieved. This allows for more accurate extraction of local information and provides more detailed and comprehensive information for subsequent processing, thus helping to improve the accuracy of 3D digital humans.

[0104] In this embodiment, a standard human body model (e.g., an SMSLX model) can be segmented to obtain the various human body parts of the subject. This ensures that the boundaries between the various human body parts are clear and logically consistent.

[0105] Step 902: Obtain at least one sub-image corresponding to each human body part.

[0106] In this context, a sub-image refers to a local image cropped or generated from the original image to focus on a specific part of the human body. Compared to the complete image, a sub-image focuses on a specific region, preserving more detailed information and facilitating subsequent processing modules to model or extract features from that specific part of the human body.

[0107] In this application embodiment, when dividing the human body into various parts from the main body, the division can be based on different viewpoints and different scales. For example, taking the head as an example, sub-images from multiple viewpoints such as front and side can be obtained, as well as sub-images containing the head and neck, sub-images containing the head and upper arms, sub-images containing the head and upper body, etc., thus obtaining multiple sub-images corresponding to the head. This application does not exhaustively list the viewpoints and scales used when dividing human body parts.

[0108] This application embodiment, by dividing the main body of the object into various human body parts and obtaining at least one sub-image corresponding to each human body part, can obtain more pixel information in the local area, thereby helping to improve the overall quality of the reconstructed three-dimensional digital human.

[0109] Step 903: Based on the sub-images corresponding to each human body part, determine the point cloud subset corresponding to each human body part.

[0110] A point cloud subset refers to a set of three-dimensional coordinate points obtained from local images, used to represent the geometric structure of a specific human body part.

[0111] In one implementation, a point cloud template corresponding to a standard human body model can be obtained, where the standard human body model is, for example, an initialized SMPLX model. For each sub-image, the sub-image is matched one by one with the point cloud template corresponding to the standard human body model. Multiple three-dimensional coordinate points of the human body parts corresponding to the sub-image are obtained from the point cloud template corresponding to the standard human body model. The set of these three-dimensional coordinate points is the point cloud subset corresponding to the sub-image.

[0112] In another implementation, deep learning models or other image processing algorithms can be used to extract key geometric features from each sub-image and convert these features into point cloud data, thus obtaining a subset of point clouds for each sub-image. For example, for a hand sub-image, a set of point cloud data can be generated to represent the shape of the fingers, joint positions, etc.

[0113] In another implementation, a local partitioning module can divide the point cloud into subsets from the initialized SMPLX model based on each sub-image. This local partitioning module can be, for example, a neural network model.

[0114] For example, for each sub-image, the sub-image undergoes feature extraction and enhancement via an image encoder to obtain image features. Simultaneously, the point cloud template corresponding to the standard human body model also undergoes feature extraction and enhancement via point cloud encoding (IQ) to obtain point cloud features. After encoder processing, the local segmentation module can more easily segment the corresponding locations.

[0115] Then, the local segmentation module predicts an image mask based on the image features using a cross-attention mechanism, and then applies the image mask to the point cloud features to obtain the point cloud data.

[0116] As shown in formula (4), the two-dimensional image data I will pass through the image encoder. Image features are obtained. Point cloud data P (a point cloud template corresponding to a standard human body model) will be processed by a point cloud encoder. Feature extraction and enhancement are performed to obtain point cloud features. The local segmentation module, based on the cross-attention mechanism, predicts an image mask based on image features and applies the image mask to the point cloud features to obtain point cloud data, which is the point cloud subset corresponding to the sub-image.

[0117] Formula (4) Where I represents two-dimensional image data; P represents the point cloud template corresponding to a standard human body model; For image encoders; σ is the point cloud encoder; D is the local partitioning module; σ is the activation function used to regress the values ​​to 0-1; M is the image mask; The data is masked point cloud data.

[0118] For example, please refer to Figure 10 , Figure 10 This is a flowchart illustrating a local partitioning method provided in an embodiment of this application. Figure 10 As shown, based on the same standard human body model, by dividing it locally, we can obtain full-body datasets, head datasets, half-body datasets, etc.

[0119] In this embodiment of the application, after obtaining the point cloud subset corresponding to each sub-image, it is necessary to fuse the sub-images and their point cloud subsets corresponding to the same human body part to eliminate overlap or conflict, retain as much detail information as possible, and obtain the final point cloud subset corresponding to the human body part.

[0120] The point cloud subsets corresponding to human body parts obtained in this way can acquire more detailed information and ensure that the boundaries between each human body part are clear and logically consistent.

[0121] Step 904: Based on the point cloud subsets corresponding to each human body part, determine the preset point cloud template corresponding to the main body of the object.

[0122] Among them, the preset point cloud template refers to a complete human body point cloud template formed by fusing and optimizing subsets of point clouds from various human body parts. The preset point cloud template not only contains a complete geometric structure but also has a unified topological relationship. The preset point cloud template can be used for subsequent driving and rendering.

[0123] In practical applications, subsets of point clouds from all human body parts can be stitched together, and overlapping or conflicting parts can be eliminated through optimization algorithms to ultimately generate a high-quality preset point cloud template. For example, a smoothing algorithm based on KNN interpolation can be used to make the transition between adjacent points in the preset point cloud template more natural.

[0124] This application embodiment divides the main body of the object into multiple human body parts and processes them separately, which can more precisely control the reconstruction quality of each part, thereby improving the overall performance of the digital human.

[0125] This application embodiment divides the main body of the human body into multiple human body parts, determines a point cloud subset for each sub-image corresponding to each human body part, and then fuses the point cloud subsets corresponding to each sub-image to determine the point cloud subset corresponding to each human body part. This method can enhance the realism of the details of each human body part and improve the accuracy of the three-dimensional spatial information of each human body part in the preset point cloud template. This can improve the human body shape of the reconstructed 3D digital human and its naturalness in dynamic driving, thereby better supporting the high-fidelity interactive experience of 3D digital humans in application scenarios such as games and virtual meetings.

[0126] Based on the above embodiments, this application also provides a method for driving a three-dimensional digital human. For example... Figure 11 As shown, Figure 11 This is a flowchart illustrating a driving method for a three-dimensional digital human provided in an embodiment of this application. The method includes: Step 1101: Determine the skin weights associated with the 3D digital human based on the preset point cloud template.

[0127] In this embodiment, the preset point cloud template is obtained based on a standard human body model. Therefore, the preset point cloud template is a standardized point cloud model containing a skeletal topology. The preset point cloud template can provide accurate joint positions, skin surface distribution, and the mapping relationship between bones and skin, thus serving as the basis for skin weight calculation.

[0128] Skinning weights are key parameters used in 3D animation or digital human driving to define the degree of influence of bones on model vertices. Each vertex is controlled by multiple bones, and the sum of the weights of these bones is 1. For example, each vertex is typically influenced by up to four bones, and the skinning weights determine the proportion of control that vertex receives from each bone. Appropriate weight allocation allows for smoother and more natural limb deformation in 3D digital humans during joint movements.

[0129] This application provides a method for determining skin weights based on a preset point cloud template. This method combines the KNN algorithm, interpolation method, and voxel smoothing technique to make the weight allocation more reasonable and effectively avoid the local discontinuity or distortion problems that may occur in 3D digital humans.

[0130] In practical applications, such as game development or virtual character creation, the method of determining skinning weights based on preset point cloud templates can quickly adapt to user input images of different body types, ensuring that the reconstructed 3D digital human corresponding to the target object maintains a natural and smooth posture during pose changes, without distortion or breakage. Furthermore, this method can effectively improve the generalization ability between different individuals, enabling the system to adapt to more diverse input data.

[0131] Step 1102: Receive the driving signal and generate a 3D digital human that matches the driving signal based on the driving signal and skin weights.

[0132] Actuating signals refer to external input information used to control the movements and expressions of a 3D digital human corresponding to a target object, including but not limited to shape parameters, posture parameters, and expression parameters. Actuating signals typically originate from real-time user input (such as motion capture devices, voice commands, facial expression capture, etc.), or they can be pre-recorded motion sequences or simulated data generated based on artificial intelligence technology.

[0133] There is a close data dependency between the skinning weights and the actuation signals. The actuation signals are used to trigger changes in the posture and shape of the 3D digital human. The skinning weights, on the other hand, are based on the actuation signals to finely control the posture changes of the 3D digital human, enabling it to exhibit natural and smooth movements.

[0134] Upon receiving the driving signal, the process involves two steps: firstly, parsing the signal to obtain key parameters such as joint angles and facial expression coefficients; secondly, matching these parameters with the skeletal structure of the 3D digital human corresponding to the target object; and thirdly, calculating the new position of each mesh vertex based on existing skinning weights and assigning appropriate skinning weights to the bones corresponding to each mesh vertex, ultimately generating a dynamic and interactive 3D digital human corresponding to the target object.

[0135] By introducing skinning weights and driving signals, the generated 3D digital human possesses dynamic performance capabilities. The 3D digital human generated based on the method provided in this application can support dynamic changes such as facial expressions and movements, contributing to a more natural interactive experience.

[0136] The digital human provided in this application not only supports the switching of static poses, but also supports the continuous playback of complex action sequences, achieving a highly realistic digital human performance.

[0137] This application's embodiments combine skin weights and driving signals to reduce computational overhead and improve rendering efficiency while ensuring natural motion.

[0138] Based on the above embodiments, this application also proposes a model optimization method for texture optimization of 3D digital humans. Please refer to... Figure 12 , Figure 12 This is a flowchart illustrating a model optimization method provided in an embodiment of this application. The method includes: Step 1201: Obtain multi-view images of the 3D digital human corresponding to the target object.

[0139] In this embodiment, images can be cropped from different perspectives based on the 3D digital human corresponding to the target object to obtain multi-view images. These multi-view images do not require multi-camera joint shooting, which is more efficient and facilitates widespread application.

[0140] Step 1202: Generate texture maps corresponding to the 3D digital human based on multi-view images.

[0141] In this embodiment of the application, after obtaining sufficient multi-view images, high-quality two-dimensional textures can be generated using the following two methods: The first approach is to generate texture maps corresponding to the 3D digital human based on the perspective information of multi-view images and the visibility of the 3D digital human in the multi-view images.

[0142] The viewpoint information includes, but is not limited to, the orientation, position, focal length, and pixel size of each multi-view image. Viewpoint information determines the mapping relationship between the two-dimensional image and the surface of the three-dimensional model, used to determine which position in the three-dimensional digital human body a point in the image corresponds to.

[0143] The visibility of a 3D digital human in multi-view images refers to whether each triangular facet / vertex on the surface of the 3D digital human can be captured (without occlusion) in a certain multi-view image. The visible area is the part of the digital human's surface that faces the camera and is not occluded by itself or other objects; the image pixels in this part can be directly used for texture sampling.

[0144] In this embodiment of the application, for each vertex of the visible area, its pixel coordinates in the corresponding image are calculated based on the viewpoint information. Image information such as color and texture details of the coordinates are extracted. Then, based on the two-dimensional texture mapping technology, these image information are assigned to the corresponding two-dimensional texture coordinates of the two-dimensional texture map to obtain the texture map corresponding to the three-dimensional digital human.

[0145] The second approach is to obtain feature information corresponding to multi-view images, reproject the feature information onto a 3D mesh model to obtain a multi-view feature set, and input the multi-view feature set into an image generation network to generate texture maps corresponding to the 3D digital human.

[0146] The feature information corresponding to multi-view images does not refer to the original pixel information of the image, but rather to the image features obtained after feature extraction and enhancement by the network model.

[0147] In this embodiment, multi-view images can be input into a convolutional neural network or a Transformer network to extract image features corresponding to the multi-view images. Then, these feature information is used to back-assign values ​​to a 3D mesh model to determine the image features corresponding to each mesh vertex. Ultimately, each surface element of the 3D mesh is associated with a set of features from different view images, i.e., a multi-view feature set. Finally, the multi-view feature set is input into an image generation network to obtain the texture map corresponding to the 3D digital human output by the image generation network.

[0148] The image generation model can be either a U-Net network or a Transformer network.

[0149] The two methods provided in this application significantly improve the detail representation of digital human surfaces by generating high-quality texture maps using multi-view images. Furthermore, both methods can be combined with differentiable rendering for subsequent optimization to further enhance texture clarity and cross-view accuracy. Figure 1 To the point of being responsive.

[0150] Based on the aforementioned techniques, generating high-quality texture maps using multi-view images can significantly improve the detail representation of digital human surfaces. Compared to single-view mapping methods, this strategy effectively reduces the impact of shadows and occlusion, thereby generating more natural and realistic texture effects.

[0151] Based on the above embodiments, the method for generating a three-dimensional digital human provided in this application will be described below in conjunction with specific scenarios. Please refer to... Figure 13 , Figure 13 This is an architectural diagram of a three-dimensional digital human generation method provided in an embodiment of this application. Figure 13 As shown: First, receive an input image and the corresponding SMPLEX driver information.

[0152] The SMPLX driving information includes parameters such as pose and expression parameters. Pose parameters are used to adjust the posture of the human joints, and include parameters such as hand rotation angle, body node rotation angle, and head pitch angle. Expression parameters are used to adjust the facial expressions of the 3D digital human.

[0153] Then, the local partitioning module partitions the input image and the SMPLX point cloud template (the point cloud module corresponding to the SMPLX model), extracts a subset of the point cloud, and constructs a preset point cloud template based on the subset of the point cloud.

[0154] Next, the Transformer network searches for the required image features for each coordinate point in the preset point cloud template from the input image and generates an implicit representation of the digital human.

[0155] A Gaussian decoder is used to decode the T-pose Gaussian digit from the implicit representation, where the T-pose Gaussian digit carries Gaussian properties and skin weights. An SDF decoder is then used to decode the complete and continuous SDF field from this implicit representation. The FlexiCubes algorithm is then invoked, which calculates the corresponding T-pose mesh representation based on the SDF field, thereby constructing the mesh structure and obtaining a 3D mesh model. Finally, based on the Gaussian properties and skin weights, the mesh model is rendered, mapped, and initial skin weights are assigned to obtain the initial 3D digit.

[0156] Finally, the motion of the 3D digital human is driven by the information from the SMPLX drive system.

[0157] In this embodiment of the application, after obtaining the three-dimensional digital human, the posture parameters and expression parameters in the SMSLX driving information are used to control the three-dimensional digital human to present the posture corresponding to the posture parameters and the expression corresponding to the expression parameters.

[0158] It should be noted that the essence of using SMSLX driving information to drive the movement of a 3D digital human is to use SMSLX driving information to control the posture of each mesh vertex in the 3D mesh model corresponding to the 3D digital human, and to assign new skinning weights to each mesh vertex during the posture change process, so as to make the movement of the 3D digital human smoother and more natural.

[0159] In this embodiment, high-quality multi-view images can also be rendered based on Gaussian properties, and these high-quality multi-view images can be mapped from two dimensions to a three-dimensional mesh model to optimize the 3D digital human. In this embodiment, the 3D digital human can also be output in FBX format for easy integration with the rendering engine.

[0160] FBX format is a cross-platform 3D data exchange file format widely used for storing model, animation, and material data. When a 3D digital human is output in FBX format, an FBX file is obtained. FBX files can contain the 3D digital human's geometry, texture maps, skeletal animation, lighting, camera positions, and even audio and video elements, ensuring that no critical information is lost during data transmission. In practical applications, FBX files can be directly interfaced with common 3D mesh rendering engines.

[0161] In summary, the embodiments of this application, by introducing high-precision point cloud templates, combining the complementary advantages of Gaussian models and mesh models, and employing advanced deep learning architectures for feature extraction and decoding, have achieved high-quality, drivable 3D digital human generation. This solves the problems of low accuracy, poor geometric structure, and unsatisfactory driving effects in related solutions, and has broad application prospects.

[0162] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps; or steps from different embodiments may be combined into a new technical solution.

[0163] In another embodiment of this application, a three-dimensional digital human generation device is provided; please refer to [reference needed]. Figure 14 , Figure 14 This is a logic block diagram of a three-dimensional digital human generation device provided in an embodiment of this application. The three-dimensional digital human generation device 1400 may include: an acquisition module 1401, a determination module 1402, and a generation module 1403. Wherein: The acquisition module 1401 is used to acquire feature data corresponding to the target object; the target object includes the object body and the attachments of the object body; The determination module 1402 is used to determine the three-dimensional Gaussian model and three-dimensional mesh model corresponding to the target object based on feature data; The generation module 1403 is used to obtain a 3D digital human corresponding to the target object based on a 3D Gaussian model and a 3D mesh model.

[0164] In one embodiment, the acquisition module 1401 is specifically used to: acquire a preset point cloud template corresponding to the object body and two-dimensional image data corresponding to the target object; wherein, the preset point cloud template includes each coordinate point corresponding to the object body; Based on two-dimensional image data, the weight information corresponding to each coordinate point in the preset point cloud template is determined; wherein, the weight information is used to characterize the mapping relationship between each coordinate point in the preset point cloud template and the two-dimensional image data; Based on weight information, the feature data corresponding to the target object is determined.

[0165] In one embodiment, the acquisition module 1401 is specifically used for: Divide the main body of the object into different parts of the human body; Obtain at least one sub-image corresponding to each human body part; Based on the sub-images corresponding to each human body part, determine the point cloud subsets corresponding to each human body part. Based on the point cloud subsets corresponding to each part of the human body, a preset point cloud template corresponding to the main body of the object is determined.

[0166] In one embodiment, the determining module 1402 is specifically used for: Based on feature data, determine the coordinate points corresponding to the target object; Based on the distance information from each coordinate point of the target object to the surface of the target object, determine each mesh vertex corresponding to the target object; Based on each grid vertex, construct a 3D mesh model corresponding to the target object.

[0167] In one embodiment, the determining module 1402 is specifically used for: The feature data is input into the Gaussian decoder to obtain multiple Gaussian points corresponding to the target object and the attribute information associated with each Gaussian point; the attribute information is used to characterize the spatial relationship and appearance features of multiple Gaussian points. Based on multiple Gaussian points and attribute information, a three-dimensional Gaussian model corresponding to the target object is obtained.

[0168] In one embodiment, the generation module 1403 is specifically used for: Based on multiple Gaussian points corresponding to the target object and the attribute information associated with each Gaussian point, the attribute information associated with each mesh vertex in the 3D mesh model is determined. Based on the 3D mesh model and the attribute information associated with each mesh vertex, a 3D digital human corresponding to the target object is obtained.

[0169] In one embodiment, the generation module 1403 is specifically used for: Based on a preset point cloud template, determine the skin weights associated with the 3D digital human; It receives driving signals and generates a 3D digital human that matches the driving signals based on the driving signals and skin weights.

[0170] In one embodiment, the generation module 1403 is specifically used for: Acquire multi-view images of the 3D digital human corresponding to the target object; Based on multi-view images, generate texture maps corresponding to 3D digital humans; Among them, the texture maps generated based on multi-view images for the 3D digital human include at least one of the following: Based on the perspective information of multi-view images and the visibility of 3D digital humans in multi-view images, generate texture maps corresponding to 3D digital humans. The feature information corresponding to multi-view images is obtained, and the feature information is reprojected onto a 3D mesh model to obtain a multi-view feature set. The multi-view feature set is then input into an image generation network to generate a texture map corresponding to a 3D digital human.

[0171] Each module in the aforementioned 3D digital human generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0172] In another embodiment of this application, an electronic device is provided, comprising a display device, at least one processor, and a target application running on the processor. The target application is capable of independently executing or calling at least one artificial intelligence model to perform the following operations: acquiring feature data corresponding to a target object; the target object includes an object body and attachments to the object body; determining a three-dimensional Gaussian model and a three-dimensional mesh model corresponding to the target object based on the feature data; and obtaining a three-dimensional digital human corresponding to the target object based on the three-dimensional Gaussian model and the three-dimensional mesh model. This display device is used to display a three-dimensional digital human corresponding to a target object.

[0173] It should be noted that, in the embodiments of this application, the target application can be an intelligent agent or an artificial intelligence assistant. The at least one artificial intelligence model is, for example, the image generation model used in the foregoing embodiments.

[0174] In this embodiment, the target application can also independently execute or invoke at least one artificial intelligence model to execute the three-dimensional digital human generation method described in any of the foregoing embodiments.

[0175] Please refer to Figure 15 , Figure 15This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a communication interface 1501, a memory 1502, a processor 1503, and a display device 1505; the various components are coupled together through a bus system 1504. It is understood that the bus system 1504 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 1504 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 15 The general labeled all buses as Bus System 1504.

[0176] In this embodiment, the communication interface 1501 is used to send and receive information with other external devices; the memory 1502 is used to store computer programs that can run on the processor 1503; the processor 1503 is used to execute the steps of the three-dimensional digital human generation method described in any of the foregoing embodiments when running the computer program.

[0177] In some implementations, the computer program is, for example, a program running on the processor 1503 corresponding to a target application, which is capable of independently executing or invoking at least one artificial intelligence model to execute the steps of the three-dimensional digital human generation method described in any of the foregoing embodiments.

[0178] In this embodiment of the application, the display device 1505 is used to display a three-dimensional digital human corresponding to the target object.

[0179] It is understood that the memory 1502 in this embodiment of the application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 1502 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0180] The processor 1503 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 1503 or by software instructions. The processor 1503 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 1502. Processor 1503 reads the information in memory 1502 and completes the steps of the above method in conjunction with its hardware.

[0181] It is also understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0182] For software implementation, the techniques described herein can be implemented through modules (e.g., procedures, functions, etc.) that perform the functions described herein. Software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or externally. Wherein, if implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] In another embodiment of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the three-dimensional digital human generation method described in the foregoing embodiments.

[0184] In another embodiment of this application, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps of the three-dimensional digital human generation method as described in the foregoing embodiments.

[0185] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, devices, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage and optical storage) containing computer-usable program code.

[0186] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0187] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0188] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined to obtain new product embodiments without conflict. Similarly, the features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined to obtain new method or device embodiments without conflict. The above descriptions are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for generating a three-dimensional digital human, the method comprising: Obtain the feature data corresponding to the target object; The target object includes the object body and the attachments of the object body; Based on the feature data, determine the three-dimensional Gaussian model and three-dimensional mesh model corresponding to the target object; Based on the three-dimensional Gaussian model and the three-dimensional mesh model, a three-dimensional digital human corresponding to the target object is obtained.

2. The method according to claim 1, wherein obtaining the feature data corresponding to the target object includes: Obtain a preset point cloud template corresponding to the main body of the object and two-dimensional image data corresponding to the target object; wherein, the preset point cloud template includes each coordinate point corresponding to the main body of the object; Based on the two-dimensional image data, the weight information corresponding to each coordinate point in the preset point cloud template is determined; wherein, the weight information is used to characterize the mapping relationship between each coordinate point in the preset point cloud template and the two-dimensional image data; Based on the weight information, the feature data corresponding to the target object is determined.

3. The method according to claim 2, wherein obtaining the preset point cloud template corresponding to the object body includes: Divide the main body of the object into various human body parts; Obtain at least one sub-image corresponding to each human body part; Based on the sub-images corresponding to each human body part, a subset of point clouds corresponding to each human body part is determined; Based on the point cloud subsets corresponding to each human body part, a preset point cloud template corresponding to the main body of the object is determined.

4. The method according to claim 2, wherein determining the three-dimensional mesh model corresponding to the target object based on the feature data includes: Based on the feature data, determine the coordinate points corresponding to the target object; Based on the distance information from each coordinate point corresponding to the target object to the surface of the target object, determine each grid vertex corresponding to the target object; Based on each of the grid vertices, a three-dimensional mesh model corresponding to the target object is constructed.

5. The method according to claim 4, wherein determining the three-dimensional Gaussian model corresponding to the target object based on the feature data includes: The feature data is input into a Gaussian decoder to obtain multiple Gaussian points corresponding to the target object and the attribute information associated with each Gaussian point; wherein, the attribute information is used to characterize the spatial relationship and appearance features of the multiple Gaussian points; Based on the multiple Gaussian points and the attribute information, a three-dimensional Gaussian model corresponding to the target object is obtained.

6. The method according to claim 5, wherein obtaining the 3D digital human corresponding to the target object based on the 3D Gaussian model and the 3D mesh model comprises: Based on the multiple Gaussian points corresponding to the target object and the attribute information associated with each Gaussian point, the attribute information associated with each mesh vertex in the three-dimensional mesh model is determined. Based on the three-dimensional mesh model and the attribute information associated with each mesh vertex, a three-dimensional digital human corresponding to the target object is obtained.

7. The method according to any one of claims 2-6, further comprising: Based on the preset point cloud template, determine the skin weight associated with the 3D digital human; The system receives a driving signal and generates a three-dimensional digital human that matches the driving signal based on the driving signal and the skin weight.

8. The method according to claim 1, further comprising: Obtain multi-view images of the 3D digital human corresponding to the target object; Based on the multi-view images, generate the texture map corresponding to the 3D digital human; The step of generating the texture map corresponding to the 3D digital human based on the multi-view images includes at least one of the following: Based on the perspective information of the multi-view images and the visibility of the 3D digital human in the multi-view images, a texture map corresponding to the 3D digital human is generated. The feature information corresponding to the multi-view images is obtained, and the feature information is reprojected onto the three-dimensional mesh model to obtain a multi-view feature set. The multi-view feature set is then input into an image generation network to generate a texture map corresponding to the three-dimensional digital human.

9. A three-dimensional digital human generation device, the device comprising: The acquisition module is used to acquire feature data corresponding to the target object; The target object includes the object body and the attachments of the object body; The determination module is used to determine the three-dimensional Gaussian model and three-dimensional mesh model corresponding to the target object based on the feature data; The generation module is used to obtain a three-dimensional digital human corresponding to the target object based on the three-dimensional Gaussian model and the three-dimensional mesh model.

10. An electronic device, comprising a display device, at least one processor, and a target application running on the processor, the target application being capable of independently executing or invoking at least one artificial intelligence model to perform the following operations: Obtain feature data corresponding to the target object; the target object includes the object body and the attachments of the object body; based on the feature data, determine the three-dimensional Gaussian model and the three-dimensional mesh model corresponding to the target object; based on the three-dimensional Gaussian model and the three-dimensional mesh model, obtain the three-dimensional digital human corresponding to the target object; The display device is used to display a three-dimensional digital human corresponding to the target object.

Images