Method and device for generating digital copies

Abstract

The invention discloses a method and a device for generating a digital copy. The method comprises the following steps: acquiring a half-body photo; inputting the half-body picture into an image analysis module to obtain a standard 3D model corresponding to the half-body picture; inputting the half-body picture into a pre-trained 3D reconstruction model to obtain a prediction 3D model corresponding to the half-body picture; wherein the 3D reconstruction model is composed of a 3D face reconstruction module, a 3D hair reconstruction module and 3D modeling software; the 3D face reconstruction module is an HRN model; the 3D hair reconstruction module is a NeuralHDHair model; and fusing the standard 3D model and the predicted 3D model to obtain a final 3D model corresponding to the half-body photo. Therefore, the standard 3D model is used as guidance, calculation and optimization are performed on the constructed prediction 3D model, so that the finally generated model can keep relatively high similarity with a real user, and the problem that a relatively accurate face model cannot be obtained is solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and device for generating a digital avatar. Background Art

[0002] A digital avatar refers to a digital virtual character customized based on a real person's image. It can not only meet users' requirements for high-quality customized digital images, but also replace real people in many work roles, such as virtual customer service, virtual anchor, work assistant, service guide, etc. Therefore, the generation of digital avatars is particularly important.

[0003] Among the existing technologies, it mainly relies on artificial intelligence technology, that is, using 3D face reconstruction technology to generate a 3D model of a person's head based on photos provided by the user, thereby obtaining a digital avatar.

[0004] However, the digital avatars generated using 3D face reconstruction technology often ignore the modeling of the upper body of the character, and the authenticity of the generated digital avatars is low, resulting in the inability to obtain a more accurate face model. Summary of the invention

[0005] Based on the above-mentioned deficiencies of the prior art, the present application provides a method and device for generating a digital avatar to solve the problem that the prior art cannot obtain a more accurate face model.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] The first aspect of the present application provides a method for generating a digital avatar, comprising:

[0008] Get a half-length photo;

[0009] Input the half-length photo into an image analysis module to obtain a standard 3D model corresponding to the half-length photo; wherein the image analysis module is composed of a drawing software and a 3D modeling software;

[0010] Input the half-length photo into a pre-trained 3D reconstruction model to obtain a predicted 3D model corresponding to the half-length photo; wherein the 3D reconstruction model is composed of a 3D face reconstruction module, a 3D hair reconstruction module and the 3D modeling software; the 3D face reconstruction module is an HRN model; and the 3D hair reconstruction module is a NeuralHDHair model;

[0011] The standard 3D model and the predicted 3D model are fused to obtain a final 3D model corresponding to the half-length photo.

[0012] Optionally, in the above-mentioned method for generating a digital avatar, the image analysis module includes a multi-classification deep learning model, an acquisition module, an image size measurement module and a 3D modeling software, and the step of inputting the half-length photo into the image analysis module to obtain a standard 3D model corresponding to the half-length photo includes:

[0013] Performing description processing on the half-length photo through a multi-classification deep learning model to obtain a description label corresponding to the half-length photo;

[0014] The acquisition module acquires the standard head model corresponding to the description tag and its corresponding size from the database;

[0015] Measuring the human body in the half-body photo by using the drawing software to obtain the ratio of the upper body to the head size, and calculating the product of the size and the ratio by using the drawing software to obtain the upper body model;

[0016] The standard head model and the upper body model are combined by the 3D modeling software to obtain a standard 3D model corresponding to the half-body photo.

[0017] Optionally, in the above-mentioned method for generating a digital avatar, inputting the half-length photo into a pre-trained 3D reconstruction model to obtain a predicted 3D model corresponding to the half-length photo includes:

[0018] Inputting the half-length photo into a pre-trained HRN model to obtain a 3D face model; wherein the HRN model is pre-trained using the sample half-length photo;

[0019] Inputting the half-length photo into a pre-trained NeuralHDHair model to obtain a 3D hair model; wherein the NeuralHDHair model is pre-trained using the sample half-length photo;

[0020] The 3D face model and the 3D hair model are input into the 3D modeling software for splicing to obtain a predicted 3D model corresponding to the half-length photo.

[0021] Optionally, in the above-mentioned method for generating a digital avatar, the NeuralHDHair model includes an input module, a hair generation network and a hair growth network, and the step of inputting the half-body photo into a pre-trained NeuralHDHair model to obtain a 3D hair model includes:

[0022] Calculate the 2D direction map and brightness map of the half-length photo through the input module;

[0023] Aligning the 2D direction map and the brightness map with a preset half-body model through the input module to obtain a half-body depth map;

[0024] Predicting 3D hair geometric features on the 2D direction map, the brightness map, and the half-body depth map through the hair generation network to obtain a 3D direction field and a 3D occupancy field;

[0025] A 3D hair model is generated according to the 3D direction field and the 3D occupancy field by the hair growth network.

[0026] Optionally, in the above-mentioned method for generating a digital avatar, fusing the standard 3D model with the predicted 3D model to obtain a final 3D model corresponding to the half-length photo includes:

[0027] Aligning the standard 3D model with the predicted 3D model;

[0028] Determine mesh vertices of a facial contour portion of the predicted 3D model according to the coordinates of each vertex included in the aligned predicted 3D model;

[0029] Processing mesh vertices of the facial contour portion using Laplace smoothing to obtain a facial smoothness prediction 3D model;

[0030] Delete the head mesh above the neck in the aligned standard 3D model to obtain the target 3D model;

[0031] The facial smoothness prediction 3D model and the target 3D model are input into the 3D modeling software for splicing to obtain a final 3D model corresponding to the half-length photo.

[0032] Optionally, in the above-mentioned method for generating a digital avatar, determining the mesh vertices of the facial contour part of the predicted 3D model according to the coordinates of each vertex included in the aligned predicted 3D model includes:

[0033] Find multiple target vertex coordinates with the shortest distance from each vertex in each vertex coordinate included in the aligned standard 3D model, and use each of the target vertex coordinates as a new vertex coordinate;

[0034] Calculate the barycentric coordinates of the new vertex coordinates;

[0035] Calculating the distance between each vertex coordinate included in the predicted 3D model and the center of gravity coordinate;

[0036] Multiply each of the distances by a preset weight value to obtain a target distance corresponding to each of the vertex coordinates included in the predicted 3D model;

[0037] For each vertex coordinate included in the predicted 3D model, the target distance corresponding to the vertex coordinate is added to the vertex coordinate to obtain a new vertex coordinate;

[0038] The new vertex coordinates are determined as mesh vertices of the facial contour part of the predicted 3D model.

[0039] A second aspect of the present application provides a device for generating a digital avatar, comprising:

[0040] A photo acquisition unit, used for acquiring a half-body photo;

[0041] A first input unit, used to input the half-length photo into an image analysis module to obtain a standard 3D model corresponding to the half-length photo; wherein the image analysis module is composed of a drawing software and a 3D modeling software;

[0042] The second input unit is used to input the half-length photo into a pre-trained 3D reconstruction model to obtain a predicted 3D model corresponding to the half-length photo; wherein the 3D reconstruction model is composed of a 3D face reconstruction module, a 3D hair reconstruction module and the 3D modeling software; the 3D face reconstruction module is an HRN model; and the 3D hair reconstruction module is a NeuralHDHair model;

[0043] The model fusion unit is used to fuse the standard 3D model and the predicted 3D model to obtain a final 3D model corresponding to the half-length photo.

[0044] Optionally, in the above-mentioned digital avatar generation device, the image analysis module includes a multi-classification deep learning model, an acquisition module, an image size measurement module and 3D modeling software, and the first input unit includes:

[0045] A description processing unit, used to perform description processing on the half-length photo through a multi-classification deep learning model to obtain a description label corresponding to the half-length photo;

[0046] A size acquisition unit, used to acquire a standard head model corresponding to the description tag and its corresponding size from a database through the acquisition module;

[0047] A measuring unit, used to measure the human body in the half-body photo by using the drawing software to obtain the ratio of the upper body to the head, and to calculate the product of the size and the ratio by using the drawing software to obtain an upper body model;

[0048] A combining unit is used to combine the standard head model and the upper body model through the 3D modeling software to obtain a standard 3D model corresponding to the half-body photo.

[0049] Optionally, in the above-mentioned digital avatar generation device, the second input unit includes:

[0050] A third input unit is used to input the half-length photo into a pre-trained HRN model to obtain a 3D face model; wherein the HRN model is pre-trained using the sample half-length photo;

[0051] A fourth input unit is used to input the half-length photo into a pre-trained NeuralHDHair model to obtain a 3D hair model; wherein the NeuralHDHair model is pre-trained using the sample half-length photo;

[0052] The fifth input unit is used to input the 3D face model and the 3D hair model into the 3D modeling software for splicing to obtain a predicted 3D model corresponding to the half-length photo.

[0053] Optionally, in the above-mentioned digital avatar generation device, the NeuralHDHair model includes an input module, a hair generation network and a hair growth network, and the fourth input unit includes:

[0054] A first calculation unit, used for calculating a 2D direction map and a brightness map of the half-length photo through the input module;

[0055] an alignment unit, configured to align the 2D directional map and the brightness map with a preset half-body model through the input module to obtain a half-body depth map;

[0056] A hair prediction unit, configured to predict 3D hair geometric features on the 2D direction map, the brightness map, and the half-body depth map through the hair generation network to obtain a 3D direction field and a 3D occupancy field;

[0057] A generating unit is used to generate a 3D hair model according to the 3D direction field and the 3D occupancy field through the hair growth network.

[0058] Optionally, in the above-mentioned digital avatar generation device, the model fusion unit includes:

[0059] A model alignment unit, used for aligning the standard 3D model with the predicted 3D model;

[0060] A determination unit, used to determine the mesh vertices of the facial contour part of the predicted 3D model according to the coordinates of each vertex included in the aligned predicted 3D model;

[0061] A processing unit, used for processing the mesh vertices of the facial contour part by using Laplace smoothing to obtain a facial smoothness prediction 3D model;

[0062] A deletion unit is used to delete the head mesh above the neck in the aligned standard 3D model to obtain a target 3D model;

[0063] The sixth input unit is used to input the facial smoothness prediction 3D model and the target 3D model into the 3D modeling software for splicing to obtain a final 3D model corresponding to the half-length photo.

[0064] Optionally, in the above-mentioned digital avatar generation device, the determining unit includes:

[0065] A search unit, used for searching for multiple target vertex coordinates with the shortest distance to each vertex from each vertex coordinate included in the aligned standard 3D model, and using each of the target vertex coordinates as a new vertex coordinate;

[0066] A second calculation unit, used for calculating the barycentric coordinates of the new vertex coordinates;

[0067] A third calculation unit, used for calculating the distance between each vertex coordinate included in the predicted 3D model and the barycenter coordinate;

[0068] A fourth calculation unit, configured to multiply each of the distances by a preset weight value to obtain a target distance corresponding to each of the vertex coordinates included in the predicted 3D model;

[0069] a fifth calculation unit, configured to add the target distance corresponding to each vertex coordinate included in the predicted 3D model to the vertex coordinate to obtain a new vertex coordinate;

[0070] The determination subunit is used to determine each of the new vertex coordinates as a mesh vertex of the facial contour part of the predicted 3D model.

[0071] The present application provides a method for generating a digital avatar, which obtains a half-length photo, then inputs the half-length photo into an image analysis module to obtain a standard 3D model corresponding to the half-length photo, wherein the image analysis module is composed of a drawing software and a 3D modeling software, then inputs the half-length photo into a pre-trained 3D reconstruction model to obtain a predicted 3D model corresponding to the half-length photo, wherein the 3D reconstruction model is composed of a 3D face reconstruction module, a 3D hair reconstruction module and a 3D modeling software, wherein the 3D face reconstruction module is an HRN model, and the 3D hair reconstruction module is a NeuralHDHair model, and finally the standard 3D model and the predicted 3D model are fused to obtain a final 3D model corresponding to the half-length photo. Thus, the standard 3D model is used as a guide to calculate and tune the constructed predicted 3D model, so that the finally generated model can maintain a high degree of similarity with the real user, thereby solving the problem of not being able to obtain a more accurate face model. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0073] Figure 1 A flowchart of a method for generating a digital avatar provided in an embodiment of the present application;

[0074] Figure 2 A flowchart of a method for obtaining a standard 3D model provided in an embodiment of the present application;

[0075] Figure 3 A schematic diagram of measuring head and body size provided in an embodiment of the present application;

[0076] Figure 4 A flowchart of a method for obtaining a predicted 3D model provided in an embodiment of the present application;

[0077] Figure 5 A flowchart of a method for generating a 3D hair model provided in an embodiment of the present application;

[0078] Figure 6 A flowchart of a method for obtaining a final 3D model provided in an embodiment of the present application;

[0079] Figure 7 A flowchart of a method for determining mesh vertices provided in an embodiment of the present application;

[0080] Figure 8A schematic diagram of the principle of Laplace smoothing provided in an embodiment of the present application;

[0081] Fig. 9 A schematic structural diagram of a device for generating a digital avatar provided in another embodiment of the present application. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0083] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0084] The present application embodiment provides a method for generating a digital avatar, such as Figure 1 As shown, the specific steps include:

[0085] S101, obtaining a half-length photo.

[0086] Optionally, the user can upload a half-body photo to the system through a photo upload interface provided by the system, so that the system can subsequently accurately generate a corresponding 3D model based on the half-body photo.

[0087] S102, inputting the half-length photo into an image analysis module to obtain a standard 3D model corresponding to the half-length photo.

[0088] Among them, the image analysis module is composed of a combination of drawing software and 3D modeling software.

[0089] Specifically, the half-body photo is input into the image analysis module to first describe the facial features, obtain the corresponding standard head 3D model, then calculate the ratio of the head and shoulders, obtain the upper body size model, and finally merge the standard head 3D model and the upper body size model to obtain the standard 3D model.

[0090] Optionally, in another embodiment of the present application, the image analysis module includes a multi-classification deep learning model, an acquisition module, an image size measurement module and a 3D modeling software, so a specific implementation of step S102 is as follows: Figure 2 As shown, the following steps are included:

[0091] S201. Perform description processing on the half-length photo through a multi-classification deep learning model to obtain a description label corresponding to the half-length photo.

[0092] It should be noted that the half-body picture will be input into the multi-classification deep learning model in the image analysis module for processing, and the corresponding description label will be obtained.

[0093] In addition, the multi-classification deep learning model is obtained by pre-training with a self-made face image database (which contains a large number of standard head model images of different face shapes after fine-tuning of the size) and the CAS-PEAL face database (3,000 frontal faces with natural lighting and no expression are taken, and 5 labelers are used to manually label them, and the label types are consistent with the classification of the present invention). Specifically, the multi-classification deep learning model in the embodiment of the present application is composed of a simple convolutional neural network (3 layers of convolutional layers with a convolution kernel size of 5×5, an average pooling layer and a ReLU activation layer) and a Softmax classifier. The convolutional neural network is responsible for extracting image features, and the Softmax classifier is responsible for calculating the probability of each category label based on the image features. The label with the largest probability value is the classification label of the input image. This model uses the cross entropy loss function to calculate the gap between the predicted value and the true value. The formula is as follows:

[0094]

[0095] Among them, L represents the cross entropy loss, s i represents the true classification value, o i Represents the predicted categorical value. The training loss is back-propagated using the Adam optimization algorithm.

[0096] S202: Obtain a standard head model corresponding to the description tag and its corresponding size from a database through an acquisition module.

[0097] Specifically, in the embodiment of the present application, a plurality of standard head 3D models are first preset according to the features of the facial shape, wherein each facial shape feature corresponds to a unique description label, and the description label may include square face, round face, diamond face, oval face, long face, heart-shaped face, and pear-shaped face. Therefore, according to the description features outputted in step S201, the corresponding standard head model can be obtained from the database, and each standard head model has a corresponding head size.

[0098] S203, measuring the human body in the half-body photo by using an image size measurement module to obtain a ratio of the upper body to the head size, and calculating the product of the size and the ratio by using a drawing software to obtain an upper body model.

[0099] It should be noted that in the embodiment of the present application, two standard upper body 3D models for men and women are pre-set, wherein the models all come with basic shirt clothing, and of course other customized clothing can be replaced as needed, and then the upper body 3D model can be adjusted according to the head-to-body ratio. According to specific needs, the half-body photo can be input into the image size measurement module to obtain the ratio of the upper body and head size, and then the ratio is multiplied by the size of the standard head model to obtain the upper body model size, that is, the upper body model will be obtained.

[0100] It should also be noted that the image size measurement module uses Photoshop drawing software to measure the pixel value of the human body in the photo. The ratio of the upper body to the head size is obtained by dividing the pixel value of the maximum width of the shoulder by the pixel value of the maximum width of the head. For details, please refer to Figure 3 Schematic diagram of head and body size measurements shown.

[0101] S204, combining the standard head model and the upper body model by using 3D modeling software to obtain a standard 3D model corresponding to the half-body photo.

[0102] Specifically, after obtaining the standard head model and the upper body model, the standard head model and the upper body model can be imported into the 3D modeling software, that is, imported into Blender for vertex combination, and then the combined model is trimmed, and finally the standard 3D model can be obtained.

[0103] S103, inputting the half-length photo into a pre-trained 3D reconstruction model to obtain a predicted 3D model corresponding to the half-length photo.

[0104] The 3D reconstruction model consists of a 3D face reconstruction module, a 3D hair reconstruction module and 3D modeling software. The 3D face reconstruction module is an HRN model. The 3D hair reconstruction module is a NeuralHDHair model.

[0105] Specifically, in order to realize the conversion of two-dimensional photos into three-dimensional models, the embodiment of the present application inputs the two-dimensional half-body photos into the 3D face reconstruction module and the 3D hair reconstruction module respectively to generate 3D models of the face and hair, that is, inputs them into the 3D reconstruction model, thereby saving the manual carving costs and improving the generation efficiency.

[0106] Optionally, in another embodiment of the present application, a specific implementation of step S103 is as follows: Figure 4 As shown, the following steps are included:

[0107] S401, input the half-body photo into a pre-trained HRN model to obtain a 3D face model.

[0108] Among them, the HRN model is pre-trained using sample half-body photos.

[0109] It should be noted that the Hierarchical Recurrent Network (HRN) model is a novel hierarchical representation network that can perform high-precision face reconstruction on input images. Specifically, it decouples the geometric details of the face and introduces hierarchical representation to achieve fine face modeling, and combines the 3D features of facial details to improve the accuracy of the results. It effectively makes up for the shortcomings of other face reconstruction methods that cannot restore high-frequency facial details such as wrinkles and dimples. Therefore, by inputting a half-body photo into a pre-trained HRN model, a high-precision face model can be obtained.

[0110] S402: Input the half-body photo into a pre-trained NeuralHDHair model to obtain a 3D hair model.

[0111] Among them, the NeuralHDHair model is pre-trained using sample half-body photos.

[0112] It should be noted that the NeuralHDHair model is an automatic and efficient hair modeling model, which includes two networks: IRHairNet and GrowingNet. Among them, IRHairNet usually uses architectures such as convolutional neural networks (CNN) or generative adversarial networks (GANs) to learn the features of a large number of real hair images and generate new images. GrowingNet usually uses generative adversarial networks (GANs) or other deep learning architectures to continuously generate and optimize the structure of hair to improve the realism of the generated results. Therefore, through the neural network technology in the NeuralHDHair model, a highly realistic hair model, that is, a 3D hair model, can be successfully generated.

[0113] Optionally, in another embodiment of the present application, the NeuralHDHair model includes an input module, a hair generation network and a hair growth network, and a specific implementation of step S402 is as follows: Figure 5 As shown, the following steps are included:

[0114] S501, calculating a 2D orientation map and a brightness map of the half-length photo through an input module.

[0115] It should be noted that the direction map is used to capture the directional information of the surface or texture of the object in the image to help the model understand the orientation and flow direction of the hair. In the NeuralHDHair model, edge information in the image can be extracted through edge detection (such as the Sobel operator or the Canny operator). Hair usually has complex textures and details, so accurate edge information is very important. The gradient of the image is then calculated to obtain the direction of each pixel. For the hair part, the model needs to identify the growth direction of each hair, so that a more natural hair effect can be simulated. Then for the three-dimensional scene, the model may estimate the surface normal direction based on the edge information of the image. In the hair model, this is crucial for lighting calculation and reflection rendering. Finally, the output of the direction map will be a two-dimensional array representing the directional information of each pixel, usually expressed in angle form (for example: 0 to 360 degrees).

[0116] It should also be noted that the brightness map usually refers to the brightness value of each pixel in the image, which is the grayscale version of the image. When calculating the brightness map of an image, the model usually uses the standard weighted average formula to convert the RGB color space to the grayscale space.

[0117] Therefore, the NeuralHDHair model helps achieve high-quality hair rendering by calculating the 2D direction map and brightness map of the input image (such as a half-length photo). In the model, the direction map mainly captures the direction and flow of the hair, while the brightness map helps calculate the lighting and shadow effects. Through these image features, the deep learning network is able to generate more natural and realistic hair effects.

[0118] S502: Align the 2D direction map and the brightness map with the preset half-body model through an input module to obtain a half-body depth map.

[0119] First, to get a half-body depth map, the model needs a preset half-body model, which is usually a 3D model with a standardized skeleton and structure, which may be a face or the upper part of the human body (for example, from the shoulders to the head). Secondly, the 2D direction map and brightness map are matched with the spatial data of the 3D preset half-body model. This process usually includes the following steps:

[0120] Geometric alignment: Map the 2D image to the 3D model by detecting key points in the image (such as facial feature points, hair outlines, etc.). Common methods include pose estimation and feature matching. For example, facial recognition algorithms (such as OpenPose or Dlib) are used to extract feature points in the image and align these points to the corresponding positions in the preset 3D model.

[0121] Texture alignment: Convert the direction map and brightness map into texture information and map it to the surface of the 3D model. This step ensures that the surface of the model is aligned with the elements in the image, such as hair and skin color, in terms of lighting and surface details.

[0122] Depth estimation: By estimating the depth information in the image, combined with the existing lighting, surface normal and direction map data, a depth map of the model is generated. The depth map represents the distance information of each pixel relative to the viewpoint, helping the model to correctly reconstruct objects in three-dimensional space.

[0123] Then, after the alignment is completed, the network will calculate the depth value of each pixel based on the aligned direction map and brightness map and the surface geometry information of the 3D model. This process is usually implemented with the help of a convolutional neural network (CNN), which can infer the depth of each pixel in the 3D space based on the illumination and texture information of the image, and finally output a half-body depth map based on the depth value of each pixel, where the half-body depth map is a distance map representing each pixel relative to the observer's viewpoint.

[0124] S503, predicting 3D hair geometric features of the 2D direction map, the brightness map, and the half-body depth map through a hair generation network to obtain a 3D direction field and a 3D occupancy field.

[0125] Among them, the hair generation network is IRHairNet, which is a deep learning network specifically used for hair geometry reconstruction. It can predict three-dimensional hair structure from two-dimensional image features.

[0126] Specifically, the IRHairNe network maps the input two-dimensional image (such as the direction map and brightness map) into a three-dimensional volume space, and obtains a series of useful three-dimensional data, including 3D direction field and 3D occupancy field. Each voxel in this space (i.e., a small unit in a three-dimensional grid) obtains corresponding features based on the input image information.

[0127] It should be noted that the 3D direction field represents the local hair growth direction within each voxel (a small area in 3D space). This direction is usually the main direction of hair flow and reflects how the hair is arranged in three-dimensional space.

[0128] The orientation of each voxel can be represented by a vector that indicates the direction the hair is growing in. The network predicts these directions by learning information such as the hair's growth pattern, light reflection, and surface characteristics.

[0129] The 3D occupancy field indicates whether there are hair voxels in the three-dimensional space. In simple terms, it determines whether each voxel is covered or penetrated by hair.

[0130] Each voxel of the occupancy field is a binary value, 0 means that the voxel has no hair passing through it, and 1 means that the voxel contains hair (i.e. hair passes through this location). This helps IRHairNet generate an accurate hair volume model, ensuring that the hair "fills" the correct area in three-dimensional space.

[0131] S504: Generate a 3D hair model according to the 3D direction field and the 3D occupancy field through a hair growth network.

[0132] Among them, the hair growth network refers to GrowingNet, which is a network architecture that uses deep learning technology to generate a complete hair model from a three-dimensional hair direction field (3D Direction Field) and a three-dimensional occupancy field (3D Occupancy Field).

[0133] Specifically, GrowingNet imitates the growth process of real hair, growing from the root (scalp) to the outside (hair tip). During this process, the network infers the growth trajectory of the hair based on the vector information in the 3D direction field. Hair growth is dynamic, so the network needs to take into account the characteristics of hair such as curves, bifurcations, and deformations. The 3D direction field provides the global direction of hair growth, and GrowingNet will plan a specific growth path for each hair based on this direction. This means that the generation of each hair strand depends not only on local direction information, but also on the global hair growth pattern. Secondly, the occupancy field determines the distribution of hair in three-dimensional space. For areas with an occupancy value of 1, GrowingNet will generate the corresponding hair strands. For areas with an occupancy value of 0, the network will avoid generating hair. The occupancy field helps avoid generating unnatural "blanks" or unreasonable hair structures. Then, when generating hair strands, GrowingNet must not only consider the density and distribution of the hair, but also pay attention to the details of the hair (such as the thickness, curvature, and gloss of the hair). The network learns these details through training to generate realistic hair effects. Finally, when generating the hair model, GrowingNet divides the three-dimensional space into multiple voxels (small three-dimensional units). The state of each voxel (whether there is hair, the direction of the hair, etc.) will be filled according to the information of the 3D direction field and the occupancy field, and finally a complete hair mesh model will be formed.

[0134] S403: Input the 3D face model and the 3D hair model into 3D modeling software for splicing to obtain a predicted 3D model corresponding to the half-length photo.

[0135] Specifically, after obtaining the 3D face model and the 3D hair model, the 3D face model and the 3D hair model will be input into Blender for splicing and fusion, and finally a predicted head 3D model, that is, a predicted 3D model, can be obtained.

[0136] S104: Fusing the standard 3D model and the predicted 3D model to obtain a final 3D model corresponding to the half-length photo.

[0137] It should be emphasized that although the predicted 3D model accurately restores the image of the real person in the photo, the realistic image often has facial contours that are not smooth and regular enough, and cannot meet people's pursuit of a beautiful virtual image. Therefore, the embodiment of the present application adaptively integrates the predicted 3D model and the standard 3D model to ultimately generate a beautified digital avatar model, which is the final 3D model corresponding to the half-body photo.

[0138] Optionally, in another embodiment of the present application, a specific implementation of step S104 is as follows: Figure 6 As shown, the following steps are included:

[0139] S601: Align the standard 3D model and the predicted 3D model.

[0140] Specifically, in order to help optimize the details of the model and make the generated digital avatar model more realistic, the predicted 3D model and the standard 3D model are aligned in size and position through rotation and scaling operations to ensure that the positions of the four boundary vertices of the upper, lower, left and right sides of the heads of the two models are consistent.

[0141] S602: Determine mesh vertices of a facial contour portion of the predicted 3D model according to the coordinates of each vertex included in the aligned predicted 3D model.

[0142] It is understandable that after the standard 3D model and the predicted 3D model are aligned, the mesh vertices of the facial contours of the two models need to be calculated to achieve the effect of optimizing the predicted 3D model. Specifically, the mesh of the standard 3D model is regarded as a guide for the deformation target, and the predicted 3D model is regarded as the object to be deformed. Since there is no one-to-one correspondence between the mesh vertices of the two models, the mesh vertices of the facial contour of the predicted 3D model can be determined based on the coordinates of each vertex contained in the predicted 3D model.

[0143] Optionally, in another embodiment of the present application, a specific implementation of step S602 is as follows: Figure 7 As shown, the following steps are included:

[0144] S701 , searching for multiple target vertex coordinates with the shortest distance to each vertex from each vertex coordinate included in the aligned standard 3D model, and using each target vertex coordinate as a new vertex coordinate.

[0145] It can be understood that in order to obtain the mesh vertices of the facial contour part of the predicted 3D model, it is first necessary to find the three vertices P1 (x1, y1, z1), P2 (x2, y2, z2), and P3 (x3, y3, z3) closest to each vertex of the predicted 3D model, which can be determined by calculating the Euclidean distance metric between the coordinates of each vertex.

[0146] S702, calculating the centroid coordinates of the new vertex coordinates.

[0147] Specifically, after obtaining the three nearest vertices P1 (x1, y1, z1), P2 (x2, y2, z2), and P3 (x3, y3, z3), the barycentric coordinates can be calculated based on the coordinates of these three new vertices, where the calculation expression of the barycentric coordinates is:

[0148] x=(x1+x2+x3) / 3,y=(y1+y2+ y3 ) / 3, z=(z1+z2+z3) / 3

[0149] S703: Calculate the distance between each vertex coordinate and the center of gravity coordinate included in the predicted 3D model.

[0150] It is understandable that in order to analyze and predict the geometric characteristics, mass distribution, stability, symmetry, optimization space and physical behavior of the 3D model from multiple angles, the analysis can be performed by calculating the distance, that is, calculating and predicting the coordinates P of each vertex contained in the 3D model. i (x i ,y i , z i ) and the center of gravity coordinate G i (x gi ,y gi , z gi ) between the two.

[0151] S704: Multiply each distance by a preset weight value to obtain a target distance corresponding to each vertex coordinate included in the predicted 3D model.

[0152] Specifically, since the calculated distance values ​​have positive and negative values, each distance value needs to be multiplied by a weight value ωi (generally 0.5).

[0153] S705 , for each vertex coordinate included in the predicted 3D model, add the target distance corresponding to the vertex coordinate to the vertex coordinate to obtain a new vertex coordinate.

[0154] Specifically, the calculation formula for the new vertex coordinates (that is, the fusion coordinates) is:

[0155] x' i =x i+(x i -x gi )×ω i

[0156] y' i =y i +(y i -y gi )×ω i

[0157] z' i =z i +(z i -z gi )×ω i

[0158] S706: Determine each new vertex coordinate as a mesh vertex of the facial contour part of the predicted 3D model.

[0159] S603: Process the mesh vertices of the facial contour part by using Laplace smoothing to obtain a facial smoothness prediction 3D model.

[0160] It should be noted that, in order to keep the mesh smooth, the embodiment of the present application uses the Laplace smoothing method to process the mesh of the facial contour part, that is, each vertex is moved to the average position of the adjacent vertices, and a smooth predicted 3D model of the facial contour is obtained. For details, please refer to Figure 8 Schematic diagram of the principle of Laplace smoothing shown.

[0161] S604: Delete the head mesh above the neck in the aligned standard 3D model to obtain a target 3D model.

[0162] Specifically, after obtaining the facial smoothness prediction 3D model, it is also necessary to discard the head mesh above the neck of the standard 3D model, which can not only improve efficiency but also make the model more focused on the dynamic features of the face.

[0163] S605 , input the facial smoothness prediction 3D model and the target 3D model into 3D modeling software for splicing, and obtain a final 3D model corresponding to the half-length photo.

[0164] Specifically, the facial smoothness prediction 3D model and the target 3D model are input into Blender to perform splicing and fusion of the two models, and a final 3D model corresponding to the half-body photo is obtained.

[0165] The present application provides a method for generating a digital avatar, which obtains a half-length photo, then inputs the half-length photo into an image analysis module to obtain a standard 3D model corresponding to the half-length photo, wherein the image analysis module is composed of a combination of drawing software and 3D modeling software, and then inputs the half-length photo into a pre-trained 3D reconstruction model to obtain a predicted 3D model corresponding to the half-length photo, wherein the 3D reconstruction model is composed of a 3D face reconstruction module, a 3D hair reconstruction module and 3D modeling software. The 3D face reconstruction module is an HRN model, and the 3D hair reconstruction module is a NeuralHDHair model. Finally, the standard 3D model and the predicted 3D model are fused to obtain the final 3D model corresponding to the half-length photo. Thus, the standard 3D model is used as a guide to calculate and tune the constructed predicted 3D model, so that the finally generated model can maintain a high degree of similarity with the real user, thereby solving the problem of not being able to obtain a more accurate face model.

[0166] Another embodiment of the present application provides a device for generating a digital avatar, such as Fig. 9 As shown, it includes the following units:

[0167] The photo acquisition unit 901 is used to acquire a half-body photo.

[0168] The first input unit 902 is used to input the half-length photo into the image analysis module to obtain a standard 3D model corresponding to the half-length photo. The image analysis module is composed of a drawing software and a 3D modeling software.

[0169] The second input unit 903 is used to input the half-length photo into the pre-trained 3D reconstruction model to obtain a predicted 3D model corresponding to the half-length photo. The 3D reconstruction model is composed of a 3D face reconstruction module, a 3D hair reconstruction module and a 3D modeling software. The 3D face reconstruction module is an HRN model. The 3D hair reconstruction module is a NeuralHDHair model.

[0170] The model fusion unit 904 is used to fuse the standard 3D model and the predicted 3D model to obtain a final 3D model corresponding to the half-length photo.

[0171] It should be noted that the specific working process of the above-mentioned units in the embodiment of the present application can refer to steps S101 to S104 in the above-mentioned method embodiment, and will not be repeated here.

[0172] Optionally, in a digital avatar generation device provided in another embodiment of the present application, the image analysis module includes a multi-classification deep learning model, an acquisition module, an image size measurement module and 3D modeling software, and the first input unit 902 includes:

[0173] The description processing unit is used to perform description processing on the half-length photo through a multi-classification deep learning model to obtain a description label corresponding to the half-length photo.

[0174] The size acquisition unit is used to acquire the standard head model corresponding to the description tag and its corresponding size from the database through the acquisition module.

[0175] The measuring unit is used to measure the human body in the half-body photo through drawing software to obtain the ratio of the upper body and head size, and calculate the product of the size and the ratio through the drawing software to obtain the upper body model.

[0176] The combining unit is used to combine the standard head model and the upper body model through 3D modeling software to obtain a standard 3D model corresponding to the half-body photo.

[0177] Optionally, in a digital avatar generation device provided in another embodiment of the present application, the second input unit 903 includes:

[0178] The third input unit is used to input the half-length photo into a pre-trained HRN model to obtain a 3D face model, wherein the HRN model is pre-trained using the sample half-length photo.

[0179] The fourth input unit is used to input the half-length photo into a pre-trained NeuralHDHair model to obtain a 3D hair model. The NeuralHDHair model is pre-trained using a sample half-length photo.

[0180] The fifth input unit is used to input the 3D face model and the 3D hair model into the 3D modeling software for splicing to obtain a predicted 3D model corresponding to the half-length photo.

[0181] Optionally, in a device for generating a digital avatar provided in another embodiment of the present application, the NeuralHDHair model includes an input module, a hair generation network, and a hair growth network, and the fourth input unit includes:

[0182] The first calculation unit is used to calculate the 2D direction map and brightness map of the half-length photo through the input module.

[0183] The alignment unit is used to align the 2D direction map and the brightness map with the preset half-body model through the input module to obtain a half-body depth map.

[0184] The hair prediction unit is used to predict the 3D hair geometric features of the 2D direction map, brightness map and half-body depth map through the hair generation network to obtain the 3D direction field and 3D occupancy field.

[0185] The generation unit is used to generate a 3D hair model according to the 3D direction field and the 3D occupancy field through a hair growth network.

[0186] Optionally, in a digital avatar generation device provided in another embodiment of the present application, the model fusion unit 904 includes:

[0187] The model alignment unit is used to align the standard 3D model with the predicted 3D model.

[0188] The determination unit is used to determine the mesh vertices of the facial contour part of the predicted 3D model according to the coordinates of each vertex included in the aligned predicted 3D model.

[0189] The processing unit is used to process the mesh vertices of the facial contour part by using Laplace smoothing to obtain a facial smoothness prediction 3D model.

[0190] The deletion unit is used to delete the head mesh above the neck in the aligned standard 3D model to obtain the target 3D model.

[0191] The sixth input unit is used to input the facial smoothness prediction 3D model and the target 3D model into the 3D modeling software for splicing to obtain a final 3D model corresponding to the half-length photo.

[0192] Optionally, in a digital avatar generation device provided in another embodiment of the present application, the determination unit includes:

[0193] The search unit is used to search for multiple target vertex coordinates with the shortest distance to each vertex from each vertex coordinate contained in the aligned standard 3D model, and use each target vertex coordinate as a new vertex coordinate.

[0194] The second calculation unit is used to calculate the centroid coordinates of the new vertex coordinates.

[0195] The third calculation unit is used to calculate the distance between each vertex coordinate and the center of gravity coordinate included in the predicted 3D model.

[0196] The fourth calculation unit is used to multiply each distance by a preset weight value to obtain a target distance corresponding to each vertex coordinate included in the predicted 3D model.

[0197] The fifth calculation unit is used to add the target distance corresponding to each vertex coordinate included in the predicted 3D model to the vertex coordinate to obtain a new vertex coordinate.

[0198] The determination subunit is used to determine each new vertex coordinate as a mesh vertex of a facial contour part of the predicted 3D model.

[0199] It should be noted that the specific working process of each unit provided in the above embodiments of the present application can refer to the corresponding steps in the above method embodiments, and will not be repeated here.

[0200] It should also be noted that the digital avatar generation device provided in the embodiment of the present application has the technical effects of any of the above embodiments, and the embodiment of the present application will not be described in detail here.

[0201] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0202] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a digital avatar, characterized in that: include: Get a half-length photo; Input the half-length photo into an image analysis module to obtain a standard 3D model corresponding to the half-length photo; wherein the image analysis module is composed of a drawing software and a 3D modeling software; Input the half-length photo into a pre-trained 3D reconstruction model to obtain a predicted 3D model corresponding to the half-length photo; wherein the 3D reconstruction model is composed of a 3D face reconstruction module, a 3D hair reconstruction module and the 3D modeling software; the 3D face reconstruction module is an HRN model; and the 3D hair reconstruction module is a NeuralHDHair model; The standard 3D model and the predicted 3D model are fused to obtain a final 3D model corresponding to the half-length photo.

2. The method according to claim 1, characterized in that The image analysis module includes a multi-classification deep learning model, an acquisition module, an image size measurement module and a 3D modeling software. The half-length photo is input into the image analysis module to obtain a standard 3D model corresponding to the half-length photo, including: Performing description processing on the half-length photo through a multi-classification deep learning model to obtain a description label corresponding to the half-length photo; The acquisition module acquires the standard head model corresponding to the description tag and its corresponding size from the database; Measuring the human body in the half-body photo by using the drawing software to obtain the ratio of the upper body to the head size, and calculating the product of the size and the ratio by using the drawing software to obtain the upper body model; The standard head model and the upper body model are combined by the 3D modeling software to obtain a standard 3D model corresponding to the half-body photo.

3. The method according to claim 1, characterized in that The step of inputting the half-length photo into a pre-trained 3D reconstruction model to obtain a predicted 3D model corresponding to the half-length photo comprises: Inputting the half-length photo into a pre-trained HRN model to obtain a 3D face model; wherein the HRN model is pre-trained using the sample half-length photo; Inputting the half-length photo into a pre-trained NeuralHDHair model to obtain a 3D hair model; wherein the NeuralHDHair model is pre-trained using the sample half-length photo; The 3D face model and the 3D hair model are input into the 3D modeling software for splicing to obtain a predicted 3D model corresponding to the half-length photo.

4. The method according to claim 3, characterized in that: The NeuralHDHair model includes an input module, a hair generation network and a hair growth network. The half-length photo is input into the pre-trained NeuralHDHair model to obtain a 3D hair model, including: Calculate the 2D direction map and brightness map of the half-length photo through the input module; Aligning the 2D direction map and the brightness map with a preset half-body model through the input module to obtain a half-body depth map; Predicting 3D hair geometric features of the 2D direction map, the brightness map, and the half-body depth map through the hair generation network to obtain a 3D direction field and a 3D occupancy field; A 3D hair model is generated according to the 3D direction field and the 3D occupancy field by the hair growth network.

5. The method according to claim 1, characterized in that The step of fusing the standard 3D model with the predicted 3D model to obtain a final 3D model corresponding to the half-length photo includes: Aligning the standard 3D model with the predicted 3D model; Determine mesh vertices of a facial contour portion of the predicted 3D model according to the coordinates of each vertex included in the aligned predicted 3D model; Processing mesh vertices of the facial contour portion using Laplace smoothing to obtain a facial smoothness prediction 3D model; Delete the head mesh above the neck in the aligned standard 3D model to obtain the target 3D model; The facial smoothness prediction 3D model and the target 3D model are input into the 3D modeling software for splicing to obtain a final 3D model corresponding to the half-length photo.

6. The method according to claim 5, characterized in that The step of determining mesh vertices of a facial contour portion of the predicted 3D model according to the coordinates of each vertex included in the aligned predicted 3D model comprises: Find multiple target vertex coordinates with the shortest distance from each vertex in each vertex coordinate included in the aligned standard 3D model, and use each of the target vertex coordinates as a new vertex coordinate; Calculate the barycentric coordinates of the new vertex coordinates; Calculating the distance between each vertex coordinate included in the predicted 3D model and the center of gravity coordinate; Multiply each of the distances by a preset weight value to obtain a target distance corresponding to each of the vertex coordinates included in the predicted 3D model; For each vertex coordinate included in the predicted 3D model, the target distance corresponding to the vertex coordinate is added to the vertex coordinate to obtain a new vertex coordinate; The new vertex coordinates are determined as mesh vertices of the facial contour part of the predicted 3D model.

7. A device for generating a digital avatar, characterized in that: include: A photo acquisition unit, used for acquiring a half-body photo; A first input unit, used to input the half-length photo into an image analysis module to obtain a standard 3D model corresponding to the half-length photo; wherein the image analysis module is composed of a drawing software and a 3D modeling software; The second input unit is used to input the half-length photo into a pre-trained 3D reconstruction model to obtain a predicted 3D model corresponding to the half-length photo; wherein the 3D reconstruction model is composed of a 3D face reconstruction module, a 3D hair reconstruction module and the 3D modeling software; the 3D face reconstruction module is an HRN model; and the 3D hair reconstruction module is a NeuralHDHair model; The model fusion unit is used to fuse the standard 3D model and the predicted 3D model to obtain a final 3D model corresponding to the half-length photo.

8. The device according to claim 7, characterized in that The image analysis module includes a multi-classification deep learning model, an acquisition module, an image size measurement module and 3D modeling software, and the first input unit includes: A description processing unit, used to perform description processing on the half-length photo through a multi-classification deep learning model to obtain a description label corresponding to the half-length photo; A size acquisition unit, used to acquire a standard head model corresponding to the description tag and its corresponding size from a database through the acquisition module; A measuring unit, used to measure the human body in the half-body photo by using the drawing software to obtain the ratio of the upper body to the head, and to calculate the product of the size and the ratio by using the drawing software to obtain an upper body model; A combining unit is used to combine the standard head model and the upper body model through the 3D modeling software to obtain a standard 3D model corresponding to the half-body photo.

9. The device according to claim 7, characterized in that The second input unit comprises: A third input unit is used to input the half-length photo into a pre-trained HRN model to obtain a 3D face model; wherein the HRN model is pre-trained using the sample half-length photo; A fourth input unit is used to input the half-length photo into a pre-trained NeuralHDHair model to obtain a 3D hair model; wherein the NeuralHDHair model is pre-trained using the sample half-length photo; The fifth input unit is used to input the 3D face model and the 3D hair model into the 3D modeling software for splicing to obtain a predicted 3D model corresponding to the half-length photo.

10. The device according to claim 9, characterized in that The NeuralHDHair model includes an input module, a hair generation network and a hair growth network, and the fourth input unit includes: A first calculation unit, used for calculating a 2D direction map and a brightness map of the half-length photo through the input module; an alignment unit, configured to align the 2D directional map and the brightness map with a preset half-body model through the input module to obtain a half-body depth map; A hair prediction unit, configured to predict 3D hair geometric features on the 2D direction map, the brightness map, and the half-body depth map through the hair generation network to obtain a 3D direction field and a 3D occupancy field; A generating unit is used to generate a 3D hair model according to the 3D direction field and the 3D occupancy field through the hair growth network.

Images