3D human-shaped hand generation method and device, storage medium and computer equipment

Abstract

According to the 3D human-shaped hand generation method and device, the storage medium and the computer equipment provided by the invention, after the original face image uploaded by the user and the selected hand template model are acquired, the 3D face model can be obtained after face reconstruction and repair are performed on the original face image, and then the original face image is aligned and mapped to the grid, so that the 3D human-shaped hand generation efficiency is improved. The method comprises the following steps of: obtaining a 3D face map, and aligning face areas of a 3D face model and a hand template model based on positioning point coordinates of the 3D face model and the hand template model, so that the workload of manual point marking is greatly reduced, the working efficiency is improved, the alignment quality can be ensured, and finally, the accuracy of point marking is improved. According to the method, the aligned 3D face model can be used for fitting the face area of the handmade template model, the 3D face map is mapped to the fitted handmade template model, and finally, the high-quality 3D human-shaped handmade which is highly similar to the picture uploaded by the user and fused with the specified template can be quickly obtained.

Description

Technical Field

[0001] The present application relates to the technical field of 3D figure production, and in particular to a method, device, storage medium and computer equipment for generating 3D humanoid figures. Background Art

[0002] Currently, 3D human-shaped figures, as a highly creative and personalized work of art, are extremely popular in the animation, game, film and television derivative and collectibles markets, and are widely used in many fields such as interior decoration and gift giving.

[0003] However, the production process of traditional 3D figures often relies on cumbersome and inefficient methods, such as full-body scanning technology. Although this technology can accurately capture the details of the prototype, it requires the person to be on-site and enter the scanning chamber for scanning. After reconstruction using the multi-view reconstruction algorithm, manpower is still needed to repair the model defects, and finally a printable 3D humanoid figure is obtained. The reconstruction process takes a long time, and the scanning chamber and manpower mold repair costs are high. Another way is to have experienced craftsmen manually mold the clay prototype, and then go through multiple processes such as remolding and coloring. This process is not only time-consuming and labor-intensive, but also greatly limits the production efficiency and output of the figures, resulting in the high price of the final product, which is difficult to be widely accepted by consumers.

[0004] Therefore, there is an urgent need in the market for a new technology that can produce high-quality 3D humanoid figures efficiently and at low cost, so as to increase production and lower price thresholds while maintaining artistic quality, thereby broadening market acceptance and meeting the growing personalized consumer demand. Summary of the invention

[0005] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the technical defects in the prior art that 3D humanoid figures have a long production cycle, high production cost, and cannot meet personalized consumer needs.

[0006] The present application also provides a method for generating a 3D humanoid figure, the method comprising:

[0007] Obtain the original face image uploaded by the user and the selected figure template model;

[0008] Reconstructing and repairing the original face image to obtain a 3D face model, and aligning the original face image and mapping it onto a grid to obtain a 3D face map;

[0009] Based on the coordinates of the positioning points of the 3D face model and the figure template model, aligning the face area of ​​the 3D face model with the face area of ​​the figure template model;

[0010] After fitting the face area of ​​the figure template model using the aligned 3D face model, the 3D face map is mapped onto the fitted figure template model to obtain a 3D humanoid figure.

[0011] Optionally, the reconstructing and repairing the original face image to obtain a 3D face model comprises:

[0012] Inputting the original face image into a preset face reconstruction model to obtain a three-dimensional face mesh output by the face reconstruction model;

[0013] Using a hole filling technique to fill the oral cavity of the three-dimensional face mesh, to obtain a filled three-dimensional face mesh;

[0014] The oral cavity mesh patching technology is used to patch the oral cavity part of the completed three-dimensional face mesh to obtain a final 3D face model.

[0015] Optionally, the face reconstruction model includes a convolutional layer, a first fully connected layer and a second fully connected layer;

[0016] The step of inputting the original face image into a preset face reconstruction model to obtain a three-dimensional face mesh output by the face reconstruction model includes:

[0017] Inputting the original face image into the convolution layer, and extracting feature information in the original face image through the convolution layer;

[0018] Mapping the feature information extracted by the convolutional layer to the parameter space of the 3DMM through the first fully connected layer and outputting shape parameters;

[0019] The shape parameters output by the first fully connected layer are restored to a three-dimensional face mesh for face reconstruction through the second fully connected layer.

[0020] Optionally, the face reconstruction model includes at least a second fully connected layer, and the second fully connected layer is used to perform nonlinear offset prediction on mesh vertices;

[0021] The method of using the human face oral mesh repair technology to repair the oral cavity part of the repaired three-dimensional human face mesh to obtain the final 3D human face model includes:

[0022] Inputting the original face image into a preset normal prediction model to obtain a normal image output by the normal prediction model and corresponding to the original face image;

[0023] Inputting the vertex coordinates of the oral cavity in the padded three-dimensional face mesh into the second fully connected layer to obtain the vertex offset predicted by the second fully connected layer;

[0024] After updating the coordinates of each vertex according to the vertex offset, optimizing the updated coordinates of each vertex using the normal image until a preset optimization target is reached, thereby obtaining optimized coordinates of each vertex;

[0025] The final 3D face model is generated according to the optimized coordinates of each vertex.

[0026] Optionally, the normal prediction model includes a variational autoencoder and a U-Net network;

[0027] The step of inputting the original face image into a preset normal prediction model to obtain a normal image output by the normal prediction model and corresponding to the original face image comprises:

[0028] Compressing the original face image into a latent space through an encoder of the variational autoencoder, and outputting latent features of the original face image;

[0029] Generate latent features of a normal image corresponding to latent features of the original face image through the U-Net network;

[0030] The decoder of the variational autoencoder reconstructs the potential features of the normal image back into the pixel space to obtain a corresponding normal image.

[0031] Optionally, inputting the vertex coordinates of the oral cavity in the padded three-dimensional face mesh into the second fully connected layer to obtain the vertex offset predicted by the second fully connected layer includes:

[0032] After the number of vertices of the oral cavity in the padded three-dimensional face mesh is expanded by the second fully connected layer, the coordinates of the vertices of the oral cavity after the expansion are obtained, and nonlinear offset prediction is performed on the obtained vertex coordinates to obtain predicted vertex offsets.

[0033] Optionally, the optimizing the updated coordinates of each vertex by using the normal image until a preset optimization target is reached, to obtain the optimized coordinates of each vertex, comprises:

[0034] Projecting the updated coordinates of each vertex onto a 2D normal map, and calculating the MSE pixel loss between the 2D normal map and the normal image at the oral position;

[0035] Determining whether the MSE pixel loss converges;

[0036] If yes, the updated vertex coordinates are used as optimized vertex coordinates;

[0037] Otherwise, after replacing the vertex coordinates of the oral part in the padded three-dimensional face mesh with the updated vertex coordinates, return to execute the step of inputting the vertex coordinates of the oral part in the padded three-dimensional face mesh into the second fully connected layer, obtaining the vertex offset predicted by the second fully connected layer and its subsequent steps until the MSE pixel loss converges.

[0038] Optionally, aligning the 3D face model with the face area of ​​the figure template model based on the positioning point coordinates of the 3D face model and the figure template model includes:

[0039] Obtaining the coordinates of the nose tip point, the back of the head point and the top of the head of the figure template model, as well as the 3D coordinates of multiple positioning points in the 3D face model;

[0040] Rendering the figure template model into a 2D face image according to the nose point coordinates, the back of the head point coordinates and the top of the head coordinates;

[0041] Performing face detection on the 2D face image, and obtaining 2D coordinates corresponding to each positioning point in the 3D face model according to the detection result, and converting each 2D coordinate into a 3D coordinate;

[0042] The 3D coordinates of the 3D face model and the 3D coordinates of the hand-figure template model are rigidly transformed to align the 3D face model to the face area of ​​the hand-figure template model.

[0043] Optionally, rendering the figure template model into a 2D face image according to the nose point coordinates, the back of the head point coordinates and the top of the head coordinates includes:

[0044] Determine, according to the nose tip point coordinates, the back of the head point coordinates and the top of the head coordinates, a first vector pointing from the back of the head of the figure template model to the nose tip, and a second vector pointing from the center point of the first vector to the top of the head of the figure template model;

[0045] After constructing a camera coordinate system according to the first vector and the second vector and setting corresponding camera intrinsic parameters, the figure template model is rendered as a 2D face image through the camera coordinate system and the camera intrinsic parameters.

[0046] Optionally, the step of fitting the face area of ​​the figure template model using the aligned 3D face model to obtain the fitted figure template model comprises:

[0047] Multiplying each vertex of the face area in the figure template model by the matrix parameters of the preset deformation matrix to obtain a fitted figure template model that fits the aligned 3D face model and maintains the structure of the figure template model;

[0048] The matrix parameters of the preset deformation matrix are obtained by optimizing each vertex of the face area in the figure template model and each vertex of the aligned 3D face model.

[0049] Optionally, the process of optimizing the matrix parameters of the deformation matrix includes:

[0050] Obtaining a first vertex set formed by each vertex of the face area in the sample figure template model and a second vertex set formed by each vertex of the aligned sample 3D face model;

[0051] Set the matrix parameters of the deformation matrix;

[0052] A KNN nearest neighbor algorithm is used to find the vertex in the second vertex set that is closest to each vertex in the first vertex set, and the matrix parameters are optimized according to the loss function between each vertex pair to obtain optimized matrix parameters.

[0053] Optionally, mapping the 3D face map onto a fitted figure template model to obtain a 3D humanoid figure includes:

[0054] Expand the fitted figure template model and the face area of ​​the aligned 3D face model into UV space, and scale the expanded figure template model and the 3D face model to the same resolution;

[0055] According to the position mapping relationship between the scaled figure template model and the 3D face model, mapping the texture colors of the 3D face texture to the face area of ​​the scaled figure template model one by one;

[0056] The color of the junction between the face and body of the mapped figure template model is transitioned through Poisson fusion to obtain the final 3D human figure.

[0057] The present application also provides a 3D humanoid hand-made generating device, comprising:

[0058] The data acquisition module is used to obtain the original face image uploaded by the user and the selected figure template model;

[0059] A face reconstruction module, used to reconstruct and repair the original face image to obtain a 3D face model, and to align the original face image and map it onto a grid to obtain a 3D face map;

[0060] A face alignment module, used to align the face area of ​​the 3D face model with the face area of ​​the hand-figure template model based on the positioning point coordinates of the 3D face model and the hand-figure template model;

[0061] The retopology module is used to fit the face area of ​​the figure template model with the aligned 3D face model, and then map the 3D face map onto the fitted figure template model to obtain a 3D human figure.

[0062] The present application also provides a computer-readable storage medium, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the 3D humanoid figure generation method as described in any of the above embodiments.

[0063] The present application also provides a computer device, comprising: one or more processors, and a memory;

[0064] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the 3D human figure generation method described in any one of the above embodiments are performed.

[0065] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0066] The 3D humanoid hand-made model generation method, device, storage medium and computer equipment provided by the present application can obtain the original face image uploaded by the user and the selected hand-made model template model, and then reconstruct and repair the original face image to obtain the 3D face model. In this way, the two-dimensional image can be converted into a three-dimensional face model, and the face can be repaired during the reconstruction process to solve the problem that the hollow mouth cannot be printed, so that the final 3D face model presents a natural state, so as to better integrate with the hand-made model template model; then, the present application can align the original face image and map it to the grid to obtain a 3D face model. D face map, and then based on the positioning point coordinates of the 3D face model and the figure template model, align the 3D face model with the face area of ​​the figure template model, thereby greatly saving the workload of manual annotation points, improving work efficiency while ensuring the high quality of alignment. Finally, the application can use the aligned 3D face model to fit the face area of ​​the figure template model, and map the 3D face map to the fitted figure template model. Finally, a high-quality 3D human-shaped figure with a high similarity to the user-uploaded picture and integrated with the specified template can be quickly obtained, thereby meeting personalized consumer needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0068] Figure 1 A schematic diagram of a process for generating a 3D humanoid figure provided in an embodiment of the present application;

[0069] Figure 2 A schematic diagram of a process for repairing the oral cavity in a three-dimensional face mesh provided in an embodiment of the present application;

[0070] Figure 3 A schematic diagram of a process for optimizing updated vertex coordinates using a normal image provided in an embodiment of the present application;

[0071] Figure 4 A schematic diagram of a model for converting a figure template model into a 3D humanoid figure provided in an embodiment of the present application;

[0072] Figure 5 A schematic diagram of the structure of a 3D humanoid figure generating device provided in an embodiment of the present application;

[0073] Figure 6 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0075] In one embodiment, Figure 1 As shown, Figure 1 A schematic diagram of a process of generating a 3D humanoid figure provided in an embodiment of the present application; the present application also provides a method for generating a 3D humanoid figure, the method may include:

[0076] S110: Acquire the original face image uploaded by the user and the selected figure template model.

[0077] In this step, when making a 3D human-shaped figure, you can first obtain the original face image uploaded by the user and the figure template model selected by the user, so that you can merge the original face image into the figure template model to obtain a high-quality 3D human-shaped figure that meets the user's needs.

[0078] Among them, the present application can deploy a single-view humanoid 3D figure generation system on a cloud server. The system can receive the original face image uploaded by the user and the figure template model selected from multiple preset template models, and then perform face reconstruction, repair, alignment and other operations based on the received original face image and the figure template model, and finally obtain a high-quality 3D humanoid figure with high similarity to the user-uploaded picture and fused with the specified template.

[0079] It can be understood that the original facial image here refers to the two-dimensional picture uploaded by the user to be fused with the selected figure template model. The picture can be a picture of the user himself or other pictures with complete facial features. The specific setting can be based on actual conditions and is not limited here.

[0080] S120: Reconstructing and repairing the original face image to obtain a 3D face model, and aligning the original face image and mapping it onto a grid to obtain a 3D face map.

[0081] In this step, after obtaining the original face image uploaded by the user and the selected figure template model through S110, the original face image can be reconstructed and repaired to obtain a 3D face model. Moreover, since there is a certain difference in skin color between the figure template model and the original face image, the present application can align the original face image and map it to the grid. The obtained 3D face map can be used for face map mapping to obtain a 3D humanoid figure with uniform skin color on the face and body.

[0082] Specifically, when reconstructing the original face image, the present application takes into account that the user generally has facial expressions when transmitting photos, especially with an open mouth. After the traditional 3DMM basic model reconstructs the face from the photo, the oral part is a non-closed grid, which will affect the internal mapping of the oral cavity and the final 3D printing. Therefore, the present application can design a facial oral mesh repair technology based on face reconstruction, and use this technology to repair the reconstructed three-dimensional face mesh to obtain a 3D face model.

[0083] When obtaining a 3D face map, the present application can first align the original face image in UV space and then map it to the grid, thereby obtaining a 3D face map. Of course, the present application can also align the original face image and map it to the grid in other ways, which can be set according to the actual situation and is not limited here.

[0084] S130: Aligning the facial region of the 3D facial model with that of the figurine template model based on the coordinates of the positioning points of the 3D facial model and the figurine template model.

[0085] In this step, after the original face image is reconstructed and repaired to obtain a 3D face model through S120, the present application can align the facial area of ​​the 3D face model with the figure template model based on the positioning point coordinates of the 3D face model and the figure template model, so as to better perform model fusion.

[0086] It is understandable that since the reconstructed 3D face model only contains the face information model, it needs to be aligned with the face part of the figure template model selected by the user in 3D space before subsequent model fusion can be performed. Based on this, the present application can first determine the positioning point coordinates of the 3D face model and the figure template model, and then align the two according to the positioning point coordinates of the two.

[0087] Among them, when selecting the coordinates of the positioning points of the 3D face model and the hand-made template model, the present application can select the coordinates of the same positioning points on the faces of the two, such as the nose, eyes, corners of the mouth, etc., or select different positioning point coordinates, and then convert the positioning point coordinates to obtain the positioning point coordinates with the same meaning, and then perform alignment operations according to the positioning point coordinates. The specific selection can be made according to the actual situation, and there is no restriction here. The present application uses 3D face model key point positioning and alignment technology, which can greatly save the workload of manual annotation points, improve work efficiency, and ensure high quality of alignment.

[0088] S140: After fitting the face area of ​​the figure template model using the aligned 3D face model, the 3D face map is mapped onto the fitted figure template model to obtain a 3D human figure.

[0089] In this step, after aligning the 3D face model with the facial area of ​​the figure template model through S130, the present application can use the aligned 3D face model to fit the facial area of ​​the figure template model, so as to obtain a face reconstruction mesh that fits the 3D face model and maintains the structure of the figure template model. Then, the present application can map the 3D face map obtained in step S120 to the fitted figure template model, thereby obtaining the final 3D humanoid figure.

[0090] It can be understood that after the above steps, the present application still obtains two models: one is the face model reconstructed by 3DMM, and the other is the template model, whose positions are aligned. Therefore, in the next process, the present application needs to fit (adsorb) the face area grid of the template model to the 3DMM face model, and map the color map of the face model to the template model, so as to finally complete the generation of the 3D human figure.

[0091] In the above embodiment, after obtaining the original face image uploaded by the user and the selected hand-figure template model, the original face image can be reconstructed and repaired to obtain a 3D face model. In this way, the two-dimensional image can be converted into a three-dimensional face model, and the face can be repaired during the reconstruction process to solve the problem that the hollow mouth cannot be printed, so that the final 3D face model presents a natural state, so as to better integrate with the hand-figure template model; then, the present application can align the original face image and map it to the grid to obtain a 3D face map, and then based on the 3D The coordinates of the positioning points of the face model and the figure template model are used to align the 3D face model with the face area of ​​the figure template model, thereby greatly saving the workload of manual annotation points, improving work efficiency while ensuring the high quality of alignment. Finally, the application can use the aligned 3D face model to fit the face area of ​​the figure template model, and map the 3D face map to the fitted figure template model. Finally, a high-quality 3D human-shaped figure with a high similarity to the user-uploaded picture and integrated with the specified template can be quickly obtained, thereby meeting personalized consumer needs.

[0092] In one embodiment, the step of reconstructing and repairing the original face image to obtain a 3D face model in S120 may include:

[0093] S121: Inputting the original face image into a preset face reconstruction model to obtain a three-dimensional face mesh output by the face reconstruction model.

[0094] S122: Use a hole filling technology to fill the oral cavity of the three-dimensional face mesh to obtain a filled three-dimensional face mesh.

[0095] S123: using a human face oral mesh repair technology to repair the oral cavity part of the repaired three-dimensional human face mesh to obtain a final 3D human face model.

[0096] In this embodiment, when reconstructing and repairing the original facial image, the original facial image can be first input into a preset facial reconstruction model, and the original facial image can be three-dimensionally modeled through the facial reconstruction model to obtain a three-dimensional facial mesh. The facial oral mesh repair technology is then used to repair the oral part of the completed three-dimensional facial mesh to obtain the final 3D facial model.

[0097] Among them, the face reconstruction model is a model that uses computer technology to recover information such as the three-dimensional structure and texture of the face from a given image or data. Some common face reconstruction models include but are not limited to methods based on 3D MorphableModel (3DMM), methods based on convolutional neural networks (CNN), and methods based on generative adversarial networks (GAN). Of course, the above different technologies or models can also be integrated to give full play to their respective advantages to improve the performance and effect of face reconstruction. For example, CNN is combined with GAN to use the feature extraction ability of CNN and the generation ability of GAN to achieve more accurate and realistic face reconstruction; or 3DMM is combined with deep learning methods to use the prior knowledge of 3DMM and the automatic learning ability of deep learning to improve the accuracy and efficiency of reconstruction. This application can select any one or more methods to restore the three-dimensional face mesh from the original face image according to the actual situation, without limitation here.

[0098] After the present application obtains the three-dimensional face mesh, considering that the user generally has expressions when sending in photos, especially with an open mouth, and the traditional 3DMM basic model reconstructs the face of the photo, the oral part is a non-closed mesh, which will affect the internal mapping of the oral cavity and the final 3D printing. Therefore, the present application can use the traditional hole filling technology to fill the mesh on the basis of the three-dimensional face mesh to obtain a closed three-dimensional face mesh, but because the hole filling method does not have a real effect similar to the concave inside the oral cavity, it is only a flat filling, so it is unnatural and abrupt. On this basis, the present application can use the human face oral mesh repair technology to repair the oral part of the filled three-dimensional face mesh, and then obtain the final 3D face model with a closed mesh and natural and non-obtrusive.

[0099] In one embodiment, the face reconstruction model may include a convolutional layer, a first fully connected layer, and a second fully connected layer.

[0100] Inputting the original face image into a preset face reconstruction model in S121 to obtain a three-dimensional face mesh output by the face reconstruction model may include:

[0101] S1211: Input the original face image into the convolution layer, and extract feature information in the original face image through the convolution layer.

[0102] S1212: Map the feature information extracted by the convolutional layer to the parameter space of the 3DMM through the first fully connected layer and output shape parameters.

[0103] S1213: Restore the shape parameters output by the first fully connected layer into a three-dimensional face mesh for face reconstruction through the second fully connected layer.

[0104] In this embodiment, when a three-dimensional face mesh is restored from the original face image through the face reconstruction model, since the face reconstruction model of the present application includes a convolutional layer, a first fully connected layer and a second fully connected layer, after the present application inputs the original face image into the face reconstruction model, the convolutional layer can be used to extract the feature information in the original face image, and the first fully connected layer can be used to map the extracted feature information to the parameter space of the 3DMM and then output the shape parameters, and then the second fully connected layer is used to restore the shape parameters output by the first fully connected layer to the three-dimensional face mesh of face reconstruction, so as to output the three-dimensional face mesh.

[0105] In one embodiment, Figure 2 As shown, Figure 2 A schematic diagram of a process for repairing the oral cavity in a three-dimensional face mesh provided in an embodiment of the present application; the face reconstruction model includes at least a second fully connected layer, and the second fully connected layer is used to perform nonlinear offset prediction on mesh vertices.

[0106] S123 uses a human face oral mesh repair technology to repair the oral cavity part in the repaired three-dimensional human face mesh to obtain a final 3D human face model, which may include:

[0107] S1231: Input the original face image into a preset normal prediction model to obtain a normal image output by the normal prediction model that corresponds to the original face image.

[0108] S1232: Input the vertex coordinates of the oral cavity in the completed three-dimensional face mesh to the second fully connected layer to obtain the vertex offset predicted by the second fully connected layer.

[0109] S1233: After updating the coordinates of each vertex according to the vertex offset, the updated coordinates of each vertex are optimized using the normal image until a preset optimization target is reached, thereby obtaining the optimized coordinates of each vertex.

[0110] S1234: Generate a final 3D face model according to the optimized coordinates of each vertex.

[0111] In this embodiment, face reconstruction usually uses a deep convolutional neural network (CNN) to predict the Mesh parameters of the 3DMM. The parameters of the 3DMM are decomposed by PCA, and its fitting is linear, so it is difficult to ensure the reconstruction similarity. On this basis, the present application designs a face reconstruction model, which not only has the face reconstruction function of the usual model, but also can perform nonlinear offset prediction on the mesh vertices, making the reconstruction more accurate.

[0112] That is to say, the face reconstruction model of the present application can not only restore a three-dimensional face mesh from an original face image, but also include a second fully connected layer, which is used to perform nonlinear offset prediction on mesh vertices.

[0113] Specifically, when the present application uses the facial oral mesh patching technology to patch the oral part in the completed three-dimensional facial mesh, the original facial image can be first input into a preset normal prediction model to obtain a normal image corresponding to the original facial image output by the normal prediction model, and then the vertex coordinates of the oral part in the completed three-dimensional facial mesh are input into the second fully connected layer. After obtaining the vertex offset predicted by the second fully connected layer, the coordinates of each vertex are updated according to the predicted vertex offset, and the updated vertex coordinates are optimized using the normal image until the preset optimization target is reached, thereby obtaining the optimized vertex coordinates. At this time, the final 3D facial model can be generated according to the optimized vertex coordinates.

[0114] It is understandable that in architectures such as convolutional neural networks (CNNs), the previous layers (such as convolutional layers) are mainly used to extract features of data such as images, and the fully connected layers can synthesize these extracted features for the final classification (such as determining whether the object in the image is a cat or a dog) or regression (such as predicting house prices) tasks. Therefore, when the second fully connected layer of the present application obtains the vertex coordinates of the oral part in the completed three-dimensional face mesh, the vertex offset of each vertex coordinate can be predicted, so that the offset degree of the oral part in the current three-dimensional face mesh can be determined, and then the coordinates of each vertex of the oral part are updated according to the offset degree, and then the normal image is used to optimize the updated vertex coordinates until the optimal vertex coordinates are obtained, at which time the final 3D face model can be generated.

[0115] In one embodiment, the normal prediction model may include a variational autoencoder and a U-Net network.

[0116] Inputting the original face image into a preset normal prediction model in S1231 to obtain a normal image output by the normal prediction model and corresponding to the original face image may include:

[0117] S2311: compressing the original face image into a latent space through the encoder of the variational autoencoder, and outputting the latent features of the original face image.

[0118] S2312: Generate latent features of the normal image corresponding to the latent features of the original face image through the U-Net network.

[0119] S2313: Reconstruct the potential features of the normal image back to the pixel space through the decoder of the variational autoencoder to obtain the corresponding normal image.

[0120] In this embodiment, when using the face mouth mesh repair technology to repair the mouth part in the completed three-dimensional face mesh, the normal image of the original face image can be determined first, so that the vertex coordinates of the mouth part in the three-dimensional face mesh can be optimized with reference to the normal image, thereby generating a more accurate 3D face model.

[0121] Specifically, the present application can pre-train an RGB to Normal model based on SD (stable diffusion model), that is, a normal prediction model, which can include a variational autoencoder (VAE) and a U-Net network. Among them, the input of the model is an RGB image, not a text description. Therefore, the model does not require CLIP Text Encoder to process the input information. Furthermore, the input RGB image will first be compressed into a latent space through the encoder part of the VAE to form a latent feature (Latent Feature). In the latent space, the model will simulate a diffusion process, that is, gradually add noise to the latent feature, so that it gradually transforms into a completely random noise distribution. Then, the model will learn how to predict and remove these noises to restore the original image features. However, in this RGB2Normal task, the goal of the model is not to restore the original image, but to generate the corresponding normal image. The U-Net network is the key part of the SD model, which is responsible for predicting noise and optimizing the generation process. In the RGB2Normal task, the U-Net network can receive latent features as input and try to generate latent features of the corresponding normal image. Through continuous iteration and optimization, the U-Net network can gradually learn the mapping relationship from the potential features of the RGB image to the potential features of the normal image. The potential features of the normal image obtained after processing by the U-Net network will be reconstructed back to the pixel space through the decoder part of the VAE again to form the final normal image output.

[0122] After the present application has trained the original normal prediction model through the above process, the trained normal prediction model can be used to predict the normal image of the original face image, and the normal image can be used to optimize the updated coordinates of each vertex until the preset optimization target is reached.

[0123] In one embodiment, in S1232, inputting the vertex coordinates of the oral cavity in the padded three-dimensional face mesh into the second fully connected layer to obtain the vertex offset predicted by the second fully connected layer may include:

[0124] After the number of vertices of the oral cavity in the padded three-dimensional face mesh is expanded by the second fully connected layer, the coordinates of the vertices of the oral cavity after the expansion are obtained, and nonlinear offset prediction is performed on the obtained vertex coordinates to obtain predicted vertex offsets.

[0125] In this embodiment, when predicting the vertex offset of the oral part in the three-dimensional face mesh through the second fully connected layer, the number of vertices of the oral part in the completed three-dimensional face mesh can be expanded through the second fully connected layer. For example, the present application can expand the number of vertices by interpolating between points, or use other methods to expand the number of vertices. The specific selection can be made according to the actual situation and is not limited here.

[0126] After the second fully connected layer expands the vertex data, nonlinear offset prediction can be performed on the coordinates of the expanded vertices, so that a more accurate prediction result can be obtained.

[0127] In one embodiment, Figure 3 As shown, Figure 3 A schematic diagram of a process for optimizing the updated coordinates of each vertex using a normal image provided in an embodiment of the present application; in S1233, the updated coordinates of each vertex are optimized using the normal image until a preset optimization target is reached, and the optimized coordinates of each vertex are obtained, which may include:

[0128] S2331: Projecting the updated coordinates of each vertex onto the 2D normal map, and calculating the MSE pixel loss between the 2D normal map and the normal image at the oral position;

[0129] S2332: Determine whether the MSE pixel loss converges, if so, execute S2333; otherwise, execute S2334.

[0130] S2333: Using the updated coordinates of each vertex as the optimized coordinates of each vertex.

[0131] S2334: After replacing the vertex coordinates of the oral cavity in the completed three-dimensional face mesh with the updated vertex coordinates, return to execute inputting the vertex coordinates of the oral cavity in the completed three-dimensional face mesh into the second fully connected layer, obtaining the vertex offset predicted by the second fully connected layer and its subsequent steps until the MSE pixel loss converges.

[0132] In this embodiment, when the normal image is used to optimize the updated coordinates of each vertex, the updated coordinates of each vertex can be first projected onto the 2D normal map, and the MSE pixel loss between the 2D normal map and the normal image at the oral position is calculated, and then it is determined whether the MSE pixel loss converges. If converged, it means that the 2D normal map is relatively close to the normal image. At this time, the updated coordinates of each vertex can be directly used as the optimized coordinates of each vertex; if not converged, it means that there is a certain gap between the 2D normal map and the normal image. At this time, the vertex coordinates of the oral part in the padded three-dimensional face mesh can be replaced with the updated vertex coordinates, and then the vertex coordinates of the oral part in the padded three-dimensional face mesh can be input into the second fully connected layer. After the second fully connected layer outputs the predicted vertex offset, the coordinates of each vertex can be updated again according to the vertex offset, and the normal image can be used to optimize the updated coordinates of each vertex until the preset optimization target is reached to obtain the optimized coordinates of each vertex. This process uses the normal image to continuously iteratively optimize the vertex coordinates of the oral part of the three-dimensional face mesh, thereby obtaining vertex coordinates that are more natural and consistent with the original face image.

[0133] In one embodiment, in S130, aligning the 3D face model with the face area of ​​the figure template model based on the positioning point coordinates of the 3D face model and the figure template model may include:

[0134] S131: Obtain the nose point coordinates, the back of the head point coordinates and the top of the head coordinates of the figure template model, as well as the 3D coordinates of multiple positioning points in the 3D face model.

[0135] S132: Rendering the figure template model into a 2D face image according to the nose point coordinates, the back of the head point coordinates and the top of the head coordinates.

[0136] S133: Performing face detection on the 2D face image, and obtaining 2D coordinates corresponding to each positioning point in the 3D face model according to the detection result, and converting each 2D coordinate into a 3D coordinate.

[0137] S134: Performing a rigid transformation on the 3D coordinates of the 3D face model and the 3D coordinates of the hand-figure template model to align the 3D face model to the face area of ​​the hand-figure template model.

[0138] In this embodiment, when aligning the 3D face model with the face area of ​​the hand-made template model, since the hand-made template model is a fixed type preset in the system, and each model will be marked with positioning points in advance, such as marking a point on the top of the head, a point on the nose, and a point on the back of the head corresponding to the nose, the coordinates of these three points are stored. Therefore, when the present application makes a 3D humanoid hand-made figure, the nose point coordinates, the back of the head point coordinates and the top of the head coordinates of the hand-made template model selected by the user can be directly obtained from the database, and the 2D face picture of the hand-made template model can be rendered to obtain the 2D face picture. In addition, since the 3D face model is a three-dimensional model, after the present application obtains the 2D face picture of the hand-made template model, the 2D face picture can be subjected to face detection, and after obtaining the 2D coordinates corresponding to each positioning point in the 3D face model according to the detection results, each 2D coordinate is converted into a 3D coordinate, so that the 3D face model can be aligned with the face area of ​​the hand-made template model.

[0139] In this process, the present application can also obtain the 3D coordinates of multiple positioning points in the 3D face model. For example, the key facial points reconstructed by 3DMM in the present application include but are not limited to two eyes, left and right eye corners, nose tip, two mouth corners, and chin, a total of 8 positioning points. The vertex indexes of these 8 positioning points are fixed. Therefore, when making a 3D humanoid hand, you only need to follow the index to obtain the 3D coordinates of the 8 positioning points after reconstructing the face. Then, the present application can convert the 2D coordinates of the hand template model into 3D coordinates according to the number of the 3D coordinates, and perform a rigid transformation on the 3D coordinates of the 3D face model and the 3D coordinates of the hand template model to align the 3D face model to the face area of ​​the hand template model.

[0140] It should be noted that this application can use PyTorch3D for rigid transformation when performing rigid transformation on the 3D coordinates of the 3D face model and the 3D coordinates of the hand-made template model. PyTorch3D provides a series of interfaces for rigid transformation, including but not limited to rotation, translation, combined transformation, etc. This application can select the corresponding interface for rigid transformation according to its own needs. For example, for three-dimensional mesh data meshes, the update_padded method can be used in combination with the transformation matrix to update the vertex coordinates of the mesh, thereby realizing the rigid transformation of the entire mesh.

[0141] This application uses the rigid transformation interface provided by Pytorch3D to input the 8 key points of two sets of 3D coordinates to obtain the transformation matrix R, T. At this time, the reconstructed 3DMM face model can be aligned (fitted to) the face of the hand-made template model, thereby realizing the alignment operation.

[0142] In one embodiment, in S132, rendering the figure template model into a 2D face image according to the nose point coordinates, the back of the head point coordinates and the top of the head coordinates may include:

[0143] S1321: Determine a first vector pointing from the back of the head to the nose of the figure template model and a second vector pointing from the center point of the first vector to the top of the head of the figure template model according to the coordinates of the nose tip point, the coordinates of the back of the head point and the coordinates of the top of the head.

[0144] S1322: construct a camera coordinate system according to the first vector and the second vector, and after setting corresponding camera intrinsic parameters, render the figure template model into a 2D face image through the camera coordinate system and the camera intrinsic parameters.

[0145] In this embodiment, when rendering the hand-figure template model as a 2D face image based on the nose point coordinates, back of the head point coordinates and top of the head coordinates of the hand-figure template model, the first vector pointing from the back of the head to the nose of the hand-figure template model and the second vector pointing from the center point of the first vector to the top of the head of the hand-figure template model can be determined based on the nose point coordinates, back of the head point coordinates and top of the head coordinates, and then a camera coordinate system is constructed based on the first vector and the second vector, and after setting the corresponding camera intrinsic parameters, the hand-figure template model is rendered as a 2D face image through the camera coordinate system and the camera intrinsic parameters.

[0146] In a specific implementation method, the present application can mark the three positioning points annotated on the hand-made template model as nose, back, and top, respectively, where nose represents the nose point, back represents the back of the head point, and top represents the top of the head. The first vector front_v pointing from the back of the head to the nose is obtained through nose-back, and the middle point is marked as center, and then the second vector top_v pointing from the center point to the top of the head is obtained by top-center; at the same time, the present application can obtain the center origin position of the camera that needs to be set (facing the direction of the nose of the template face) through the calculation method of front_v*4.5, and record it as the z-axis, and then calculate the normal perpendicular to this plane through the plane formed by top_v, recorded as the x-axis, and then calculate the normal through the plane formed by the z-axis and the x-axis to obtain the y-axis, and construct a new camera coordinate system with the set camera origin position, and set the corresponding camera intrinsic parameters at the same time, and then render the template model through the Pytorch3D interface to obtain the 2D face image corresponding to the template model.

[0147] Next, the present application can use an open source 2D face key point detection model to perform face detection on the rendered 2D face image, and simultaneously obtain the 2D coordinates of 8 positioning points with the same meaning as the 3D positioning points.

[0148] In one embodiment, in S140, fitting the face area of ​​the figure template model using the aligned 3D face model to obtain the fitted figure template model may include:

[0149] S141: Multiply each vertex of the face area in the figure template model by the matrix parameters of the preset deformation matrix to obtain a fitted figure template model that fits the aligned 3D face model and maintains the structure of the figure template model.

[0150] In this embodiment, when the aligned 3D face model is used to fit the face area of ​​the figure template model, the fitting operation can be performed by using the model retopology technology. Specifically, the present application can first obtain the matrix parameters corresponding to the preset deformation matrix, which are obtained by optimizing each vertex of the face area in the figure template model and each vertex of the aligned 3D face model.

[0151] It is understandable that the deformation matrix is ​​a mathematical tool used to describe the shape change of an object. It is widely used in many fields such as computer graphics, computer vision, robotics and physics. It is usually a square matrix, whose elements represent the deformation information of the object in various dimensions, such as expansion, rotation, shear, etc.

[0152] After the present application uses the vertices of the facial area in the hand-figure template model and the vertices of the aligned 3D face model to optimize the matrix parameters in the deformation matrix, the optimized deformation matrix can be used to perform fitting operations on the vertices of the facial area in the hand-figure template model, thereby obtaining a fitted hand-figure template model that fits the aligned 3D face model and maintains the structure of the hand-figure template model.

[0153] In one embodiment, the optimization process of the matrix parameters of the deformation matrix in S141 may include:

[0154] S411: Obtain a first vertex set formed by each vertex of the face area in the sample figure template model, and a second vertex set formed by each vertex of the aligned sample 3D face model.

[0155] S412: Setting matrix parameters of the deformation matrix.

[0156] S413: Use the KNN nearest neighbor algorithm to find the vertex in the second vertex set that is closest to each vertex in the first vertex set, and optimize the matrix parameters according to the loss function between each vertex pair to obtain optimized matrix parameters.

[0157] In this embodiment, when optimizing the matrix parameters in the deformation matrix, the present application can pre-acquire multiple sample hand-figure template models and multiple aligned sample 3D face models, so that after obtaining the first vertex set formed by the vertices of the facial area in the sample hand-figure template model and the second vertex set formed by the vertices of the aligned sample 3D face model, the KNN nearest neighbor algorithm can be used to find the vertex in the second vertex set that is closest to each vertex in the first vertex set, and optimize the matrix parameters in the pre-set deformation matrix according to the loss function between each vertex pair, so as to obtain the final optimized matrix parameters.

[0158] For example, the present application can first define the model of the hand-made template as "Ms", the number of vertices as "Ns", the 3D face model as "Mt", and the number of vertices as "Nt"; then design the neural network optimization parameters as a 4*4 deformation matrix, the number of which is Ns, that is, an Ns*4*4 matrix; in each network iteration process, first use the KNN nearest neighbor algorithm to find the nearest vertex in Mt from each vertex of Ms, and consider the position of this point to be fitted by Ms (distances beyond the threshold are not fitted, such as non-face areas). The Ns*4*4 matrix parameters to be optimized are updated through the loss function (point-by-point Euclidean distance, rigid constraints). The optimization parameters are learned through about 150 rounds of iterations. Finally, each vertex in Ms is multiplied by the corresponding matrix parameters learned to obtain a face reconstruction Mesh that fits Mt and maintains the Ms structure.

[0159] In one embodiment, mapping the 3D face map onto the fitted figure template model in S140 to obtain a 3D human figure may include:

[0160] S142: Expand the fitted figure template model and the facial area of ​​the aligned 3D face model into the UV space, and scale the expanded figure template model and the 3D face model to the same resolution.

[0161] S143: Mapping the texture colors of the 3D face texture to the face area of ​​the scaled figure template model one by one according to the position mapping relationship between the scaled figure template model and the 3D face model.

[0162] S144: The color of the interface between the face and the body of the mapped figure template model is transitioned through Poisson fusion to obtain the final 3D humanoid figure.

[0163] In this embodiment, in the process of mapping the 3D face map to the fitted hand-figure template model, the facial area of ​​the fitted hand-figure template model and the aligned 3D face model can be first expanded into the UV space, and then the expanded hand-figure template model and the 3D face model can be scaled to the same resolution. In this way, the texture color of the 3D face map can be mapped one by one to the facial area of ​​the scaled hand-figure template model according to the position mapping relationship between the scaled hand-figure template model and the 3D face model. Then, the present application can also transition the color of the junction between the face and body of the mapped hand-figure template model through Poisson fusion, thereby obtaining the final 3D humanoid hand.

[0164] It can be understood that "U" and "V" in the UV space are two coordinate axes in the texture coordinate system, similar to the "X" and "Y" axes in a two-dimensional plane. The UV space is mainly used to accurately map a 2D texture image to the surface of a 3D model. In the UV space, the value range of the U axis and the V axis is usually from 0 to 1. The entire UV space represents a square area, where (0, 0) represents the lower left corner of the square and (1, 1) represents the upper right corner. The present application can achieve seamless texture mapping by reasonably setting the UV coordinates, and the setting of the UV space can control the degree of stretching and deformation of the texture on the surface of the model, that is, the present application can scale the unfolded hand template model and the 3D face model to the same resolution in the UV space, so that the position mapping can be performed more accurately. After determining the position mapping relationship between the scaled hand template model and the 3D face model, the texture color of the 3D face map can be mapped one by one to the face area of ​​the scaled hand template model, so that the color of the face area of ​​the hand template model is consistent with the color of the 3D face map.

[0165] In addition, since the body area of ​​the figure template model is a pre-set color, when the color of the facial area of ​​the figure template model is unified with the color of the 3D face map, the present application can also use Poisson fusion to transition the color of the junction between the face and body of the mapped figure template model, thereby obtaining the final 3D humanoid figure.

[0166] Indicatively, Figure 4 As shown, Figure 4 A schematic diagram of a model for converting a figure template model into a 3D humanoid figure provided in an embodiment of the present application; Figure 4 It can be seen that the 3D humanoid figure generation method of the present application can be used to generate Figure 4 The template model on the far left is quickly made into a 3D human-shaped figure on the right that matches the user's consumption needs.

[0167] The following is a description of a 3D humanoid figure generating device provided in an embodiment of the present application. The 3D humanoid figure generating device described below and the 3D humanoid figure generating method described above can be referenced to each other.

[0168] In one embodiment, Figure 5 As shown, Figure 5 A schematic diagram of the structure of a 3D humanoid hand-generating device provided in an embodiment of the present application; the present application also provides a 3D humanoid hand-generating device, including a data acquisition module 210, a face reconstruction module 220, a face alignment module 230, and a retopology module 240, which specifically include the following:

[0169] The data acquisition module 210 is used to acquire the original face image uploaded by the user and the selected figure template model.

[0170] The face reconstruction module 220 is used to reconstruct and repair the original face image to obtain a 3D face model, and to align the original face image and map it onto a grid to obtain a 3D face map.

[0171] The face alignment module 230 is used to align the face area of ​​the 3D face model with the face area of ​​the hand-figure template model based on the positioning point coordinates of the 3D face model and the hand-figure template model.

[0172] The retopology module 240 is used to fit the face area of ​​the figure template model using the aligned 3D face model, and then map the 3D face map onto the fitted figure template model to obtain a 3D human figure.

[0173] In the above embodiment, after obtaining the original face image uploaded by the user and the selected hand-figure template model, the original face image can be reconstructed and repaired to obtain a 3D face model. In this way, the two-dimensional image can be converted into a three-dimensional face model, and the face can be repaired during the reconstruction process to solve the problem that the hollow mouth cannot be printed, so that the final 3D face model presents a natural state, so as to better integrate with the hand-figure template model; then, the present application can align the original face image and map it to the grid to obtain a 3D face map, and then based on the 3D The coordinates of the positioning points of the face model and the figure template model are used to align the 3D face model with the face area of ​​the figure template model, thereby greatly saving the workload of manual annotation points, improving work efficiency while ensuring the high quality of alignment. Finally, the application can use the aligned 3D face model to fit the face area of ​​the figure template model, and map the 3D face map to the fitted figure template model. Finally, a high-quality 3D human-shaped figure with a high similarity to the user-uploaded picture and integrated with the specified template can be quickly obtained, thereby meeting personalized consumer needs.

[0174] In one embodiment, the present application also provides a computer-readable storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the 3D humanoid figure generation method as described in any of the above embodiments.

[0175] In one embodiment, the present application also provides a computer device, including: one or more processors, and a memory.

[0176] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the 3D human figure generation method described in any one of the above embodiments are performed.

[0177] Indicatively, Figure 6 As shown, Figure 6 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 300 may be provided as a server. Figure 6 The computer device 300 includes a processing component 302, which further includes one or more processors, and a memory resource represented by a memory 301 for storing instructions executable by the processing component 302, such as an application. The application stored in the memory 301 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 302 is configured to execute instructions to perform the 3D humanoid hand generation method of any of the above embodiments.

[0178] The computer device 300 may further include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate based on an operating system stored in the memory 301, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0179] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0180] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0181] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0182] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a 3D humanoid figure, characterized in that: The method comprises: Obtain the original face image uploaded by the user and the selected figure template model; Reconstructing and repairing the original face image to obtain a 3D face model, and aligning the original face image and mapping it onto a grid to obtain a 3D face map; Based on the coordinates of the positioning points of the 3D face model and the figure template model, aligning the face area of ​​the 3D face model with the face area of ​​the figure template model; After fitting the face area of ​​the figure template model using the aligned 3D face model, the 3D face map is mapped onto the fitted figure template model to obtain a 3D humanoid figure.

2. The method for generating a 3D humanoid figure according to claim 1, characterized in that: The step of reconstructing and repairing the original face image to obtain a 3D face model comprises: Inputting the original face image into a preset face reconstruction model to obtain a three-dimensional face mesh output by the face reconstruction model; Using a hole filling technique to fill the oral cavity of the three-dimensional face mesh, to obtain a filled three-dimensional face mesh; The oral cavity mesh patching technology is used to patch the oral cavity part of the completed three-dimensional face mesh to obtain a final 3D face model.

3. The method for generating a 3D humanoid figure according to claim 2, characterized in that: The face reconstruction model includes a convolutional layer, a first fully connected layer and a second fully connected layer; The step of inputting the original face image into a preset face reconstruction model to obtain a three-dimensional face mesh output by the face reconstruction model includes: Inputting the original face image into the convolution layer, and extracting feature information in the original face image through the convolution layer; Mapping the feature information extracted by the convolutional layer to the parameter space of the 3DMM through the first fully connected layer and outputting shape parameters; The shape parameters output by the first fully connected layer are restored to a three-dimensional face mesh for face reconstruction through the second fully connected layer.

4. The method for generating a 3D humanoid figure according to claim 2, characterized in that: The face reconstruction model comprises at least a second fully connected layer, wherein the second fully connected layer is used for performing nonlinear offset prediction on mesh vertices; The method of using the human face oral mesh repair technology to repair the oral cavity part of the repaired three-dimensional human face mesh to obtain the final 3D human face model includes: Inputting the original face image into a preset normal prediction model to obtain a normal image output by the normal prediction model and corresponding to the original face image; Inputting the vertex coordinates of the oral cavity in the padded three-dimensional face mesh into the second fully connected layer to obtain the vertex offset predicted by the second fully connected layer; After updating the coordinates of each vertex according to the vertex offset, optimizing the updated coordinates of each vertex using the normal image until a preset optimization target is reached, thereby obtaining optimized coordinates of each vertex; The final 3D face model is generated according to the optimized coordinates of each vertex.

5. The method for generating a 3D humanoid figure according to claim 4, characterized in that: The normal prediction model includes a variational autoencoder and a U-Net network; The step of inputting the original face image into a preset normal prediction model to obtain a normal image output by the normal prediction model and corresponding to the original face image comprises: Compressing the original face image into a latent space through an encoder of the variational autoencoder, and outputting latent features of the original face image; Generate latent features of a normal image corresponding to latent features of the original face image through the U-Net network; The decoder of the variational autoencoder reconstructs the potential features of the normal image back into the pixel space to obtain a corresponding normal image.

6. The method for generating a 3D humanoid figure according to claim 4, characterized in that: The step of inputting the vertex coordinates of the oral cavity in the padded three-dimensional face mesh into the second fully connected layer to obtain the vertex offset predicted by the second fully connected layer includes: After the number of vertices of the oral cavity in the padded three-dimensional face mesh is expanded by the second fully connected layer, the coordinates of the vertices of the oral cavity after the expansion are obtained, and nonlinear offset prediction is performed on the obtained vertex coordinates to obtain predicted vertex offsets.

7. The method for generating a 3D humanoid figure according to claim 4, characterized in that: The method of optimizing the updated coordinates of each vertex by using the normal image until a preset optimization target is reached to obtain the optimized coordinates of each vertex includes: Projecting the updated coordinates of each vertex onto a 2D normal map, and calculating the MSE pixel loss between the 2D normal map and the normal image at the oral position; Determining whether the MSE pixel loss converges; If yes, the updated vertex coordinates are used as optimized vertex coordinates; Otherwise, after replacing the vertex coordinates of the oral part in the padded three-dimensional face mesh with the updated vertex coordinates, return to execute the step of inputting the vertex coordinates of the oral part in the padded three-dimensional face mesh into the second fully connected layer, obtaining the vertex offset predicted by the second fully connected layer and its subsequent steps until the MSE pixel loss converges.

8. The method for generating a 3D humanoid figure according to any one of claims 1 to 7, characterized in that: The aligning the 3D face model with the face area of ​​the hand-figure template model based on the positioning point coordinates of the 3D face model and the hand-figure template model includes: Obtaining the coordinates of the nose tip point, the back of the head point and the top of the head of the figure template model, as well as the 3D coordinates of multiple positioning points in the 3D face model; Rendering the figure template model into a 2D face image according to the nose point coordinates, the back of the head point coordinates and the top of the head coordinates; Performing face detection on the 2D face image, and obtaining 2D coordinates corresponding to each positioning point in the 3D face model according to the detection result, and converting each 2D coordinate into a 3D coordinate; The 3D coordinates of the 3D face model and the 3D coordinates of the hand-figure template model are rigidly transformed to align the 3D face model to the face area of ​​the hand-figure template model.

9. The method for generating a 3D humanoid figure according to claim 8, characterized in that: The rendering of the figure template model into a 2D face image according to the nose point coordinates, the back of the head point coordinates and the top of the head coordinates includes: Determine, according to the nose tip point coordinates, the back of the head point coordinates and the top of the head coordinates, a first vector pointing from the back of the head of the figure template model to the nose tip, and a second vector pointing from the center point of the first vector to the top of the head of the figure template model; After constructing a camera coordinate system according to the first vector and the second vector and setting corresponding camera intrinsic parameters, the figure template model is rendered as a 2D face image through the camera coordinate system and the camera intrinsic parameters.

10. The method for generating a 3D humanoid figure according to any one of claims 1 to 7, characterized in that: The step of fitting the face area of ​​the figure template model using the aligned 3D face model to obtain the fitted figure template model includes: Multiplying each vertex of the face area in the figure template model by the matrix parameters of the preset deformation matrix to obtain a fitted figure template model that fits the aligned 3D face model and maintains the structure of the figure template model; The matrix parameters of the preset deformation matrix are obtained by optimizing each vertex of the face area in the figure template model and each vertex of the aligned 3D face model.

11. The method for generating a 3D humanoid figure according to claim 10, characterized in that: The optimization process of the matrix parameters of the deformation matrix includes: Obtaining a first vertex set formed by each vertex of the face area in the sample figure template model and a second vertex set formed by each vertex of the aligned sample 3D face model; Set the matrix parameters of the deformation matrix; A KNN nearest neighbor algorithm is used to find the vertex in the second vertex set that is closest to each vertex in the first vertex set, and the matrix parameters are optimized according to the loss function between each vertex pair to obtain optimized matrix parameters.

12. The method for generating a 3D humanoid figure according to any one of claims 1 to 7, characterized in that: The step of mapping the 3D face map onto the fitted figure template model to obtain a 3D humanoid figure comprises: Expand the fitted figure template model and the face area of ​​the aligned 3D face model into UV space, and scale the expanded figure template model and the 3D face model to the same resolution; According to the position mapping relationship between the scaled figure template model and the 3D face model, mapping the texture colors of the 3D face texture to the face area of ​​the scaled figure template model one by one; The color of the junction between the face and body of the mapped figure template model is transitioned through Poisson fusion to obtain the final 3D human figure.

13. A 3D humanoid figure generating device, characterized in that: include: The data acquisition module is used to obtain the original face image uploaded by the user and the selected figure template model; A face reconstruction module, used to reconstruct and repair the original face image to obtain a 3D face model, and to align the original face image and map it onto a grid to obtain a 3D face map; A face alignment module, used to align the face area of ​​the 3D face model with the face area of ​​the hand-figure template model based on the positioning point coordinates of the 3D face model and the hand-figure template model; The retopology module is used to fit the face area of ​​the figure template model with the aligned 3D face model, and then map the 3D face map onto the fitted figure template model to obtain a 3D human figure.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the 3D humanoid figure generation method as described in any one of claims 1 to 12.

15. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the 3D humanoid figure generation method as described in any one of claims 1 to 12 are performed.

Images