A whole-body reconstruction method based on a single photo

Abstract

The present application discloses a method for full-body reconstruction based on a single photo, the method comprising: extracting a user's face image and generating a head model hm, forming a spliced ​​model map ht based on the fusion of hm and the texture map of the user's face; constructing a full-body model full_model, setting an array corre of mapping relationships between hm and the head vertices on full_model; constructing a transformation equation, replacing the vertex h_v_sub on the facial area with the corresponding position of full_model; reconstructing the uv coordinates of full_mode according to the corresponding relationship between the facets of hm and full_model, and performing texture reconstruction; prefabricating a full-body map f_pref, and converting and mapping the model map ht to the prefabricated full-body map f_pref through the uv coordinates and indexes of the reconstructed full_mode, to generate a complete full-body map that can be directly applied to a 3D model. The full-body reconstruction method of the present application can not only achieve high-quality full-body detail restoration, but also significantly improve the reconstruction efficiency, and can generate a realistic three-dimensional full-body model.

Description

Technical Field

[0001] The present application belongs to the technical field of three-dimensional digital human, and specifically relates to a whole-body reconstruction method based on a single photo. Background Art

[0002] 3D reconstruction technology is an important direction in computer vision and has been widely used in virtual reality, augmented reality, medical imaging, games, digital humans and other fields. In the field of 3D reconstruction, traditional reconstruction techniques usually rely on multi-view images or video sequences to capture complete information of the object. The main methods include 3D scanning based on structured light, stereo vision technology, and the use of depth cameras.

[0003] In recent years, with the rapid development of deep learning and computer vision technology, 3D reconstruction methods based on a single photo have gradually become a research hotspot. This type of method analyzes the features in a single image and uses a deep learning model to infer the 3D shape of an object. The advantage of this technology is that it is easy to operate and can perform 3D reconstruction with just one photo, which greatly reduces the technical threshold for users.

[0004] However, existing single-image reconstruction techniques still have significant deficiencies in detail restoration, texture mapping, and lighting processing. Although some algorithms can achieve preliminary reconstruction of facial features, they often lack fine processing in the detailed representation of the whole body model, resulting in insufficient realism in the reconstruction results.

[0005] In addition, current methods often lead to unstable model reconstruction when dealing with different lighting conditions, posture changes and background complexity, resulting in unnatural combination of face and body parts, affecting the final user experience. Summary of the invention

[0006] The purpose of this application is to provide a whole-body reconstruction method based on a single photo, so as to achieve high-quality whole-body model reconstruction through advanced algorithms and improve the efficiency and accuracy of reconstruction.

[0007] According to the above purpose, the first aspect of the present application provides a whole body reconstruction method based on a single photo, the method comprising:

[0008] Get the user's complete facial photo;

[0009] Extracting the user's face image from the face photo and generating a head model hm, and fusing the head model hm with the texture map of the user's face to form a spliced ​​model map ht;

[0010] Filter out all vertices h_v_sub in the facial area of ​​the head model hm, and obtain vertex indexes h_v_subindex of all vertices h_v_sub on the head model hm;

[0011] Construct a full-body model full_model, set the mapping relationship array corre between the head model hm and the head vertices on the full-body model full_model, and ensure that the k-th point on the head model hm corresponds to the corre[k]-th point of the full-body model full_model;

[0012] Construct the transformation equation to replace the vertex h_v_sub on the facial area with the corresponding position of the full body model full_model;

[0013] According to the facet correspondence between the head model hm and the full body model full_model, the uv coordinates of the full body model full_mode are reconstructed to perform texture reconstruction;

[0014] A full-body texture f_pref is prefabricated, and the model texture ht is converted and mapped to f_pref through the uv coordinates and indexes of the reconstructed full-body model full_mode, so as to generate a complete full-body texture that can be directly applied to the 3D model.

[0015] Further, forming a spliced ​​model map ht based on the fusion of the head model hm and the texture map of the user's face includes:

[0016] Get the user's face image;

[0017] Generate a high-modulus mesh model M_h from the acquired face image based on the FFHQ-UV algorithm, where M_h=(V_h, F_h), V_h is a vertex set, and F_h is a face set;

[0018] The generated high-poly mesh model M_h is merged with the lightweight low-poly mesh M_l to generate a lightweight low-poly mesh M_l';

[0019] After adjusting the eyeball position on the low-poly mesh M_l', the final head model hm is obtained;

[0020] Detect multiple facial key points in the original face image and align them with multiple pre-labeled key points in the head model hm, project the head model hm into the two-dimensional face image, and extract the facial texture map T_face;

[0021] The occluded area is excluded by the face segmentation algorithm to generate a mask area, and a gradient transition area image mask T_transition is generated at the edge of the mask area;

[0022] Generate a skin color reference map C_ref, match the prefabricated map T_pre with the skin color reference map C_ref, and generate an adjusted map T_pre';

[0023] Using the generated mask T_transition containing the transition area, the texture map T_face of the user's face is weightedly fused with the adjusted pre-made map T_pre' to form a spliced ​​head model map ht.

[0024] Furthermore, adjustments are made to key vertices around the eyes, and the position of the eyeball is fine-tuned so that the adjusted vertices fit on the surface of the eyeball.

[0025] Furthermore, replacing the vertex h_v_sub on the facial region to the corresponding position of the full body model full_model includes:

[0026] Construct the transformation equation and use singular value decomposition to calculate the rotation matrix R, translation vector t, and scaling parameter s from the vertex h_v_sub on the face to the corre[h_v_subindex] position of the whole body vertex. After h_v_sub is transformed, it replaces the corre[h_v_subindex] vertex of the whole body, so that the facial vertex is accurately replaced to the corresponding position of the whole body model. At the same time, the natural transition of the model is achieved through the following optimization equation:

[0027]

[0028] where h v_sub is a subset of facial vertices, corre[h v_subindex ] is the subset point in the whole body model v full In the corresponding index, i is the index of the subset point, from 0 to the number of subset vertices N-1, R is the rotation matrix, t is the translation vector t, and s is the scaling factor.

[0029] Furthermore, the method further comprises: using a Laplace smoothing algorithm to perform multiple smoothing operations on the seam area, wherein the smoothing formula is as follows:

[0030]

[0031] Among them, v i are the vertices to be smoothed, Yes i The set of vertices in the neighborhood, is the number of vertices, v t represent One of the points in the neighborhood, traverse all the points in the neighborhood and sum them, and replace v with the average of all the points in the neighborhood i , smoothing is completed.

[0032] Furthermore, according to the facet correspondence between the head model hm and the full body model full_model, the uv coordinates of the full body model full_model are reconstructed, and the texture reconstruction includes:

[0033] First, the uv coordinates in the head model hm are appended below the uv coordinates in the full-body model full_model. Then, for each face fa, fb, fc of the head model hm, and based on the corresponding uv indexes fta, ftb, ftc, the corresponding face is found in the full-body model. Then, the uv index corresponding to this corresponding face is replaced with m_full+fta, m_full+ftb, m_full+ftc, where m_full is the length of the original uv coordinate in the full-body model.

[0034] Furthermore, a full-body texture f_pref is pre-made, and after the model texture ht and the full-body texture f_pref are converted to the LAB color space, the brightness of the L channel is adjusted so that the brightness of the full-body texture f_pref is close to the brightness of the model texture ht.

[0035] Furthermore, after the integration is completed, the 3D model data is exported into different formats, including obj, fbx, and glb formats.

[0036] In a second aspect of the present application, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the whole body reconstruction method as described in the first aspect is executed.

[0037] The whole-body reconstruction method based on a single photo disclosed in this application can not only achieve high-quality restoration of facial and whole-body details, but also significantly improve the reconstruction efficiency and can quickly generate a realistic three-dimensional whole-body model. Through the technology of whole-body texture mapping and automatic skin color matching, the problem of inconsistent skin color and texture under changing lighting in traditional methods is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the flow of a whole-body reconstruction method based on a single photo in an embodiment of the present invention.

[0039] FIG. 2( a ) is a schematic diagram of a head model hm in an embodiment of the present invention.

[0040] FIG. 2( b ) is a schematic diagram of a head model map ht in an embodiment of the present invention.

[0041] Figure 3 A schematic diagram of the fixed points of the screened facial region in an embodiment of the present invention.

[0042] Figure 4 It is a schematic diagram of a prefabricated whole body model in an embodiment of the present invention.

[0043] Figure 5 Schematic diagram of a reconstructed whole body model in an embodiment of the present invention.

[0044] Figure 6 It is a schematic diagram of a prefabricated full-body map in an embodiment of the present invention.

[0045] Figure 7 Schematic diagram of a reconstructed full-body map in an embodiment of the present invention.

[0046] Figure 8 It is a schematic diagram of a rendered three-dimensional model in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] It should be noted that the diagrams provided in this embodiment are only used to illustrate the basic concept of the present invention. The structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the conditions under which the present invention can be implemented. Any modification of the structure, change in the proportion relationship or adjustment of the size should still fall within the scope of the technical content disclosed by the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0049] The directions or positional relationships indicated by the words "front", "rear", "left", "right", "middle", "longitudinal", "lateral", "horizontal", "inside", "outside" and the like in this specification are based on the directions or positional relationships shown in the drawings and are only for the convenience of simplifying the description. They do not indicate or imply that the devices or elements referred to must have a specific direction, be constructed and operate in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0050] Embodiment 1

[0051] Reference Figures 1 to 8 As shown, the embodiment of the present invention discloses a whole body reconstruction method based on a single photo, which specifically includes the following process:

[0052] 1. Use a mobile phone, camera or other camera equipment to take a photo with even lighting, clear face and the user's complete face, then extract the user's face image from the face photo and generate a head model hm, and merge the head model hm with the texture map of the user's face to form a spliced ​​head model map ht. In addition, you can also set the head material hl, as well as the two eyeball maps leye_ht and reye_ht. hl is the file that associates hm and ht.

[0053] Before performing full-body three-dimensional modeling, a head model map is created first. The creation of the head model map and the patent previously applied for by the applicant, "Head reconstruction method based on a single photo (application number 2024117669223)", have been applied for and disclosed, so they will not be repeated here and only a brief description will be given.

[0054] Specifically, the FFHQ-UV method is used to generate a high-modulus mesh model M_h = (V_h, F_h) for the extracted face image I, where V_h is a vertex set and F_h is a face set. The algorithm optimizes the distribution and structure of mesh vertices so that the generated model can show the geometric details of the face, such as the nose, eye corners and lip contours.

[0055] Specifically, after generating the high-poly mesh M_h, the system fuses it with the lightweight low-poly meshM_l obtained by the wrapping algorithm. In order to establish a correspondence between mesh models of different resolutions, it can ensure that the consistency of features and shapes is retained when the model resolution is changed. The implementation process is: for each vertex v_l on the low-poly mesh M_l, find the corresponding three nearest vertices v_h1, v_h2, and v_h3 on the high-poly mesh M_h, calculate the Euclidean distance with all points on the high-poly mesh, sort and take the smallest three points, which are the nearest three points, and then calculate the weight coefficients w_1, w_2, and w_3, and use linear interpolation to generate the lightweight mesh M_l'. The formula is as follows: v_l=w_1*v_h1+w_2*v_h2+w_3*v_h3,where w_1+w_2+w_3=1.

[0056] Specifically, due to the complex geometric features of the eyeball and eye socket areas, the system needs to optimize the mesh vertices in these areas. Since the FLAME head model needs to be reconstructed, the common face mesh model FLAME usually provides predefined area indexes. For example, the vertex index of the FLAME eyeball is 3931~5022. Through these indexes, the vertex V_eyes of the eyeball position can be found and its position can be adjusted through a smoothing algorithm. By adjusting the position of each vertex to the average of its neighboring vertices, the drastic changes in the vertex position are reduced. The basic idea is to gradually make the vertex approach the average position of its neighbors, smooth the local network, prevent the occurrence of model penetration, and obtain the final head model hm. The optimization process uses a curve fitting method:

[0057] V_smooth=α*V_original+(1-α)*V_neighbors,

[0058] Among them, V_original represents the original position of the vertex. In each smoothing iteration, the new smoothing position can be calculated based on the initial original vertex position. V_smooth represents the new position of the vertex after smoothing, that is, the new position to which the current vertex is to be updated after this smoothing iteration. α is the smoothing coefficient, and V_neighbors is the set of neighboring vertices, which ensures that the eye area is visually real and natural.

[0059] Specifically, the system detects 68 facial key points P_img={p_1, p_2, ..., p_68} in the face image I through an algorithm, and aligns them with the 68 pre-labeled key points P_mesh in the hm model. The PnP (Perspective-n-Point) algorithm is used to solve the projection matrix M_p of the model and map the three-dimensional key points to the two-dimensional image plane:

[0060] M_p=K*[R|t];

[0061] Among them, K is the camera intrinsic parameter matrix, [R|t] is the rotation and translation matrix. This process ensures the accurate registration between the model and the image. After projecting the head model hm onto the two-dimensional image, the facial texture map T_face can be extracted.

[0062] Specifically, after extracting the facial texture map T_face, the system excludes the occluded area (such as hair) through the face segmentation algorithm, generates a mask area mask_face, and generates a gradient transition area mask T_transition at the edge of the mask area to ensure a smooth transition of the map during fusion.

[0063] Specifically, a skin color reference map C_ref can be generated by using the mask image. This is used to adjust the pre-made texture skin color and the facial skin color to match. Because the face of the texture finally generated is the user's, and the other parts are pre-made, a fusion operation is required.

[0064] Specifically, the prefabricated texture T_pre and the skin color reference image C_ref are converted to the Lab color space, and the value range of the L channel is adjusted to match the prefabricated texture with the reference image. The adjusted texture is represented as T_pre'. In addition, the brightness of the prefabricated eye texture is adjusted to obtain the adjusted eye textures T_lefteye and T_righteye, so that the eye area is displayed more realistically on the face.

[0065] Afterwards, the mask T_transition containing the transition area generated previously is used to weightedly fuse the texture map T_face of the user's face with the adjusted prefabricated map T_pre' to ensure seamless connection and natural transition between the face and the prefabricated part, and finally obtain the spliced ​​head model map ht.

[0066] ht=T_transition*T_face+(1-T_transition)*T_pre'

[0067] This method effectively solves the problem of splicing different materials and colors, making the generated head model more natural and realistic visually.

[0068] Finally, the generated model (hm, ht) is loaded and rendered to form a rendering of the model after 3D reconstruction.

[0069] For example, Meshlab can be used to load and render the generated model (hm, ht) to display the model's geometric structure and texture effects, so that users can verify the results of the reconstruction. For example, the model data hm can be directly exported as an .obj file by calling a general library function, and the texture data ht can be saved as a .png file format. These two files correspond to the model's geometric structure and texture map, respectively, and can be directly loaded and rendered by mainstream 3D software (such as Blender, Maya, 3ds Max, and Unreal Engine, etc.).

[0070] The above process completes the head model hm and the textured head model map ht, as shown in Figure 2(a) and Figure 2(b) respectively. Through the high-precision reconstruction algorithm, a realistic and detailed three-dimensional head model is generated, ensuring the subsequent integration effect with the whole body model, laying a solid foundation for generating a complete model with natural transition.

[0071] 2. Manually filter all vertices h_v_sub in the facial area on the head model hm, and calculate the point closest to each point in hm, so as to obtain the vertex index h_v_subindex of h_v_sub on hm. Among them, hm represents a complete head model, and h_v_sub is a part of the point set of the head model. If you need to determine the index position of a point a in h_v_sub in hm, that is, its position number in hm, you can find the nearest point by calculating the distance from a to each point on hm, so as to obtain the corresponding index. This step ensures the accurate extraction of facial feature vertices, lays the foundation for the subsequent combination of facial data and full-body models, provides accurate spatial position reference, and avoids errors and unnatural effects during splicing. The filtered facial area vertices are as follows: Figure 3 shown.

[0072] 3. The full-body model is built based on SMPL-X, called full_model. The corresponding body shape can also be adjusted through the user's full-body photos. In the SMPL-X model, gender setting is the basic parameter for generating a full-body model. The model supports three gender options: male, female, and neutral. The full-body models of each gender have different performances in body shape characteristics. For example, the male model pays more attention to muscle and broad shoulder structure, while the female model shows a softer body shape and narrower shoulders. Users can choose the appropriate gender model according to their needs, laying the foundation for subsequent detail adjustments. In addition, the choice of gender parameters also affects the detailed proportions of the full-body model, such as the length of the limbs and the structural morphology of the bones, so that the generated full_model is more in line with the actual situation. The shape parameter can control the height, fatness and thinness of the model. By setting male and female, and pre-making some height and fatness parameters, a full-body model can be generated, such as Figure 4 shown.

[0073] Fourth, in order to achieve accurate matching between the face and the whole body model, a head vertex mapping relationship array corre is defined to associate the vertices of the whole body model with the independent face model. On the independent face model (such as hm), the kth point (0≤k≤5022) is mapped to the corre[k]th point of full_model through the corre array. The definition of the corre array is based on the preset data of the SMPL-X model. These data are defined and fixed by the user during the model training process to ensure consistency in different application scenarios. Through this mapping relationship, the geometric features of the user's face (such as facial features, facial expressions, etc.) can be seamlessly transferred to the head area of ​​the whole body model to ensure the consistency of details and the coordination between the whole body and the local. With this mapping relationship, the high-precision facial data obtained through the face model (such as hm) can be directly mapped to the head area of ​​the whole body model, so as to quickly generate a full_model with rich details and true proportions.

[0074] 5. Construct the transformation equation and use singular value decomposition to calculate the rotation matrix R, translation vector t, and scaling parameter s from the vertex h_v_sub on the face to the corre[h_v_subindex] position of the vertex on the whole body model, and transform h_v_sub to replace the corre[h_v_subindex] vertex on the whole body. This step ensures that the facial vertices are accurately replaced to the corresponding positions of the whole body model, and further ensures the natural transition of the model. The following optimization equation is used:

[0075]

[0076] Among them, h v_sub is a subset point set (a subset of facial vertices), corre[h v_subindex ] is the subset point in the whole body model v full The corresponding index in . i is the index of the subset point (from 0 to the number of subset vertices N-1), R is the rotation matrix (3×3), and t is the translation vector t. s is the scaling factor (scalar). The specific implementation process is as follows:

[0077] First, decentralized processing can reduce the impact of translation:

[0078]

[0079] Then construct the covariance matrix: Perform singular value decomposition of the covariance matrix C = U∑V T , decompose the covariance matrix into three parts to calculate the rotation matrix R = VU T, the rotation matrix R has been aligned with the direction of the subset vertices, but the size (scale) of the subset vertices may be inconsistent with the whole body vertices. We compare the overall size difference between the subset vertices and the target vertices. Based on this size difference, we derive a unified scale s:

[0080]

[0081] Among them, Tr(Σ) is the sum of the diagonal elements of Σ, which represents the degree of stretching of the covariance matrix C of the subset vertex and the target vertex. is the overall square distance after the subset vertices are decentralized (that is, the energy of the point cloud). The translation vector t is used to eliminate the offset between the subset vertices and the target vertex in the overall position. Even after the rotation and scaling are completed, the center positions of the subset vertices and the target vertex may not be consistent, and need to be adjusted by translation:

[0082] t=mean(v full [corre[h v_subindex ]])-s·R·mean(h vsub ),

[0083] Among them, mean(v full [corre[h v_subindex ]]) is the centroid of the target vertex, that is, the center position of the target point set, s·R·mean(h v_sub ) is the center of mass of the subset vertices after scaling and rotation, that is, the center position after transformation. Align the center of the target vertex with the center of the subset vertex after transformation. This offset is the translation vector t. After the above calculations, we get the rotation matrix R, scaling factor s, and translation vector t. Next, apply these transformations to the subset vertices to get the final aligned vertex position. Transform through the following formula:

[0084] h′ vsub [i] = s·R·h v_sub [i]+t,

[0085] s.R.h v_sub [i] is to rotate and adjust the direction of a single subset of vertices, and adjust the size according to the scaling factor, and finally move the transformed vertex to the target position through the translation vector, h′ v_sub [i]Replace the corre[h_v_subindex] vertices of the whole body to complete the basic reconstruction of the whole body and face.

[0086] 6. To ensure the natural connection between the face and the whole body model, this application also uses the Laplace smoothing algorithm for the seam area and performs 2 to 3 smoothing operations. Through smoothing, the splicing transition becomes more natural, effectively avoiding the inconsistency that may be caused by vertex replacement and transformation, and improving the overall visual effect of the model. The smoothing algorithm used is as follows:

[0087]

[0088] v i are the vertices to be smoothed, Yes i The set of vertices in the neighborhood, is the number of vertices, v t represent One of the points in the neighborhood, traverse all the points in the neighborhood and sum them, and replace v with the average value of all the points in the neighborhood i , complete smoothing. The reconstructed full body model is as follows Figure 5 shown.

[0089] 7. As a preferred embodiment of the present invention, the application also adjusts the key vertices around the eyes and fine-tunes the eyeball position to ensure the natural integration of the facial model and the eyeball, and avoids the problem of misalignment or penetration of the eyeball during the previous transformation process. Through precise adjustment of the eyes, the model not only has a natural feel in static display, but also has higher visual expressiveness when dynamically driven, improving the usability of the model in various application scenarios.

[0090] Specifically, in order to achieve the fit between the key vertex of the orbit and the eyeball, the distance between the center of the eyeball and the key vertex of the orbit is first calculated, d i =||v i -c eye ||,

[0091] Among them, d i is the orbital key vertex v i To the center of the eye eye The distance is used to judge the fit between the vertex and the eyeball; v i is a key vertex on the eye socket, expressed as a 3D coordinate; c eye is the geometric center of the eyeball, also expressed as a 3D coordinate; when the vertex is too far from the eyeball surface or the vertex penetrates the eyeball surface, it is adjusted by the following formula:

[0092]

[0093] where r eye is the radius of the eyeball, indicating the size of the eyeball, v i ′ is the adjusted vertex position, ensuring that the vertex fits on the surface of the eyeball.

[0094] 8. Texture reconstruction: The uv coordinates and uv indexes of the head in the head model hm and the full-body model full_model are different, so the uv of full_model should be reconstructed according to the corresponding relationship between the two models. Specifically, first add the uv coordinates in hm to the uv coordinates in full_model, and then find the corresponding face in the full-body model for the uv indexes fta, ftb, ftc corresponding to each face fa, fb, fc of hm, and then replace the uv index corresponding to this face with m_full+fta, m_full+ftb, m_full+ftc, where m_full is the length of the original uv coordinate in the full-body model. In a 3D model, a face is a planar area formed by connecting several points. Usually, a face is represented by the index of the point. For example, 2, 6, 10 means that the second point, the sixth point and the tenth point are connected to form a triangular face. In this way, many faces are spliced ​​together to form the surface of the entire model. Among them, UV is a set of two-dimensional coordinates used to represent the correspondence between the surface of a 3D model and a 2D texture. Each point on the 3D model will correspond to a UV coordinate, indicating its position on the texture. The color information of a point on the 3D model will find the corresponding color from the texture through this UV coordinate, thereby adding texture or details to the model. In a 3D model, fa, fb, and fc represent the three vertices (point indices) connected by a triangular face, and fta, ftb, and ftc are the indices of the UV coordinates corresponding to these three vertices, which are used to specify the position of each vertex in the 2D texture, and are used to map the vertices on the 3D model to the 2D texture, thereby determining the texture color at the vertex.

[0095] 9. To avoid the difference in skin color between the face and body in the texture, a full-body texture f_pref is pre-made. After ht and f_pref are converted to the LAB color space, the brightness of the L channel is adjusted to make the brightness of f_pref close to ht. This method unifies the overall skin color of the model, achieves natural visual consistency, and enables the final model to maintain visual consistency in both static and dynamic displays. The schematic diagram of the full-body texture is shown below: Figure 6 shown.

[0096] 10. Prepare the reconstructed face and eyeball textures. Use the reconstructed full body UV coordinates and indices to convert and map ht to f_pref to generate a complete full body texture. The specific process is as follows. The head texture (ht) is a high-resolution texture specifically used to describe the details of the head. In order to integrate it into the full body texture (f_pref), the UV coordinates of the whole body need to be replanned so that the UV area of ​​the head is consistent with the structure of the full body texture. At the same time, the textures of the detail areas such as the eyeballs also need to be processed separately and included in the reconstructed UV design. In this way, exclusive texture areas can be reserved for the head and eyeballs in the full body UV space. Then, according to the reconstructed UV coordinates, the head texture content is mapped from the original coordinate system to the specified area of ​​the full body texture. In this process, the resolution, scale, and pixel details of the head model texture need to be adjusted to ensure consistency with the rest of the full body texture. After the mapping is completed, the prepared face texture, eyeball texture, and head model texture (ht) are integrated into the full body texture (f_pref). The final full-body texture not only contains the texture information of the head, eyeballs and body, but also visually ensures the consistency of details and the overall naturalness. This complete full-body texture can be directly applied to the full-body 3D model to provide high-quality texture support for rendering. The integrated texture covers the whole body and can naturally present every detail texture. Figure 7 shown.

[0097] 11. After the integration is completed, the 3D model data is exported to support obj, fbx, glb and other formats to ensure that the model can be used smoothly in different rendering engines (such as Unity, Unreal Engine). This support greatly expands the application scenarios of the model, from game development, film and television animation to virtual reality and augmented reality, which are all compatible to meet the diverse needs of users. Among them, the full-body model reconstructed in the fifth and sixth steps can be directly exported as an .obj file through the general library function, and the full-body map reconstructed in the ninth step can be saved in the .png file format. These two files correspond to the geometric structure and texture map of the model, respectively, and can be directly loaded and rendered by mainstream 3D software (such as Blender, Maya, 3ds Max and Unreal Engine, etc.). Among them, the display of the model in meshlab (3D geometry processing system) is as follows Figure 8 As shown. In this way, users can access the system through online services, whether it is a computer, mobile phone or any device with a camera, and can easily take photos and upload them. Users only need to select gender, upload photos and wait for reconstruction results, and quickly achieve high-quality three-dimensional full-body reconstruction. The online service model greatly improves the user experience, meets the needs of easy operation anytime and anywhere, and reduces the technical threshold for reconstruction.

[0098] Embodiment 2

[0099] Embodiment 2 of the present application provides a computer storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the whole body reconstruction method described in the above embodiment 1.

[0100] If the above functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0101] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A whole-body reconstruction method based on a single photo, characterized in that: The method comprises: Get the user's complete facial photo; Extracting the user's face image from the face photo and generating a head model hm, and fusing the head model hm with the texture map of the user's face to form a spliced ​​model map ht; Filter out all vertices h_v_sub in the facial area of ​​the head model hm, and obtain vertex indexes h_v_subindex of all vertices h_v_sub on the head model hm; Construct a full-body model full_model, set the mapping relationship array corre between the head model hm and the head vertices on the full-body model full_model, and ensure that the k-th point on the head model hm corresponds to the corre[k]-th point of the full-body model full_model; Construct a transformation equation to replace the vertices h_v_sub on the facial area to the corresponding positions of the full body model full_model; According to the facet correspondence between the head model hm and the full body model full_model, the uv coordinates of the full body model full_model are reconstructed to perform texture reconstruction; A full-body texture f_pref is prefabricated, and the model texture ht is converted and mapped to the prefabricated full-body texture f_pref through the uv coordinates and indexes of the reconstructed full-body model full_mode, so as to generate a complete full-body texture that can be directly applied to the 3D model.

2. The whole body reconstruction method according to claim 1, characterized in that: Extracting the user's face image from the face photo and generating a head model hm, and fusing the head model hm with the texture map of the user's face to form a spliced ​​model map ht includes: Get the user's face image; Generate a high-modulus mesh model M_h from the acquired face image based on the FFHQ-UV algorithm, where M_h=(V_h, F_h), V_h is a vertex set, and F_h is a face set; The generated high-poly mesh model M_h is merged with the lightweight low-poly mesh M_l to generate a lightweight low-poly mesh M_l'; After adjusting the eyeball position on the low-poly mesh M_l', the final head model hm is obtained; Detect multiple facial key points in the original face image and align them with multiple pre-labeled key points in the head model hm, project the head model hm into the two-dimensional face image, and extract the facial texture map T_face; The occluded area is excluded by the face segmentation algorithm to generate a mask area, and a gradient transition area image mask T_transition is generated at the edge of the mask area; Generate a skin color reference map C_ref, match the prefabricated map T_pre with the skin color reference map C_ref, and generate an adjusted map T_pre'; Using the generated mask T_transition containing the transition area, the texture map T_face of the user's face is weightedly fused with the adjusted pre-made map T_pre' to form a spliced ​​head model map ht.

3. The whole body reconstruction method according to claim 2, characterized in that: Adjust the key vertices around the eyes and fine-tune the position of the eyeball so that the adjusted vertices fit well on the surface of the eyeball.

4. The whole body reconstruction method according to any one of claims 1 to 3, characterized in that: Replacing the vertices h_v_sub on the facial area to the corresponding positions of the full-body model full_model includes: Construct the transformation equation and use singular value decomposition to calculate the rotation matrix R, translation vector t, and scaling parameter s from the vertex h_v_sub on the face to the corre[h_v_subindex] position of the whole body vertex. After h_v_sub is transformed, it replaces the corre[h_v_subindex] vertex of the whole body, so that the facial vertex is accurately replaced to the corresponding position of the whole body model. At the same time, the natural transition of the model is achieved through the following optimization equation: where h v_sub is a subset of facial vertices, corre[h v_subindex ] is the subset point in the whole body model v full In the corresponding index, i is the index of the subset point, from 0 to the number of subset vertices N-1, R is the rotation matrix, t is the translation vector t, and s is the scaling factor.

5. The whole body reconstruction method according to claim 4, characterized in that: The method further includes: using the Laplace smoothing algorithm to perform multiple smoothing operations on the seam area, wherein the smoothing formula is as follows: Among them, v i are the vertices to be smoothed, Yes i The set of vertices in the neighborhood, is the number of vertices, v t represent One of the points in the neighborhood, traverse all the points in the neighborhood and sum them, and replace v with the average of all the points in the neighborhood i , smoothing is completed.

6. The whole body reconstruction method according to claim 4, characterized in that: According to the facet correspondence between the head model hm and the full-body model full_model, the uv coordinates of the full-body model full_mode are reconstructed, and the texture reconstruction includes: First, the uv coordinates in the head model hm are appended below the uv coordinates in the full-body model full_model. Then, for each face fa, fb, fc of the head model hm, and based on the corresponding uv indexes fta, ftb, ftc, the corresponding face is found in the full-body model. Then, the uv index corresponding to this corresponding face is replaced with m_full+fta, m_full+ftb, m_full+ftc, where m_full is the length of the original uv coordinate in the full-body model.

7. The whole body reconstruction method according to claim 1 or 2, characterized in that: Pre-make a full-body texture f_pref, convert the model texture ht and the full-body texture f_pref to the LAB color space, and then adjust the brightness of the L channel to make the brightness of the full-body texture f_pref close to the brightness of the model texture ht.

8. The whole body reconstruction method according to claim 7, characterized in that: After the integration is completed, the 3D model data is exported to different formats, including obj format, fbx format, and glb format.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program is used to perform the whole body reconstruction method according to any one of claims 1 to 8.

Images