Human body three-dimensional reconstruction method, device, equipment and medium

Abstract

The invention relates to the field of three-dimensional reconstruction, and discloses a human body three-dimensional reconstruction method and device, equipment and a medium, and the method comprises the steps: obtaining a to-be-processed original human body image, carrying out the posture estimation of the original human body image, and generating a human body parameterized model; normal rendering processing is carried out on the human body parameterized model, and corresponding front and back body normal graphs are generated; performing clothes normal prediction according to the front and back body normal graphs in combination with the original human body image to obtain front and back clothes normal graphs; extracting body three-dimensional data from the front and back body normal graphs, and extracting clothes three-dimensional data from the front and back clothes normal graphs; and inputting the body three-dimensional data and the clothes three-dimensional data into a pre-trained multi-layer perceptron for three-dimensional reconstruction, and generating a three-dimensional digital person with clothes. On the basis of a human body parameterized model, the front and back body normal graphs are combined with an original human body image to carry out normal prediction on clothing details, a three-dimensional data person with the clothing details is reconstructed, and the reconstruction precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology and financial technology, and in particular to a method, device, equipment and medium for three-dimensional reconstruction of a human body. Background Art

[0002] Human 3D reconstruction technology is a technology that captures human posture and form and converts it into a 3D form and structure through a computer. With the rapid development of technologies such as the metaverse and virtual reality / augmented reality in recent years, human 3D reconstruction has broad application prospects in many fields. In the financial field, with the advancement of digital transformation, human 3D reconstruction technology can realize virtual customer service, marketing video virtual images, etc., to improve business processing efficiency.

[0003] Among them, pose estimation is an important part of human 3D reconstruction. However, in the existing reconstruction methods, many parametric models lack clothing details. Even if there are some methods to improve the problem of texture details, they also require expensive, time-consuming and labor-intensive scanning processes. In other words, it is difficult to accurately reconstruct human postures and texture details in complex postures, thereby reducing the accuracy of human 3D reconstruction. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a human body three-dimensional reconstruction method, device, equipment and medium that can be applied to financial technology or other related fields. Its main purpose is to improve the accuracy of human body three-dimensional reconstruction and realize human body three-dimensional reconstruction with texture details.

[0005] The technical solution of the present invention is as follows:

[0006] A first aspect of the present invention provides a method for three-dimensional reconstruction of a human body, comprising:

[0007] Acquire an original human body image to be processed, perform posture estimation on the original human body image, and generate a human body parameterized model;

[0008] Performing normal rendering processing on the human body parameterized model to generate corresponding front and back body normal images;

[0009] According to the front and back body normal maps, clothing normal prediction is performed in combination with the original human body image to obtain the front and back clothing normal maps;

[0010] Extracting body three-dimensional data from the front and back body normal images, and extracting clothing three-dimensional data from the front and back clothing normal images;

[0011] The three-dimensional body data and the three-dimensional clothing data are input into a pre-trained multi-layer perceptron for three-dimensional reconstruction to generate a three-dimensional digital human wearing clothing.

[0012] A second aspect of the present invention provides a human body three-dimensional reconstruction device, comprising:

[0013] A model generation module is used to obtain an original human body image to be processed, perform posture estimation on the original human body image, and generate a human body parameterized model;

[0014] A body normal rendering module, used for performing normal rendering processing on the human body parameterized model to generate corresponding front and back body normal images;

[0015] A clothing normal prediction module, used to perform clothing normal prediction based on the front and back body normal maps in combination with the original human body image to obtain the front and back clothing normal maps;

[0016] A three-dimensional data extraction module, used to extract body three-dimensional data from the front and back body normal images, and to extract clothing three-dimensional data from the front and back clothing normal images;

[0017] The three-dimensional reconstruction module is used to input the three-dimensional data of the body and the three-dimensional data of the clothing into a pre-trained multi-layer perceptron for three-dimensional reconstruction to generate a three-dimensional digital human with clothing.

[0018] A third aspect of the present invention provides a computer device, comprising at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned three-dimensional reconstruction method of the human body.

[0021] A fourth aspect of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned human body three-dimensional reconstruction method.

[0022] Beneficial effects: The present invention discloses a method, device, equipment and medium for three-dimensional reconstruction of a human body. Compared with the prior art, the embodiment of the present invention obtains an original human body image to be processed, performs posture estimation on the original human body image, and generates a human body parameterized model; performs normal rendering processing on the human body parameterized model to generate corresponding front and back body normal maps; performs clothing normal prediction based on the front and back body normal maps and in combination with the original human body image to obtain front and back clothing normal maps; extracts body three-dimensional data from the front and back body normal maps, and extracts clothing three-dimensional data from the front and back clothing normal maps; inputs the body three-dimensional data and clothing three-dimensional data into a pre-trained multi-layer perceptron for three-dimensional reconstruction, and generates a three-dimensional digital human with clothing. By further combining the front and back body normal maps with the original human body image on the basis of a rough human body parameterized model to perform normal prediction on clothing texture details, a three-dimensional data human with clothing details can be reconstructed, thereby improving the accuracy of three-dimensional reconstruction of the human body and realizing three-dimensional reconstruction of the human body with texture details. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the scheme in the present invention, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 A schematic diagram of an application environment of the human body three-dimensional reconstruction method provided by an embodiment of the present invention;

[0025] Figure 2 A flow chart of a method for three-dimensional reconstruction of a human body provided by an embodiment of the present invention;

[0026] Figure 3 A flow chart of step S202 in the method for three-dimensional reconstruction of a human body provided in an embodiment of the present invention;

[0027] Figure 4 A flow chart of step S203 in the method for three-dimensional reconstruction of a human body provided in an embodiment of the present invention;

[0028] Figure 5 A flow chart of step S204 in the method for three-dimensional reconstruction of a human body provided by an embodiment of the present invention;

[0029] Figure 6 A flow chart of step S205 in the method for three-dimensional reconstruction of a human body provided by an embodiment of the present invention;

[0030] Figure 7 A schematic diagram of the functional modules of a human body 3D reconstruction device provided by an embodiment of the present invention;

[0031] Figure 8 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0033] The human body three-dimensional reconstruction method provided by the embodiment of the present invention can be applied to Figure 1 In the application environment of FIG. 1 , it includes a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0034] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0035] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0036] The server 105 may be a server that provides various services, such as a background server that provides support for the content browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (for example only). The background server may analyze and process the received data such as the user request, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device. The server 105 may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server 105 may also be a server for a distributed system, or a server combined with a blockchain.

[0037] It should be noted that the human body 3D reconstruction method provided in the embodiment of the present application can generally be executed by the first terminal device 101, the second terminal device 102 or the third terminal device 103. Accordingly, the human body 3D reconstruction device provided in the embodiment of the present invention can also be set in the first terminal device 101, the second terminal device 102 or the third terminal device 103. Alternatively, the human body 3D reconstruction method provided in the embodiment of the present invention can also generally be executed by the server 105. Accordingly, the human body 3D reconstruction device provided in the embodiment of the present invention can generally be set in the server 105.

[0038] It should be understood that the numbers of the above terminal devices, networks and servers are only illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0039] like Figure 2 As shown, the human body 3D reconstruction method provided by the embodiment of the present invention specifically includes the following steps:

[0040] S201, obtaining an original human body image to be processed, performing posture estimation on the original human body image, and generating a human body parameterized model.

[0041] In this embodiment, the original human image that needs to be three-dimensionally reconstructed can be a static photo or a video frame extracted from a video stream, etc. The original human image contains the posture, shape and appearance information of the human body, such as a standing human photo, a side view of a running human body, etc. The original human image to be processed is obtained as the basis for reconstructing a three-dimensional human body model. By performing posture estimation on this image, the posture of the human body in the image is analyzed by algorithms such as SPIN or VIBE algorithms, and a corresponding human body parameterized model, namely the SMPL model, is generated. The SMPL model is a parameterized human body model that can describe the posture and shape of the human body. It consists of a set of bones and skin. Different human body postures are simulated by adjusting the position and rotation of the bones, so that the human body geometry can be described with fewer parameters. By extracting the posture information of the human body from a single image, and generating a human body parameterized model that accurately matches the posture of the original image, the human body parameterized model is a rough human body geometry, which provides a basis for the subsequent reconstruction of texture details such as clothing folds.

[0042] S202: Perform normal rendering processing on the human body parameterized model to generate corresponding front and back body normal images.

[0043] In this embodiment, based on the generated human body parametric model, normal rendering processing is performed on it, that is, two two-dimensional images are generated according to the normal direction of each vertex on the surface of the human body parametric model, one is a normal map showing the front of the human body, and the other is a normal map showing the back of the human body. The normal of each vertex on the surface of the human body parametric model is a vector perpendicular to the surface in three-dimensional space. These vectors define the geometric details and shape of the human body surface. Therefore, the two front and back body normal maps generated by rendering contain the shape information of the human body surface, which serves as a geometric space constraint for subsequent clothing reconstruction and texture detail prediction, thereby realizing clothing detail prediction driven by human body posture.

[0044] S203, predicting clothing normals according to the front and back body normal maps in combination with the original human body image, to obtain front and back clothing normal maps.

[0045] In this embodiment, the front and back body normal maps generated based on rendering are combined with the original human body image, and clothing normal prediction is performed through the front and back body normal maps and the clothing, color information and body surface normal information provided by the original human body image. By analyzing factors such as wrinkles and deformation of clothing in different human body postures, corresponding front and back clothing normal maps are generated, thereby providing geometric details of the clothing surface. In general, the clothing shape and texture details are provided for the three-dimensional reconstruction of the human body without the need for an additional scanning process, thereby improving the reconstruction accuracy.

[0046] S204, extracting body three-dimensional data from the front and back body normal images, and extracting clothing three-dimensional data from the front and back clothing normal images.

[0047] In this embodiment, the three-dimensional data of the body surface is extracted from the generated front and back body normal images, which includes the shape and structure information of the body. Similarly, the three-dimensional data of the clothes is extracted from the generated front and back clothes normal images. By extracting the three-dimensional data, the accurate geometric information of the body and clothes is obtained, which provides accurate and necessary data for the final three-dimensional reconstruction.

[0048] S205: Input the three-dimensional body data and the three-dimensional clothing data into a pre-trained multi-layer perceptron for three-dimensional reconstruction to generate a three-dimensional digital human wearing clothing.

[0049] In this embodiment, the extracted three-dimensional body data and three-dimensional clothing data are combined and input into a pre-trained multi-layer perceptron (MLP) network. The multi-layer perceptron is a feedforward neural network composed of multiple layers, including an input layer, a hidden layer, and an output layer. Each layer is composed of multiple neurons, which are connected by weights, and nonlinearity is introduced by activation functions, so that the multi-layer perceptron can learn complex function mappings. In this embodiment, the input of the multi-layer perceptron is two three-dimensional data sets, namely, three-dimensional body data and three-dimensional clothing data. These data sets contain geometric details of the human body surface and wrinkle information of clothing. The trained multi-layer perceptron can learn the features of these data and output a complete three-dimensional digital human model, thereby converting the input three-dimensional body and clothing data into geometric details in three-dimensional space such as three-dimensional grids or three-dimensional point clouds, and reconstructing a three-dimensional digital human with rich geometric details.

[0050] For example, in the reconstruction scenario of the financial field, when it is necessary to create a virtual bank manager image for a virtual human marketing video, a frontal photo of the bank manager can be selected from the database, and the posture of the photo can be estimated by using the SPIN or VIBE algorithm, the posture of the bank manager is analyzed and the SMPL parameters are generated. These parameters define the bone structure and shape of the human body model, thereby generating a parameterized human body model. This model will be used in the subsequent reconstruction process to ensure that the image of the virtual bank manager is consistent with the posture of the real person, increasing the realism of the video; then the normal direction of each point on the model surface is calculated, and a display is generated. The bank manager's front body normal map and a body normal map showing the back of the bank manager contain the shape information of the bank manager's body surface and provide the geometric details of the body surface. Then, the three-dimensional data of the bank manager's body is extracted from the generated body normal map, and the three-dimensional data of the clothes is extracted from the clothes normal map to obtain the precise geometric information of the bank manager's body and clothes, ensuring the accuracy and authenticity of the virtual image. Then, the extracted three-dimensional data of the bank manager's body and clothes are input into the pre-trained multi-layer perceptron network to learn the features of these data and reconstruct a three-dimensional digital human model with clothing details. In the marketing videos in the financial field, the generated three-dimensional digital human model is not only more accurate in posture, but also more realistic in texture details. It can be used in various applications, such as virtual fitting, animation production, etc., making the marketing video more attractive, improving the audience's participation and the dissemination effect of the video.

[0051] In the above embodiment, the present invention discloses a method for three-dimensional reconstruction of a human body, which obtains an original human body image to be processed, estimates the posture of the original human body image, and generates a human body parameterized model; performs normal rendering processing on the human body parameterized model to generate corresponding front and back body normal maps; performs clothing normal prediction based on the front and back body normal maps and in combination with the original human body image to obtain front and back clothing normal maps; extracts body three-dimensional data from the front and back body normal maps, and extracts clothing three-dimensional data from the front and back clothing normal maps; inputs the body three-dimensional data and clothing three-dimensional data into a pre-trained multi-layer perceptron for three-dimensional reconstruction, and generates a three-dimensional digital human with clothing. By further combining the front and back body normal maps with the original human body image to perform normal prediction on clothing texture details on the basis of a rough human body parameterized model, a three-dimensional data human with clothing details can be reconstructed, thereby improving the accuracy of three-dimensional reconstruction of the human body and realizing three-dimensional reconstruction of the human body with texture details.

[0052] In one embodiment, the obtaining of the original human body image to be processed includes:

[0053] Receiving a video stream to be processed, parsing and obtaining all video frames in the video stream to be processed;

[0054] Human target detection is performed on all the video frames, and key video frames containing human bodies are extracted from all the video frames according to the detection results as original human body images to be processed.

[0055] In this embodiment, the original human body image to be processed is obtained by processing and extracting a video stream. For example, a financial marketing video stream is obtained, which contains multiple shots of a bank manager in different occasions and postures. By receiving this video stream and parsing it into a series of video frames, such as a 30-second video may contain 900 frames, etc., the continuous video stream is decomposed into discrete frames, so that each frame can be analyzed in detail, ensuring that the human target can be accurately detected and extracted in subsequent steps.

[0056] Based on all the extracted video frames, human target detection is performed on all the parsed video frames through target detection algorithms such as YOLO (You Only Look Once) or SSD (Single Shot MultiBox Detector), and the position and outline of the human body in each frame are identified. For example, in a video of a bank manager introducing financial products, the presence of the bank manager's body in each frame is detected and the position of the body is marked. According to the human target detection results, key video frames containing human targets are selected from all video frames as the original human images to be processed, that is, the key video frames containing human targets are subsequently reconstructed in three dimensions, so that the human body in a video stream can be vividly and accurately reconstructed into the corresponding three-dimensional digital human, realizing efficient virtual image video production.

[0057] In one embodiment, the step of performing posture estimation on the original human body image to generate a human body parameterized model includes:

[0058] Performing posture estimation and shape parameter estimation on the original human body image respectively to obtain posture parameters and shape parameters in the original human body image;

[0059] Creating an initial parameterized model, configuring a bone structure in the initial parameterized model according to the posture parameters, and configuring vertex positions of the bone structure according to the shape parameters;

[0060] The configured initial parameterized model is skinned to generate a corresponding human body parameterized model.

[0061] In this embodiment, when generating a human body parametric model based on an original human body image, posture estimation and shape parameter estimation are first performed on the original human body image. Specifically, the original human body image can be processed by a SPIN (Spatial Pose and Shape Inference Network) algorithm to estimate the posture parameters (such as joint angles) and shape parameters (such as proportions of various body parts) of the human body from a single image. For example, for a key frame extracted from a financial marketing video, it is identified that the bank manager's arm is raised, the head is tilted, and the specific size of the body is identified, and accurate posture and shape information is extracted from a single image.

[0062] Then, an initial SMPL (Skinned Multi-Person Linear) model, i.e., an initial parameter model, is created, which includes a skeleton structure and a set of vertices. The joint angles in the skeleton structure are adjusted according to the estimated posture parameters to make the skeleton of the model match the posture of the bank manager; and the positions of the vertices of the skeleton structure are adjusted according to the shape parameters to match the body shape of the bank manager, thus generating an initial human model with the correct posture and shape. Through further skinning operations, the skin texture and details are mapped to the surface of the SMPL model, thereby converting the configured initial parameterized model into a complete human model, and obtaining a parameterized model without clothing, which serves as the basis for subsequent accurate human 3D reconstruction.

[0063] In one embodiment, Figure 3 As shown, step S202 includes:

[0064] S301, calculating the body normal vector of each vertex in the human body parameterized model in the current posture;

[0065] S302, encoding the body normal vector of each vertex into a corresponding pixel value according to a preset rule and mapping it to a corresponding pixel position, and rendering to generate a frontal body normal map;

[0066] S303, negate the body normal vector of each vertex, encode the negated body normal vector into a corresponding pixel value according to a preset rule, map it to a corresponding pixel position, and render to generate a reverse body normal map.

[0067] In this embodiment, when generating the front and back body normal maps, based on the human body parameterized model generated by the configuration, for each vertex in the model, its body normal vector in the current posture is calculated, and the body normal vector is perpendicular to the surface at the vertex and points to the outside. After obtaining the body normal vector of each vertex, the body normal vector is encoded into a pixel value, thereby mapping from the three-dimensional space to the two-dimensional image space. For example, the three components (x, y, z) of the body normal vector are mapped to the RGB color space, where the range of each component [-1, 1] is mapped to [0, 255], and then the pixel value of each vertex is mapped to the corresponding pixel position to generate the front body normal map. Similarly, after the front body normal map is generated, the body normal vector of each vertex is negated to obtain the body normal vector of the reverse side. For example, if the normal vector of a vertex is (x, y, z), then the inverted vector is (-x, -y, -z). According to the same rules as above, the inverted body normal vector is encoded into a pixel value and mapped to the corresponding pixel position to generate a reverse body normal map. By generating the front body normal map and the reverse body normal map, the pixel value of each pixel represents the normal direction of the corresponding vertex, thereby obtaining detailed normal information on the human body surface, providing important surface geometry information for subsequent 3D human body reconstruction, and ensuring the accuracy of 3D reconstruction.

[0068] In one embodiment, Figure 4 As shown, step S203 includes:

[0069] S401, performing image fusion processing on the front body normal image and the original human body image to obtain a front fused image;

[0070] S402, performing image fusion processing on the back body normal image and the original human body image to obtain a back fused image;

[0071] S403: input the front fusion image and the back fusion image into a pre-trained normal prediction network respectively, and predict and output corresponding front clothing normal map and back clothing normal map.

[0072] In this embodiment, after the front body normal map and the back body normal map are rendered and generated, they are merged with the original human body image to predict the clothing normal map. The front body normal map and the original human body image are subjected to image fusion processing to obtain a front fused image. The specific fusion method can be simple superposition, mixing or other image fusion techniques (such as Poisson fusion or Laplace pyramid fusion, etc.), so as to add the geometric details of the front body normal map on the basis of retaining the color information of the original image; similarly, the back body normal map and the original human body image are subjected to image fusion processing, and the geometric details of the back body normal map are added on the basis of retaining the color information of the original image.

[0073] Through image fusion processing, the geometric details of the clothes are enhanced while maintaining the real appearance of the clothes, providing richer information for the subsequent clothing normal prediction. The fused front fusion image and the back fusion image are respectively input into the pre-trained normal prediction network. Specifically, the normal prediction network can use the high-definition version of PIFu (Pixel-Aligned Implicit Function), namely PIFuHD, which is a deep learning-based framework that uses deep neural networks to learn the mapping from two-dimensional images to three-dimensional human models, so as to capture the complex geometric structure of the human body and the appearance of clothes. PIFuHD can predict the normal map of the human body surface, which contains clothing wrinkles and other detailed information, so as to predict and output the corresponding front clothing normal map and back clothing normal map based on the input front fusion image and back fusion image, so as to create a three-dimensional digital human with realistic clothing details using the front and back clothing normal maps, including clothing wrinkles, textures and shapes, making the virtual image more realistic and vivid.

[0074] In one embodiment, Figure 5 As shown, step S204 includes:

[0075] S501, respectively acquiring depth information of the front and back body normal images and the front and back clothing normal images to obtain first depth values ​​of body pixel points in the front and back body normal images and second depth values ​​of clothing pixel points in the front and back clothing normal images;

[0076] S502, determining the three-dimensional coordinates of the body pixel point according to the first depth value, and converting the two-dimensional body normal vector in the front and back body normal images into a three-dimensional body normal vector;

[0077] S503, associating the three-dimensional coordinates of the body pixel points with the three-dimensional body normal vector to generate three-dimensional body data;

[0078] S504, determining the three-dimensional coordinates of the clothing pixel point according to the second depth value, and converting the two-dimensional clothing normal vector in the front and back clothing normal image into a three-dimensional clothing normal vector;

[0079] S505: Associating the three-dimensional coordinates of the clothing pixel points with the three-dimensional clothing normal vector to generate three-dimensional clothing data.

[0080] In this embodiment, when extracting three-dimensional data of the front and back body normal maps and the front and back clothing normal maps, depth information is first acquired for the two normal maps respectively. When acquiring the depth information of the front and back body normal maps, since the human body parametric model provides a parametric representation of human body posture and shape, which includes depth information, the depth information of the vertex corresponding to each body pixel point can be directly acquired from the model as the first depth value of the body pixel point in the front and back body normal maps; when acquiring the depth information of the front and back clothing normal maps, the second depth value of each clothing pixel point in the front and back clothing normal maps can be predicted by a depth estimation algorithm, such as a structured light method, a time-of-flight method, or a deep learning network.

[0081] Using the acquired depth information, when extracting the three-dimensional body data, according to the first depth value of each body pixel point, the two-dimensional coordinates (x, y) of the body pixel point are converted into points (X, Y, Z) in the three-dimensional space to obtain the three-dimensional coordinates of the body pixel point. In addition, the first depth value is used to convert the two-dimensional body normal vector in the front and back body normal map into a three-dimensional body normal vector. For example, if the two-dimensional body normal vector is (u, v) and the depth value is d, the converted three-dimensional body normal vector can be expressed as (u·d, v·d, d), and the three-dimensional coordinates of each body pixel point are associated with its corresponding three-dimensional body normal vector to generate a complete three-dimensional body data set, thereby obtaining a three-dimensional body data set, and each point in the three-dimensional body data set contains position information and two normal vectors (front and back). Similarly, when extracting the three-dimensional clothing data, according to the second depth value of each clothing pixel point, the above three-dimensional coordinate determination and normal vector conversion are also performed, and the complete three-dimensional clothing data set is associated to obtain a three-dimensional clothing data set, and each point in the three-dimensional clothing data set contains position information and two normal vectors (front and back). By extracting three-dimensional information from the two-dimensional normal image, three-dimensional data of the body and clothing is generated, which provides accurate and complete data for the reconstruction of three-dimensional digital humans. When performing three-dimensional reconstruction of the human body, not only can an accurate body model be created, but also surface details such as clothing shape and wrinkles can be reconstructed, thus providing high-quality reconstruction content.

[0082] In one embodiment, Figure 6 As shown, step S205 includes:

[0083] S601, performing feature fusion on the three-dimensional body data and the three-dimensional clothing data to generate a corresponding fusion feature vector;

[0084] S602, inputting the fused feature vector into a pre-trained multi-layer perceptron, forward-propagating the fused feature vector through a multi-layer structure, and predicting and outputting three-dimensional reconstruction features including the body and clothing;

[0085] S603: Decode and convert the three-dimensional reconstruction features to generate corresponding three-dimensional point clouds or meshes, and reconstruct a three-dimensional digital human wearing clothes according to the three-dimensional point clouds or meshes.

[0086] In this embodiment, after obtaining two data sets of body 3D data and clothing 3D data, the two sets of data are fused into a fused feature vector by means of feature fusion such as splicing or weighted fusion. For example, the 3D coordinates and normal vectors of each body pixel point can be spliced ​​together with the 3D coordinates and normal vectors of the corresponding clothing pixel points to form a fused feature vector containing body and clothing information, thereby generating a feature vector that fuses body and clothing information, and this fused feature vector contains all the geometric and surface information required to reconstruct a 3D digital human.

[0087] The generated fusion feature vector is input into a pre-trained multi-layer perceptron. The multi-layer perceptron is pre-trained to learn and predict the 3D reconstruction features from the fusion feature vector. For example, the multi-layer perceptron contains multiple hidden layers, each of which performs nonlinear transformation on the input data through an activation function, and finally outputs the 3D reconstruction features of the body and clothing. The learning ability of the multi-layer perceptron is used to extract key 3D reconstruction features from the fusion feature vector. These features capture the geometric structure and surface details of the body and clothing, provide key information for generating realistic 3D digital humans, and ensure the accuracy and reliability of reconstruction. Finally, the 3D reconstruction features output by the multi-layer perceptron are decoded and converted back to point clouds or meshes in 3D space. According to the 3D point clouds or meshes, an accurate, detailed and realistic 3D digital human model is output. The model integrates the geometric information of the human body and clothing, so that the generated 3D digital human model can be more accurate in posture level and texture details, thereby improving the accuracy of 3D reconstruction of the human body and realizing 3D reconstruction of the human body with texture details.

[0088] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0089] Further references Figure 7 , as a response to the above Figure 2 The present invention provides an embodiment of a human body three-dimensional reconstruction device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0090] like Figure 7As shown, the human body 3D reconstruction device 70 described in this embodiment includes:

[0091] The model generation module 701 is used to obtain an original human body image to be processed, perform posture estimation on the original human body image, and generate a human body parameterized model;

[0092] A body normal rendering module 702, used for performing normal rendering processing on the human body parameterized model to generate corresponding front and back body normal images;

[0093] A clothing normal prediction module 703 is used to perform clothing normal prediction based on the front and back body normal maps in combination with the original human body image to obtain the front and back clothing normal maps;

[0094] A three-dimensional data extraction module 704, for extracting body three-dimensional data from the front and back body normal images, and extracting clothing three-dimensional data from the front and back clothing normal images;

[0095] The three-dimensional reconstruction module 705 is used to input the three-dimensional body data and the three-dimensional clothing data into a pre-trained multi-layer perceptron for three-dimensional reconstruction to generate a three-dimensional digital human with clothing.

[0096] The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, which are more suitable for describing the execution process of human body three-dimensional reconstruction than programs. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0097] In one embodiment, the model generation module 701 includes:

[0098] A receiving unit, configured to receive a video stream to be processed, and parse and obtain all video frames in the video stream to be processed;

[0099] The detection and extraction unit is used to perform human target detection on all the video frames, and extract key video frames containing human bodies from all the video frames according to the detection results as original human body images to be processed.

[0100] In one embodiment, the model generation module 701 further includes:

[0101] an estimating unit, configured to perform posture estimation and shape parameter estimation on the original human body image, respectively, to obtain posture parameters and shape parameters in the original human body image;

[0102] A creation configuration unit is used to create an initial parameterized model, configure a bone structure in the initial parameterized model according to the posture parameters, and configure vertex positions of the bone structure according to the shape parameters;

[0103] The model generation unit is used to perform skinning operations on the configured initial parameterized model to generate a corresponding human body parameterized model.

[0104] In one embodiment, the body normal rendering module 702 includes:

[0105] A normal calculation unit, used to calculate the body normal vector of each vertex in the human body parameterized model in the current posture;

[0106] A front rendering unit, used to encode the body normal vector of each vertex into a corresponding pixel value according to a preset rule and map it to a corresponding pixel position, and render and generate a front body normal map;

[0107] The reverse side rendering unit is used to invert the body normal vector of each vertex, encode the inverted body normal vector into a corresponding pixel value according to a preset rule and map it to a corresponding pixel position, and render and generate a reverse side body normal map.

[0108] In one embodiment, the clothing normal prediction module 703 includes:

[0109] A first fusion unit, configured to perform image fusion processing on the front body normal image and the original human body image to obtain a front fused image;

[0110] A second fusion unit is used to perform image fusion processing on the back body normal image and the original human body image to obtain a back fused image;

[0111] The normal prediction unit is used to input the front fusion image and the back fusion image into a pre-trained normal prediction network respectively, and predict and output the corresponding front clothing normal map and back clothing normal map.

[0112] In one embodiment, the three-dimensional data extraction module 704 includes:

[0113] a depth information acquisition unit, configured to acquire depth information of the front and back body normal images and the front and back clothing normal images, respectively, to obtain first depth values ​​of body pixels in the front and back body normal images and second depth values ​​of clothing pixels in the front and back clothing normal images;

[0114] a first conversion unit, configured to determine the three-dimensional coordinates of the body pixel point according to the first depth value, and convert the two-dimensional body normal vector in the front and back body normal images into a three-dimensional body normal vector;

[0115] A first associating unit, configured to associate the three-dimensional coordinates of the body pixel points with the three-dimensional body normal vector to generate three-dimensional body data;

[0116] a second conversion unit, configured to determine the three-dimensional coordinates of the clothing pixel point according to the second depth value, and convert the two-dimensional clothing normal vector in the front and back clothing normal image into a three-dimensional clothing normal vector;

[0117] The second associating unit is used to associate the three-dimensional coordinates of the clothing pixel points with the three-dimensional clothing normal vector to generate three-dimensional clothing data.

[0118] In one embodiment, the 3D reconstruction module 705 includes:

[0119] A feature fusion unit, used for performing feature fusion on the three-dimensional body data and the three-dimensional clothing data to generate a corresponding fusion feature vector;

[0120] A prediction output unit, used to input the fused feature vector into a pre-trained multi-layer perceptron, forward propagate the fused feature vector through a multi-layer structure, and predict and output three-dimensional reconstruction features including the body and clothing;

[0121] The decoding and reconstruction unit is used to decode and convert the three-dimensional reconstruction features to generate a corresponding three-dimensional point cloud or mesh, and reconstruct a three-dimensional digital human with clothing according to the three-dimensional point cloud or mesh.

[0122] In the above embodiment, the present invention discloses a human body three-dimensional reconstruction device, which obtains the original human body image to be processed, estimates the posture of the original human body image, and generates a human body parameterized model; performs normal rendering processing on the human body parameterized model to generate corresponding front and back body normal maps; performs clothing normal prediction based on the front and back body normal maps and in combination with the original human body image to obtain the front and back clothing normal maps; extracts body three-dimensional data from the front and back body normal maps, and extracts clothing three-dimensional data from the front and back clothing normal maps; inputs the body three-dimensional data and clothing three-dimensional data into a pre-trained multi-layer perceptron for three-dimensional reconstruction, and generates a three-dimensional digital human with clothing. By further combining the front and back body normal maps with the original human body image to perform normal prediction on clothing texture details on the basis of a rough human body parameterized model, a three-dimensional data person with clothing details can be reconstructed, thereby improving the accuracy of three-dimensional reconstruction of the human body and realizing three-dimensional reconstruction of the human body with texture details.

[0123] Another embodiment of the present invention provides a computer device, such as Figure 8 As shown, the computer device 80 includes:

[0124] One or more processors 801 and memory 802, Figure 8 A processor 801 is used as an example for the description. The processor 801 and the memory 802 may be connected via a bus or other means. Figure 8The example of connecting through bus is taken in the following.

[0125] Processor 801 is used to complete various control logics of computer device 80, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component or any combination of these components. In addition, processor 801 can also be any traditional processor, microprocessor or state machine. Processor 801 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0126] The memory 802 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the human body three-dimensional reconstruction method in the embodiment of the present invention. The processor 801 executes various functional applications and data processing of the computer device 80 by running the non-volatile software programs, instructions and units stored in the memory 802, that is, the human body three-dimensional reconstruction method in the above method embodiment is implemented.

[0127] The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device 80, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 802 may optionally include a memory remotely arranged relative to the processor 801, and these remote memories may be connected to the computer device 80 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. One or more units are stored in the memory 802, and when executed by one or more processors 801, the steps of the human body three-dimensional reconstruction method in any of the above-mentioned method embodiments are executed.

[0128] In the above embodiment, the present invention discloses a computer device, which obtains the original human body image to be processed, estimates the posture of the original human body image, and generates a human body parameterized model; performs normal rendering processing on the human body parameterized model to generate corresponding front and back body normal maps; performs clothing normal prediction based on the front and back body normal maps and in combination with the original human body image to obtain the front and back clothing normal maps; extracts body three-dimensional data from the front and back body normal maps, and extracts clothing three-dimensional data from the front and back clothing normal maps; inputs the body three-dimensional data and clothing three-dimensional data into a pre-trained multi-layer perceptron for three-dimensional reconstruction, and generates a three-dimensional digital human with clothing. By further combining the front and back body normal maps with the original human body image to perform normal prediction on clothing texture details on the basis of a rough human body parameterized model, a three-dimensional data person with clothing details can be reconstructed, thereby improving the accuracy of three-dimensional reconstruction of the human body and realizing three-dimensional reconstruction of the human body with texture details.

[0129] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the steps of the human body three-dimensional reconstruction method in any of the above method embodiments are executed.

[0130] In the above embodiment, the present invention discloses a non-volatile computer-readable storage medium, which obtains an original human body image to be processed, estimates the posture of the original human body image, and generates a human body parameterized model; performs normal rendering processing on the human body parameterized model to generate corresponding front and back body normal maps; performs clothing normal prediction based on the front and back body normal maps and in combination with the original human body image to obtain front and back clothing normal maps; extracts body three-dimensional data from the front and back body normal maps, and extracts clothing three-dimensional data from the front and back clothing normal maps; inputs the body three-dimensional data and clothing three-dimensional data into a pre-trained multi-layer perceptron for three-dimensional reconstruction, and generates a three-dimensional digital human with clothing. By further combining the front and back body normal maps with the original human body image on the basis of a rough human body parameterized model to perform normal prediction on clothing texture details, a three-dimensional data human with clothing details can be reconstructed, thereby improving the accuracy of three-dimensional reconstruction of the human body and realizing three-dimensional reconstruction of the human body with texture details.

[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0132] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0133] In summary, the present invention discloses a method, device, equipment and medium for three-dimensional reconstruction of a human body, wherein the method includes: obtaining an original human body image to be processed, performing posture estimation on the original human body image, and generating a human body parameterized model; performing normal rendering processing on the human body parameterized model, and generating corresponding front and back body normal maps; performing clothing normal prediction based on the front and back body normal maps and in combination with the original human body image, and obtaining front and back clothing normal maps; extracting body three-dimensional data from the front and back body normal maps, and extracting clothing three-dimensional data from the front and back clothing normal maps; inputting the body three-dimensional data and clothing three-dimensional data into a pre-trained multi-layer perceptron for three-dimensional reconstruction, and generating a three-dimensional digital human with clothing. By further combining the front and back body normal maps with the original human body image on the basis of a rough human body parameterized model to perform normal prediction on clothing texture details, a three-dimensional data human with clothing details can be reconstructed, thereby improving the accuracy of three-dimensional reconstruction of the human body, and realizing three-dimensional reconstruction of the human body with texture details.

[0134] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0135] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of the present application, they are merely used for illustration and do not represent actual use. It should be understood that the application of the present invention is not limited to the above examples, and that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes shall fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for three-dimensional reconstruction of a human body, characterized in that: include: Acquire an original human body image to be processed, perform posture estimation on the original human body image, and generate a human body parameterized model; Performing normal rendering processing on the human body parameterized model to generate corresponding front and back body normal images; According to the front and back body normal maps, clothing normal prediction is performed in combination with the original human body image to obtain the front and back clothing normal maps; Extracting body three-dimensional data from the front and back body normal images, and extracting clothing three-dimensional data from the front and back clothing normal images; The three-dimensional body data and the three-dimensional clothing data are input into a pre-trained multi-layer perceptron for three-dimensional reconstruction to generate a three-dimensional digital human wearing clothing.

2. The method for three-dimensional reconstruction of a human body according to claim 1, characterized in that: The step of obtaining the original human body image to be processed comprises: Receiving a video stream to be processed, parsing and obtaining all video frames in the video stream to be processed; Human target detection is performed on all the video frames, and key video frames containing human bodies are extracted from all the video frames according to the detection results as original human body images to be processed.

3. The method for three-dimensional reconstruction of a human body according to claim 1, characterized in that: The step of performing posture estimation on the original human body image to generate a human body parameterized model comprises: Performing posture estimation and shape parameter estimation on the original human body image respectively to obtain posture parameters and shape parameters in the original human body image; Creating an initial parameterized model, configuring a bone structure in the initial parameterized model according to the posture parameters, and configuring vertex positions of the bone structure according to the shape parameters; The configured initial parameterized model is skinned to generate a corresponding human body parameterized model.

4. The method for three-dimensional reconstruction of a human body according to claim 1, characterized in that: The normal rendering process is performed on the human body parameterized model to generate corresponding front and back body normal images, including: Calculating a body normal vector of each vertex in the human body parameterized model in a current posture; Encoding the body normal vector of each vertex into a corresponding pixel value according to a preset rule and mapping it to a corresponding pixel position, and rendering to generate a frontal body normal map; The body normal vector of each vertex is inverted, and the inverted body normal vector is encoded into a corresponding pixel value according to a preset rule and mapped to a corresponding pixel position, and a reverse body normal map is rendered and generated.

5. The method for three-dimensional reconstruction of a human body according to claim 4, characterized in that: The method of predicting clothing normals based on the front and back body normal maps and combining the original human body image to obtain the front and back clothing normal maps includes: Performing image fusion processing on the front body normal image and the original human body image to obtain a front fused image; Performing image fusion processing on the reverse body normal image and the original human body image to obtain a reverse fused image; The front fusion image and the back fusion image are respectively input into a pre-trained normal prediction network, and the corresponding front clothing normal map and back clothing normal map are predicted and output.

6. The method for three-dimensional reconstruction of a human body according to claim 1, characterized in that: The step of extracting the body three-dimensional data from the front and back body normal images, and extracting the clothing three-dimensional data from the front and back clothing normal images, comprises: Acquire depth information of the front and back body normal images and the front and back clothing normal images respectively, to obtain first depth values ​​of body pixel points in the front and back body normal images and second depth values ​​of clothing pixel points in the front and back clothing normal images; Determine the three-dimensional coordinates of the body pixel points according to the first depth value, and convert the two-dimensional body normal vectors in the front and back body normal images into three-dimensional body normal vectors; Associating the three-dimensional coordinates of the body pixel points with the three-dimensional body normal vector to generate three-dimensional body data; Determine the three-dimensional coordinates of the clothing pixel point according to the second depth value, and convert the two-dimensional clothing normal vector in the front and back clothing normal image into a three-dimensional clothing normal vector; The three-dimensional coordinates of the clothing pixel points are associated with the three-dimensional clothing normal vectors to generate three-dimensional clothing data.

7. The method for three-dimensional reconstruction of a human body according to claim 1, characterized in that: The step of inputting the three-dimensional body data and the three-dimensional clothing data into a pre-trained multi-layer perceptron for three-dimensional reconstruction to generate a three-dimensional digital human with clothing includes: Performing feature fusion on the three-dimensional body data and the three-dimensional clothing data to generate a corresponding fusion feature vector; Input the fused feature vector into a pre-trained multi-layer perceptron, forward propagate the fused feature vector through a multi-layer structure, and predict and output three-dimensional reconstruction features including the body and clothing; The three-dimensional reconstruction features are decoded and converted to generate corresponding three-dimensional point clouds or meshes, and a three-dimensional digital human wearing clothes is reconstructed based on the three-dimensional point clouds or meshes.

8. A human body three-dimensional reconstruction device, characterized in that: include: A model generation module is used to obtain an original human body image to be processed, perform posture estimation on the original human body image, and generate a human body parameterized model; A body normal rendering module, used for performing normal rendering processing on the human body parameterized model to generate corresponding front and back body normal images; A clothing normal prediction module, used to perform clothing normal prediction based on the front and back body normal maps in combination with the original human body image to obtain the front and back clothing normal maps; A three-dimensional data extraction module, used to extract body three-dimensional data from the front and back body normal images, and to extract clothing three-dimensional data from the front and back clothing normal images; The three-dimensional reconstruction module is used to input the three-dimensional data of the body and the three-dimensional data of the clothing into a pre-trained multi-layer perceptron for three-dimensional reconstruction to generate a three-dimensional digital human with clothing.

9. A computer device, characterized in that: comprising at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the human body three-dimensional reconstruction method described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the human body three-dimensional reconstruction method described in any one of claims 1-7.

Images