Human body reconstruction method and related device

Abstract

The embodiment of the invention discloses a human body reconstruction method and a related device, and the method comprises the steps: inputting the sparse image data of a to-be-detected human body into a target neural network model, and obtaining a three-dimensional human body image which is determined by a target parameter for reconstructing the to-be-detected human body; performing micro-rendering on the three-dimensional human body image to render the three-dimensional human body image into a two-dimensional human body image; and performing joint optimization on the two-dimensional human body image and a sparse image corresponding to the sparse image data to obtain a reconstructed human body model. By adopting the embodiment of the invention, rapid reconstruction and rendering in the electronic equipment can be realized.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a human body reconstruction method and related devices. Background Art

[0002] How to quickly and efficiently reconstruct a high-precision human body is a problem that has been studied in the field of computer vision. Recently, implicit neural representation (INR) has performed well in modeling human appearance and geometric details, such as SelfRecon and NeuralActor, but such methods require a large number of training images to ensure stable network convergence. At the same time, these works usually consume a lot of graphics processing unit (GPU) computing for mesh extraction and rendering, which makes them difficult to implement in real scenes and cannot be deployed on mobile devices for use. Summary of the invention

[0003] The embodiment of the present application provides a human body reconstruction method and related devices. The reconstructed human body model has more human body details and can be quickly reconstructed and rendered in an electronic device.

[0004] In a first aspect, an embodiment of the present application provides a human body reconstruction method, which is applied to an electronic device, and the method includes:

[0005] Inputting sparse image data of the human body to be tested into the target neural network model to obtain a three-dimensional human body image, wherein the three-dimensional human body image is determined by reconstructing target parameters of the human body to be tested, and the target parameters include: human body posture parameters, human body geometry parameters and human body surface texture parameters;

[0006] Performing differentiable rendering on the three-dimensional human body image to render the three-dimensional human body image into a two-dimensional human body image;

[0007] The two-dimensional human body image and the sparse image corresponding to the sparse image data are jointly optimized to obtain a reconstructed human body model.

[0008] In a second aspect, an embodiment of the present application provides a human body reconstruction device, which is applied to an electronic device, and includes: an input unit, a differentiable rendering unit, and a joint optimization unit, wherein:

[0009] The input unit is used to input the sparse image data of the human body to be tested into the target neural network model to obtain a three-dimensional human body image, wherein the three-dimensional human body image is determined by reconstructing target parameters of the human body to be tested, and the target parameters include: human body posture parameters, human body geometry parameters and human body surface texture parameters;

[0010] The differentiable rendering unit is used to perform differentiable rendering on the three-dimensional human body image to render the three-dimensional human body image into a two-dimensional human body image;

[0011] The joint optimization unit is used to jointly optimize the two-dimensional human body image and the sparse image corresponding to the sparse image data to obtain a reconstructed human body model.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for executing the steps of any method in the first aspect of the embodiment of the present application.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps described in any method of the first aspect of the embodiment of the present application.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in any method of the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0015] It can be seen that in the embodiment of the present application, the sparse image data of the human body to be tested is input into the target neural network model to obtain a three-dimensional human body image, wherein the three-dimensional human body image is determined by the target parameters for reconstructing the human body to be tested, and the target parameters include: human body posture parameters, human body geometry parameters and human body surface texture parameters; the three-dimensional human body image is micro-rendered to render the three-dimensional human body image into a two-dimensional human body image; the two-dimensional human body image and the sparse image corresponding to the sparse image data are jointly optimized to obtain a reconstructed human body model. In this way, the electronic device can reconstruct the sparse image data of the human body to be tested based on the target neural network model, and the reconstructed human body model has more human body details, which is conducive to improving the accuracy of human body reconstruction, and can achieve rapid reconstruction and rendering in electronic devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a structural schematic diagram of a neural network model provided in an embodiment of the present application;

[0018] Figure 2 is a flow chart of a human body reconstruction method provided in an embodiment of the present application;

[0019] Figure 3 It is a schematic diagram of a process of three-dimensional human body image reconstruction provided by an embodiment of the present application;

[0020] Figure 4 is a structural schematic diagram of a human posture prediction module provided in an embodiment of the present application;

[0021] Figure 5 is a structural schematic diagram of a multi-level implicit human detail prediction module provided in an embodiment of the present application;

[0022] Figure 6 is a schematic diagram of comparison of human body reconstruction results provided in an embodiment of the present application;

[0023] Figure 7 It is a block diagram of the functional units of a human body reconstruction device provided in an embodiment of the present application;

[0024] Figure 8 This is a block diagram of the functional units of a human body reconstruction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0027] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0028] The electronic device may be a portable electronic device that also includes other functions such as a personal digital assistant and / or a music player function, such as a mobile phone, a tablet computer, a terminal device, a wearable electronic device with a wireless communication function (such as a smart watch, smart glasses), a vehicle-mounted device, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with an IOS system, an Android system, a Microsoft system, or other operating systems. The above-mentioned portable electronic device may also be other portable electronic devices, such as a laptop computer (Laptop), etc. It should also be understood that in some other embodiments, the above-mentioned electronic device may not be a portable electronic device, but a desktop computer. In some embodiments, the electronic device includes a display screen; in some embodiments, the electronic device may also include a virtual reality device.

[0029] How to quickly and efficiently reconstruct a high-precision human body is a problem that has been studied in the field of computer vision. Early methods mainly used parametric models (such as the SMPL model (Skinned Multi-Person Linear Model), SMPL-X, and STAR) to represent the shape of the human body. Such methods can effectively simulate people of different heights and obesity, but due to limitations in parametric human training data, clothing, and other issues, they lack appearance and geometric details, and there is still a large gap with the real human body. In order to obtain an accurate human body model that is closer to the real human body, methods such as VideoAvatar and Octopus use vertex-by-vertex displacement to expand the SMPL mesh surface to form a parametric model of SMPL-D to represent the human body. However, as a model without prior knowledge, its representation ability is limited and it cannot restore fine-grained details.

[0030] Recently, neural implicit representation has performed well in modeling human appearance and geometric details, such as SelfRecon and NeuralActor, but such methods require a large number of training images (e.g., about 20 synchronized cameras) to ensure stable network convergence. At the same time, these works usually consume a lot of GPU computing for mesh extraction and rendering, which makes them difficult to implement in real scenes and cannot be deployed on mobile devices for use.

[0031] In response to the above problems, the present application proposes a human body reconstruction method and related devices, which are described in detail below.

[0032] See also Figure 1 , Figure 1 It is a structural schematic diagram of a neural network model provided by the present application; the structure may include: a human posture prediction module, a multi-level implicit human detail prediction module, a human surface texture prediction module and a fusion module.

[0033] Among them, the above-mentioned neural network model can be a preset neural network model in the embodiment of the present application, or it can be a target neural network model, and the target neural network model is obtained by training the preset neural network model.

[0034] The human body posture prediction module can process the input sparse image to obtain human body posture parameters, and the human body posture parameters can be used to characterize the initial human body posture of the human body to be measured corresponding to the sparse image.

[0035] Among them, the above-mentioned multi-level implicit human detail prediction module may include multiple or multiple layers of implicit deep neural networks. After passing through an implicit deep neural network, the grid of the human body surface corresponding to the human body posture parameters obtained by the human body posture prediction module will be subdivided, and the higher-precision grid after subdivision will be input into the subsequent one or more implicit deep neural networks, which is conducive to obtaining a higher-precision human body model.

[0036] The human body surface texture prediction module can process the RGB image corresponding to the input sparse image to predict the human body surface texture.

[0037] The above fusion module may refer to a fusion module of parameterized human body and implicit human body details, which can be understood as a joint optimizer based on differentiable rendering, for example, using NvDiffrast as a differentiable renderer. In specific implementation, the human body posture parameters, human body geometry parameters, human body surface texture parameters, human body vertex offset detail information used to characterize human body geometry parameters, and colors obtained in the previous modules can be jointly optimized based on the loss function to obtain the final high-precision, drivable human body model.

[0038] It should be noted that the “multiple” in the embodiments of the present application may refer to two or more than two, which is not limited here.

[0039] See also Figure 2 , Figure 2 1 is a flow chart of a human body reconstruction method provided in an embodiment of the present application, which is applied to electronic equipment. As shown in the figure, the human body reconstruction method includes the following operations.

[0040] S201, inputting sparse image data of a human body to be tested into a target neural network model to obtain a three-dimensional human body image, wherein the three-dimensional human body image is determined by reconstructing target parameters of the human body to be tested.

[0041] Among them, the above target neural network model can be as follows Figure 1 The neural network model shown can be used to process sparse image data of a human body to be measured, and jointly optimize target parameters such as human body posture parameters, human body geometry parameters, and human body surface texture parameters to obtain a three-dimensional human body image.

[0042] In a specific implementation, the relevant parameters of the multi-level implicit human detail prediction module obtained by training the target neural network model can be loaded into the target neural network model, so as to process the sparse image data of the human body to be tested according to multiple modules of the target neural network model to obtain a three-dimensional human body image.

[0043] The above-mentioned related parameters may include at least one of the following: implicit posture features and implicit shape features, etc., which are not limited here.

[0044] The sparse image data may include at least one of the following: a sparse image, a human body contour image data, a normal image data, etc., which are not limited here. The sparse image may include RGB image data, and the human body contour image data, the normal image data, etc. may be analyzed from an RBG image.

[0045] S202: Perform microscopic rendering on the three-dimensional human body image to render the three-dimensional human body image into a two-dimensional human body image.

[0046] Among them, in the present application, the step of performing differentiable rendering on a three-dimensional human body image to obtain a two-dimensional human body image can be implemented in the process of jointly optimizing the human body posture parameters, human body vertex offset detail information and color by the corresponding fusion module of the above-mentioned target neural network model.

[0047] Among them, NvDiffrast can be used as a differentiable renderer to perform differentiable rendering of three types of images, including human body contour images, human body normal image data, and RGB images.

[0048] In the specific implementation, in the optimization stage, because the true value of the human body model cannot be obtained, differentiable rendering can be used when performing differentiable rendering on the three-dimensional human body image, and the regularization terms of the contour, normal map and normal direction are applied to constrain the optimization, so as to render the three-dimensional human body image (which can be considered as a three-dimensional geometric image) into a two-dimensional human body image.

[0049] For example, the above constraint can be expressed as follows:

[0050]

[0051] Among them, L s is the contour loss function, used to make the rendered contour Try to fit the segmentation contour S on the original input image as closely as possible k , L n is the loss function of the normal, which is used to make the rendered normal As close as possible to the normal map N predicted by PiFu k . nr It is a normal regularization term weighted by the displacement size. While making the direction of the larger displacement as close as possible to the normal direction of the original model, it relaxes the constraints of the small displacement so that it can adjust its direction more freely.

[0052] S203: jointly optimize the two-dimensional human body image and the sparse image corresponding to the sparse image data to obtain a reconstructed human body model.

[0053] The reconstructed human body model is a drivable human body model with higher precision.

[0054] It can be seen that the human body reconstruction method described in the embodiment of the present application inputs the sparse image data of the human body to be tested into the target neural network model to obtain a three-dimensional human body image, wherein the three-dimensional human body image is determined by the target parameters for reconstructing the human body to be tested, and the target parameters include: human body posture parameters, human body geometry parameters and human body surface texture parameters; the three-dimensional human body image is micro-rendered to render the three-dimensional human body image into a two-dimensional human body image; the two-dimensional human body image is jointly optimized with the sparse image corresponding to the sparse image data to obtain a reconstructed human body model. In this way, the electronic device can reconstruct the sparse image data of the human body to be tested based on the target neural network model, and the reconstructed human body model has more human body details, which is conducive to improving the accuracy of human body reconstruction, and can achieve rapid reconstruction and rendering in electronic devices.

[0055] In a possible example, the target neural network model is obtained by the electronic device training a preset neural network model, and the preset neural network model and / or the target neural network model includes at least one of the following modules: a human body posture prediction module, a multi-level implicit human body detail prediction module, a human body surface texture prediction module and a fusion module.

[0056] In a specific implementation, the electronic device can construct a data set and train a preset neural network model based on the data set to obtain a target neural network model.

[0057] Among them, during the training process, the above-mentioned preset neural network model may include a human posture prediction module, a multi-level implicit human detail prediction module and a human surface texture prediction module, so as to obtain relevant parameters and improve the performance and accuracy of the target neural network model by training the above three modules.

[0058] The above data set may include contour images, normal images, RGB images of human bodies at different angles and postures, and specific parameter true values ​​of corresponding parameterized human body models, etc., which are not limited here.

[0059] For example, the RenderPeople dataset contains 294 human models, each of which has 300 different postures and 8 viewing angles, which means there are 294x300 human posture parameters and 294x300x8 human RGB, contour, and normal maps.

[0060] The specific data in the above dataset can be derived or simulated from the original data, and the dataset is not limited to a known dataset or a private dataset with the same skeletal features as the parameterized human body.

[0061] For example, a data set can be used to train a preset neural network structure, but the data set does not contain a human body contour or normal map. Corresponding methods such as SegmentAnything, PIFU, etc. can be used to segment and predict from RGB images to obtain a human body contour and normal map. For another example, if the data set contains a human body model but does not have a parameterized human body truth value, nonlinear optimization can be used to regress the parameterized human body truth value based on the human body model.

[0062] It can be seen that in this example, a loss function can be set to use the true value of the parameterized human body model and the human body image to supervise the training of the implicit deep learning network to optimize the parameter values ​​of all parameters in the preset neural network model structure to obtain the target neural network model, which is beneficial to improve the accuracy and performance of the target neural network model.

[0063] Optionally, when training a preset neural network model, the parameters can be iteratively updated according to the gradient back-propagation method and accelerated using a GPU until the total loss is reduced to a set threshold or the number of network iterations meets the requirement and the training is stopped.

[0064] In one possible example, the target neural network model includes: a human body posture prediction module, a multi-level implicit human body detail prediction module, a human body surface texture prediction module and a fusion module, and the target parameters include: human body posture parameters, human body geometry parameters and human body surface texture parameters; the sparse image data of the human body to be tested is input into the target neural network model to obtain a three-dimensional human body image. The above method may include the following steps: inputting the sparse image included in the sparse image data into the human body posture prediction module to obtain the human body posture parameters; inputting the human body posture parameters, as well as the human body contour image data and normal image data included in the sparse image data into the multi-level implicit human body detail prediction module to obtain the human body geometry parameters; inputting the RGB image data included in the sparse image data into the human body surface texture prediction module to obtain the human body surface texture parameters; according to the loss function, inputting the human body posture parameters, the human body geometry parameters and the human body surface texture parameters into the fusion model for joint optimization to obtain the three-dimensional human body image.

[0065] In the specific implementation, Figure 3 As shown, it is a schematic diagram of a process of three-dimensional human image reconstruction, which may include a human posture prediction module, a multi-level implicit human detail prediction module, a human surface texture prediction module and a fusion module; first, through the human posture prediction module, the human posture features corresponding to the sparse image are iteratively optimized and processed to obtain human posture parameters; then, while iteratively optimizing the human posture, the posture difference is also accumulated, and the obtained human posture parameters are used as inputs of the multi-level implicit human detail prediction module, and the human contour image data and normal image data included in the sparse image data are used as inputs to obtain human body geometric parameters; further, the RGB image data included in the sparse image data is used as inputs of the human surface texture prediction module to obtain the human body surface texture parameters; finally, the fusion module can optimize the implicit vector based on the loss function, combined with the human surface geometric parameters, human posture parameters, human surface texture parameters, etc., and the implicit vector is back-propagated to the multi-level implicit human detail prediction module, and then iteratively optimizes a high-precision, drivable three-dimensional human image.

[0066] It can be seen that in this example, by training a lightweight implicit deep neural network, that is, a multi-level implicit human detail prediction module and a patch rendering method that is compatible with the traditional rendering pipeline, the above-mentioned target neural network model can be deployed and presented on electronic devices such as mobile terminals, and the three-dimensional human image can be iteratively optimized through the target neural network model, which is beneficial to improving the accuracy of human body reconstruction.

[0067] In one possible example, the sparse image included in the sparse image data is input into the human body posture prediction module to obtain the human body posture parameters. The above method may include the following steps: sampling the sparse image according to the human body posture prediction module to obtain human body image features; predicting human body posture features from the human body image features according to the parameterized human body model obtained in the training of the target neural network model; inputting the human body posture features into the network regressor for iterative optimization, and recovering the human body posture parameters for reconstructing the human body model corresponding to the human body to be tested.

[0068] The human body posture prediction module is also used to obtain parameters such as human body patch vertices.

[0069] In the specific implementation, Figure 4 As shown, it is a structural schematic diagram of a human posture prediction module. The above sparse image can be a 224x224 image. After convolution, a multi-layer spatial image pyramid is obtained to obtain human image features. The resolution of the image pyramid can be set by itself, for example, it can be 14x14, 28x28, 56x56, etc., which is not limited here. Then, uniform sampling is first performed from the image feature pyramid as the initialization of the human posture feature; then, the human posture feature can be input into the network regressor to optimize the human posture, and the new human posture is projected onto the multi-layer image pyramid, and the new human posture feature obtained by sampling the projection area is input into the network regressor again, and the above steps are continuously repeated until the target number of iterations is reached to obtain the human posture parameters for reconstructing the human model.

[0070] It can be seen that in this example, the multi-layer image feature pyramid and network regressor can be used to iterate the human body posture features in the sparse image to obtain the human body posture parameters required for subsequent human body reconstruction, which is beneficial to improving the accuracy of human body reconstruction.

[0071] In a possible example, the multi-level implicit human detail prediction module includes multiple implicit deep neural networks; the human body posture parameters, and the human body contour image data and normal image data included in the sparse image data are input into the multi-level implicit human body detail prediction module to obtain the human body geometric parameters. The above method may include the following steps: determining the human body surface according to the human body posture parameters; adding implicit posture features and implicit shape features to the human body surface, and restoring human body vertex offset detail information through the multiple implicit deep neural networks; determining the human body geometric parameters according to the human body vertex offset detail information.

[0072] Among them, Figure 5 , which is a schematic diagram of the structure of a multi-level implicit human detail prediction module, which may include a parameterized human body model and multiple / layers of implicit deep neural networks.

[0073] For example, the multi-level implicit human detail prediction module adds 32-dimensional implicit posture features and 8-dimensional implicit shape features to each vertex on the human body surface to form a 43-dimensional joint feature vector. Each layer of implicit deep neural network can use 3 64x64 fully connected deep networks. For example, the multi-level implicit human detail prediction module can include 3 layers of implicit deep neural networks.

[0074] In the specific implementation, each layer of implicit deep neural network adds implicit posture features and implicit shape features to each vertex on the human body surface to form a joint feature vector, i.e., an implicit vector. After the implicit vector is position-encoded using Fourier series, it is input into the fully connected deep network of each layer of implicit deep neural network to obtain the human body vertex offset detail information of each human body surface vertex, i.e., the detail displacement. Furthermore, the human body surface and the obtained detail displacement are accumulated to restore the human body vertex offset detail information corresponding to the new human body model.

[0075] Finally, the electronic device will perform surface segmentation on the human body patch model, enrich the number of surface vertices of the human body model, and put the human body surface of the new human body model into the next layer of implicit deep neural network module. In this way, after being processed by multiple layers of implicit deep neural network, the human body geometric parameters used to characterize the final three-dimensional human body image are obtained.

[0076] It can be seen that in this example, the multi-level implicit human detail prediction module included in the electronic device may include multiple / layers of implicit deep neural networks. After passing through an implicit deep neural network, the obtained mesh of the human body surface will be subdivided, and the higher-precision mesh after subdivision will be input into the subsequent implicit deep neural network, which is conducive to obtaining a higher-precision human body model in the later stage.

[0077] In addition, electronic devices can train lightweight multi-layer implicit deep neural networks, namely multi-layer implicit human detail prediction modules, so that the above-mentioned target neural network model can be deployed and presented on electronic devices such as mobile terminals, which is conducive to improving the practicality of human body reconstruction methods.

[0078] In one possible example, according to the loss function, the human body posture parameters, the human body geometry parameters and the human body surface texture parameters are input into the fusion model for joint optimization to obtain the three-dimensional human body image. The above method may include the following steps: regressing the initial shape information of the human body model in the sparse image and the posture information of each frame according to the human body posture prediction module; generating a parameterized human body model corresponding to the human body model according to the human body geometry parameters and the human body surface texture parameters; determining an implicit vector according to the implicit posture features and the implicit shape features; adding the implicit vector to the parameterized human body model; jointly optimizing the parameterized human body model and the implicit vector according to the loss function through the first layer of the multiple implicit deep neural networks to obtain an initial three-dimensional human body model; according to other layers of the multiple implicit deep neural networks, gradually predicting the human body vertex offset detail information corresponding to the corresponding layer; according to the human body vertex offset detail information corresponding to the corresponding layer, gradually adding human body details to the initial three-dimensional human body model to obtain the three-dimensional human body image.

[0079] Among them, the above-mentioned fusion module includes a cross-modal feature fuser and decoder, which can be based on a differentiable renderer to achieve joint optimization of human body posture parameters, human body geometry parameters and human body surface texture parameters through the feature fuser and decoder.

[0080] It can be seen that in this example, the electronic device can obtain the human body patch vertices, initial shape information and posture information of each frame of the human body posture prediction module through the fusion module, obtain the human body surface texture parameters corresponding to the human body surface texture prediction module, and obtain the implicit posture features, implicit shape features and human body vertex offset detail information predicted by the multi-level implicit human body detail prediction module. And according to the differentiable rendering combined with the above parameters, the three-dimensional human body image can be rendered into a two-dimensional human body image, and compared with the input sparse image, and various parameters can be jointly optimized to achieve differentiable rendering of human body contour images, human body normal image data and RGB images, so as to obtain the reconstructed human body model, which is conducive to improving the accuracy of human body reconstruction.

[0081] In one possible example, the two-dimensional human body image and the sparse image corresponding to the sparse image data are jointly optimized to obtain a reconstructed human body model. The above method may include the following steps: determining a preset loss function; iteratively optimizing the two-dimensional human body image according to the preset loss function and the sparse image to obtain the reconstructed human body model.

[0082] Among them, the above-mentioned preset loss function can be set by the user or by the system default, which is not limited here; the preset loss function can be determined based on experience, for example, it can be cross entropy, error loss, etc., which is not limited here.

[0083] Among them, Figure 6 As shown, it is a schematic diagram of the comparison of the results of human body reconstruction, which may include a front human body white film truth image, and a corresponding front image of the human body reconstructed by the method of the present application; a side human body white film truth image, and a corresponding side image of the human body reconstructed by the method of the present application. It can be seen that after the human body is reconstructed by the method of the present application, the front and side views of the person to be tested are more accurate and more natural, and there will be no body asymmetry and distortion, which is conducive to improving the accuracy of human body reconstruction.

[0084] It can be seen that in this example, the electronic device can use a method that combines parameterized human body and implicit neural network to reconstruct a high-precision, drivable human body model from sparse images. Compared with the current parameterized human body method, it has higher accuracy and can run the implicit neural network model in real time on electronic devices such as mobile terminals. The obtained human body model can be widely used in various mixed reality devices.

[0085] See also Figure 7 , Figure 7 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. As shown in the figure, the electronic device includes a processor, a memory, a communication interface and one or more programs, which are applied to the electronic device, wherein the one or more programs are stored in the memory, and the one or more programs are configured to execute the following steps by the processor:

[0086] Inputting sparse image data of a human body to be tested into a target neural network model to obtain a three-dimensional human body image, wherein the three-dimensional human body image is determined by reconstructing target parameters of the human body to be tested;

[0087] Performing differentiable rendering on the three-dimensional human body image to render the three-dimensional human body image into a two-dimensional human body image;

[0088] The two-dimensional human body image and the sparse image corresponding to the sparse image data are jointly optimized to obtain a reconstructed human body model.

[0089] It can be seen that the electronic device described in the embodiment of the present application inputs the sparse image data of the human body to be tested into the target neural network model to obtain a three-dimensional human body image, wherein the three-dimensional human body image is determined by the target parameters for reconstructing the human body to be tested, and the target parameters include: human body posture parameters, human body geometry parameters and human body surface texture parameters; the three-dimensional human body image is micro-rendered to render the three-dimensional human body image into a two-dimensional human body image; the two-dimensional human body image is jointly optimized with the sparse image corresponding to the sparse image data to obtain a reconstructed human body model. In this way, the electronic device can reconstruct the sparse image data of the human body to be tested based on the target neural network model, and the reconstructed human body model has more human body details, which is conducive to improving the accuracy of human body reconstruction, and can achieve rapid reconstruction and rendering in electronic devices.

[0090] In one possible example, the target neural network model is obtained by the electronic device training a preset neural network model, and the preset neural network model and / or the target neural network model includes at least one of the following modules: a human body posture prediction module, a multi-level implicit human body detail prediction module, a human body surface texture prediction module and a fusion module.

[0091] In a possible example, the target neural network model includes: a human posture prediction module, a multi-level implicit human detail prediction module, a human surface texture prediction module and a fusion module, and the target parameters include: human posture parameters, human geometry parameters and human surface texture parameters; in the aspect of inputting the sparse image data of the human body to be measured into the target neural network model to obtain a three-dimensional human body image, the above program includes instructions for executing the following steps:

[0092] Inputting the sparse image included in the sparse image data into the human body posture prediction module to obtain the human body posture parameter;

[0093] Inputting the human body posture parameters, and the human body contour image data and normal image data included in the sparse image data into the multi-level implicit human body detail prediction module to obtain the human body geometric parameters;

[0094] Inputting the RGB image data included in the sparse image data into the human body surface texture prediction module to obtain the human body surface texture parameters;

[0095] According to the loss function, the human body posture parameters, the human body geometry parameters and the human body surface texture parameters are input into the fusion model for joint optimization to obtain the three-dimensional human body image.

[0096] In a possible example, in the aspect of inputting the sparse image included in the sparse image data into the human body posture prediction module to obtain the human body posture parameter, the above program includes instructions for performing the following steps:

[0097] According to the human posture prediction module, sampling the sparse image to obtain human image features;

[0098] Predicting human body posture features from the human body image features based on the parameterized human body model obtained in the training of the target neural network model;

[0099] The human body posture features are input into a network regressor for iterative optimization, and the human body posture parameters for reconstructing a human body model corresponding to the human body to be tested are recovered.

[0100] In one possible example, the multi-level implicit human detail prediction module includes multiple implicit deep neural networks; in the aspect of inputting the human body posture parameters, and the human body contour image data and normal image data included in the sparse image data into the multi-level implicit human body detail prediction module to obtain the human body geometric parameters, the above program includes instructions for performing the following steps:

[0101] Determining a human body surface according to the human body posture parameters;

[0102] Adding implicit posture features and implicit shape features to the human body surface, and restoring human body vertex offset detail information through the multiple implicit deep neural networks;

[0103] The human body geometric parameters are determined according to the human body vertex offset detail information.

[0104] In one possible example, in the aspect of inputting the human body posture parameters, the human body geometry parameters and the human body surface texture parameters into the fusion model for joint optimization according to the loss function to obtain the three-dimensional human body image, the above program includes instructions for performing the following steps:

[0105] Regressing the human body posture prediction module to obtain initial shape information of the human body model in the sparse image and posture information of each frame;

[0106] Generate a parameterized human body model corresponding to the human body model according to the human body geometric parameters and the human body surface texture parameters;

[0107] Determining an implicit vector according to the implicit posture feature and the implicit shape feature;

[0108] adding the implicit vector to the parameterized human body model;

[0109] By using the first layer of the multiple implicit deep neural networks, the parameterized human body model and the implicit vector are jointly optimized according to the loss function to obtain an initial three-dimensional human body model;

[0110] According to other layers of the plurality of implicit deep neural networks, gradually predicting the human body vertex offset detail information corresponding to the corresponding layers;

[0111] According to the human body vertex offset detail information corresponding to the corresponding layer, human body details are gradually added to the initial three-dimensional human body model to obtain the three-dimensional human body image.

[0112] In a possible example, in the aspect of jointly optimizing the two-dimensional human body image and the sparse image corresponding to the sparse image data to obtain a reconstructed human body model, the program includes instructions for performing the following steps:

[0113] Determine the preset loss function;

[0114] According to the preset loss function and the sparse image, the two-dimensional human body image is iteratively optimized to obtain the reconstructed human body model.

[0115] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process on the method side. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0116] The embodiment of the present application can divide the electronic device into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0117] In the case of dividing each functional module into corresponding functional modules, Figure 8 A schematic diagram of a human body reconstruction device is shown. Figure 8As shown, the device is applied to an electronic device. The human body reconstruction device 800 may include: an input unit 801, a differentiable rendering unit 802 and a joint optimization unit 803, wherein:

[0118] The input unit 801 is used to input the sparse image data of the human body to be tested into the target neural network model to obtain a three-dimensional human body image, wherein the three-dimensional human body image is determined by reconstructing the target parameters of the human body to be tested;

[0119] The differentiable rendering unit 802 is used to perform differentiable rendering on the three-dimensional human body image to render the three-dimensional human body image into a two-dimensional human body image;

[0120] The joint optimization unit 803 is used to jointly optimize the two-dimensional human body image and the sparse image corresponding to the sparse image data to obtain a reconstructed human body model.

[0121] It can be seen that the human body reconstruction device provided in the embodiment of the present application inputs the sparse image data of the human body to be tested into the target neural network model to obtain a three-dimensional human body image, wherein the three-dimensional human body image is determined by the target parameters for reconstructing the human body to be tested, and the target parameters include: human body posture parameters, human body geometry parameters and human body surface texture parameters; the three-dimensional human body image is micro-rendered to render the three-dimensional human body image into a two-dimensional human body image; the two-dimensional human body image is jointly optimized with the sparse image corresponding to the sparse image data to obtain a reconstructed human body model. In this way, the electronic device can reconstruct the sparse image data of the human body to be tested based on the target neural network model, and the reconstructed human body model has more human body details, which is conducive to improving the accuracy of human body reconstruction, and can achieve rapid reconstruction and rendering in electronic devices.

[0122] In one possible example, the target neural network model is obtained by the electronic device training a preset neural network model, and the preset neural network model and / or the target neural network model includes at least one of the following modules: a human body posture prediction module, a multi-level implicit human body detail prediction module, a human body surface texture prediction module and a fusion module.

[0123] In a possible example, the target neural network model includes: a human posture prediction module, a multi-level implicit human detail prediction module, a human surface texture prediction module and a fusion module, and the target parameters include: human posture parameters, human geometry parameters and human surface texture parameters; in the aspect of inputting the sparse image data of the human body to be measured into the target neural network model to obtain a three-dimensional human body image, the input unit 801 is specifically used for:

[0124] Inputting the sparse image included in the sparse image data into the human body posture prediction module to obtain the human body posture parameter;

[0125] Inputting the human body posture parameters, and the human body contour image data and normal image data included in the sparse image data into the multi-level implicit human body detail prediction module to obtain the human body geometric parameters;

[0126] Inputting the RGB image data included in the sparse image data into the human body surface texture prediction module to obtain the human body surface texture parameters;

[0127] According to the loss function, the human body posture parameters, the human body geometry parameters and the human body surface texture parameters are input into the fusion model for joint optimization to obtain the three-dimensional human body image.

[0128] In a possible example, in the aspect of inputting the sparse image included in the sparse image data into the human body posture prediction module to obtain the human body posture parameter, the input unit 801 is specifically used for:

[0129] According to the human posture prediction module, sampling the sparse image to obtain human image features;

[0130] Predicting human body posture features from the human body image features based on the parameterized human body model obtained in the training of the target neural network model;

[0131] The human body posture features are input into a network regressor for iterative optimization, and the human body posture parameters for reconstructing a human body model corresponding to the human body to be tested are recovered.

[0132] In a possible example, the multi-level implicit human detail prediction module includes multiple implicit deep neural networks; in the aspect of inputting the human body posture parameters, and the human body contour image data and normal image data included in the sparse image data into the multi-level implicit human body detail prediction module to obtain the human body geometric parameters, the input unit 801 is specifically used for:

[0133] Determining a human body surface according to the human body posture parameters;

[0134] Adding implicit posture features and implicit shape features to the human body surface, and restoring human body vertex offset detail information through the multiple implicit deep neural networks;

[0135] The human body geometric parameters are determined according to the human body vertex offset detail information.

[0136] In a possible example, in the aspect of inputting the human body posture parameter, the human body geometry parameter and the human body surface texture parameter into the fusion model for joint optimization according to the loss function to obtain the three-dimensional human body image, the input unit 801 is specifically used for:

[0137] Regressing the human body posture prediction module to obtain initial shape information of the human body model in the sparse image and posture information of each frame;

[0138] Generate a parameterized human body model corresponding to the human body model according to the human body geometric parameters and the human body surface texture parameters;

[0139] Determining an implicit vector according to the implicit posture feature and the implicit shape feature;

[0140] adding the implicit vector to the parameterized human body model;

[0141] By using the first layer of the multiple implicit deep neural networks, the parameterized human body model and the implicit vector are jointly optimized according to the loss function to obtain an initial three-dimensional human body model;

[0142] According to other layers of the plurality of implicit deep neural networks, gradually predicting the human body vertex offset detail information corresponding to the corresponding layers;

[0143] According to the human body vertex offset detail information corresponding to the corresponding layer, human body details are gradually added to the initial three-dimensional human body model to obtain the three-dimensional human body image.

[0144] In a possible example, in the aspect of jointly optimizing the two-dimensional human body image and the sparse image corresponding to the sparse image data to obtain the reconstructed human body model, the joint optimization unit 803 is specifically used for:

[0145] Determine the preset loss function;

[0146] According to the preset loss function and the sparse image, the two-dimensional human body image is iteratively optimized to obtain the reconstructed human body model.

[0147] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0148] The electronic device provided in this embodiment is used to execute the above-mentioned human body reconstruction method, and thus can achieve the same effect as the above-mentioned implementation method.

[0149] In the case of using an integrated unit, the electronic device may include a processing module, a storage module and a communication module. Among them, the processing module can be used to control and manage the actions of the electronic device, for example, it can be used to support the electronic device to execute the steps performed by the above-mentioned input unit 801, the micro-rendering unit 802 and the joint optimization unit 803. The storage module can be used to support the electronic device to execute stored program codes and data, etc. The communication module can be used to support the communication between the electronic device and other devices.

[0150] Among them, the processing module can be a processor or a controller. It can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc.

[0151] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps of any method described in the above method embodiments, and the above computer includes an electronic device.

[0152] The embodiment of the present application also provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, the computer program is operable to cause a computer to execute some or all of the steps of any method described in the method embodiment. The computer program product may be a software installation package, and the computer includes an electronic device.

[0153] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0154] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0155] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0156] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0157] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0158] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.

[0159] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which can include: a flash drive, a read-only memory, a random access memory, a magnetic disk or an optical disk, etc.

[0160] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A human body reconstruction method, applied to electronic equipment, characterized in that: include: Inputting sparse image data of a human body to be tested into a target neural network model to obtain a three-dimensional human body image, wherein the three-dimensional human body image is determined by reconstructing target parameters of the human body to be tested; Rendering the three-dimensional human body image into a two-dimensional human body image; The two-dimensional human body image and the sparse image corresponding to the sparse image data are jointly optimized to obtain a reconstructed human body model.

2. The method according to claim 1, characterized in that: The target neural network model includes: a human posture prediction module, a multi-level implicit human detail prediction module, a human surface texture prediction module and a fusion module, and the target parameters include: human posture parameters, human geometry parameters and human surface texture parameters; The step of inputting the sparse image data of the human body to be tested into the target neural network model to obtain a three-dimensional human body image comprises: Inputting the sparse image included in the sparse image data into the human body posture prediction module to obtain the human body posture parameter; Inputting the human body posture parameters, and the human body contour image data and normal image data included in the sparse image data into the multi-level implicit human body detail prediction module to obtain the human body geometric parameters; Inputting the RGB image data included in the sparse image data into the human body surface texture prediction module to obtain the human body surface texture parameters; According to the loss function, the human body posture parameters, the human body geometry parameters and the human body surface texture parameters are input into the fusion model for joint optimization to obtain the three-dimensional human body image.

3. The method according to claim 2, characterized in that The step of inputting the sparse image included in the sparse image data into the human body posture prediction module to obtain the human body posture parameter comprises: According to the human posture prediction module, sampling the sparse image to obtain human image features; Predicting human body posture features from the human body image features based on the parameterized human body model obtained in the training of the target neural network model; The human body posture features are input into a network regressor for iterative optimization, and the human body posture parameters for reconstructing a human body model corresponding to the human body to be tested are recovered.

4. The method according to claim 3, characterized in that The multi-level implicit human detail prediction module includes multiple implicit deep neural networks; The step of inputting the human body posture parameters, and the human body contour image data and normal image data included in the sparse image data into the multi-level implicit human body detail prediction module to obtain the human body geometric parameters comprises: Determining a human body surface according to the human body posture parameters; Adding implicit posture features and implicit shape features to the human body surface, and restoring human body vertex offset detail information through the multiple implicit deep neural networks; The human body geometric parameters are determined according to the human body vertex offset detail information.

5. The method according to claim 4, characterized in that The step of inputting the human body posture parameter, the human body geometry parameter and the human body surface texture parameter into the fusion model for joint optimization according to the loss function to obtain the three-dimensional human body image comprises: Regressing the human body posture prediction module to obtain initial shape information of the human body model in the sparse image and posture information of each frame; Generate a parameterized human body model corresponding to the human body model according to the human body geometric parameters and the human body surface texture parameters; Determining an implicit vector according to the implicit posture feature and the implicit shape feature; adding the implicit vector to the parameterized human body model; By using the first layer of the multiple implicit deep neural networks, the parameterized human body model and the implicit vector are jointly optimized according to the loss function to obtain an initial three-dimensional human body model; According to other layers of the plurality of implicit deep neural networks, gradually predicting the human body vertex offset detail information corresponding to the corresponding layers; According to the human body vertex offset detail information corresponding to the corresponding layer, human body details are gradually added to the initial three-dimensional human body model to obtain the three-dimensional human body image.

6. The method according to claim 1 or 5, characterized in that: The step of jointly optimizing the two-dimensional human body image and the sparse image corresponding to the sparse image data to obtain a reconstructed human body model comprises: Determine the preset loss function; According to the preset loss function and the sparse image, the two-dimensional human body image is iteratively optimized to obtain the reconstructed human body model.

7. The method according to claim 1, characterized in that The target neural network model is obtained by training a preset neural network model by the electronic device, and the preset neural network model and / or the target neural network model includes at least one of the following modules: a human body posture prediction module, a multi-level implicit human body detail prediction module, a human body surface texture prediction module and a fusion module.

8. A human body reconstruction device, characterized in that: The device is applied to an electronic device, and comprises: an input unit, a differentiable rendering unit and a joint optimization unit, wherein: The input unit is used to input the sparse image data of the human body to be tested into the target neural network model to obtain a three-dimensional human body image, wherein the three-dimensional human body image is determined by reconstructing the target parameters of the human body to be tested; The differentiable rendering unit is used to perform differentiable rendering on the three-dimensional human body image to render the three-dimensional human body image into a two-dimensional human body image; The joint optimization unit is used to jointly optimize the two-dimensional human body image and the sparse image corresponding to the sparse image data to obtain a reconstructed human body model.

9. An electronic device, characterized in that: The method comprises a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for executing the steps in the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program for electronic data exchange is stored, wherein the computer program enables a computer to execute the method according to any one of claims 1 to 7.

Images