Three-dimensional reconstruction method, device, electronic equipment and storage medium

Abstract

The present disclosure relates to a three-dimensional reconstruction method, device, electronic device and storage medium, the method comprising: obtaining a human body template corresponding to each human body image of a human body to be reconstructed and a normal map of each human body image; for each human body image, obtaining a first feature corresponding to the human body image, a second feature corresponding to the human body image, and a converted third feature corresponding to the human body image; obtaining a fusion feature of the human body to be reconstructed based on the first feature corresponding to each human body image, the second feature corresponding to each human body image, and the converted third feature corresponding to each human body image; sampling spatial points in a space related to the human body to be reconstructed to obtain a set of spatial points, and determining a spatial point in the set of spatial points for three-dimensional reconstruction based on a feature corresponding to each spatial point in the set of spatial points; and generating a three-dimensional model of the human body to be reconstructed based on each spatial point used for three-dimensional reconstruction.

Description

Technical Field

[0001] The present disclosure relates to the field of the Internet, and in particular to a three-dimensional reconstruction method, device, electronic device and storage medium. Background Art

[0002] The 3D reconstruction technology of the human body is widely used in the fields of remote conferencing, webcasting, etc. Multi-view 3D reconstruction technology is a 3D reconstruction technology that uses multiple human body images including the human body to be reconstructed to obtain a 3D model of the human body to be reconstructed. The multiple human body images including the human body to be reconstructed are obtained by photographing the human body to be reconstructed by multiple cameras, each with a different viewing angle.

[0003] At present, multi-view 3D reconstruction technology relies on the intrinsic and extrinsic parameters of each camera. Before 3D reconstruction is performed using multiple human body images, the user needs to calibrate each camera in advance to determine the intrinsic and extrinsic parameters of each camera. This results in the user having to perform cumbersome operations to perform 3D reconstruction using multiple human body images. Summary of the invention

[0004] The present disclosure provides a three-dimensional reconstruction method, device, electronic device and storage medium to at least solve the problem in the related art that users need to perform cumbersome operations to perform three-dimensional reconstruction. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a three-dimensional reconstruction method is provided, comprising:

[0006] Acquire a human body template corresponding to each human body image to be reconstructed, and acquire a normal map of each human body image, wherein each human body image corresponds to a different viewing angle;

[0007] For each human body image, extract a first feature corresponding to the human body image from the human body image, extract a second feature corresponding to the human body image from a normal map of the human body image, extract an original third feature corresponding to the human body image from a human body template corresponding to the human body image, and convert the original third feature corresponding to the human body image into a converted third feature corresponding to the human body image;

[0008] Based on the first feature corresponding to each human body image, the second feature corresponding to each human body image, and the converted third feature corresponding to each human body image, a fusion feature of the human body to be reconstructed is obtained;

[0009] Sampling spatial points in a space related to the human body to be reconstructed to obtain a spatial point set, and determining a spatial point in the spatial point set for three-dimensional reconstruction based on a feature corresponding to each spatial point in the spatial point set, wherein the feature corresponding to the spatial point is a feature corresponding to the position of the spatial point in the fused features of the human body to be reconstructed;

[0010] Based on each spatial point used for three-dimensional reconstruction, a three-dimensional model of the human body to be reconstructed is generated.

[0011] In some embodiments, obtaining the normal map of each human body image includes:

[0012] For each human body image, the human body image is input into a generative adversarial network to obtain a normal map of the human body image output by the generative adversarial network.

[0013] In some embodiments, obtaining the fusion feature of the human body to be reconstructed based on the first feature, the second feature and the converted third feature corresponding to each human body image includes:

[0014] The first feature corresponding to each human body image, the second feature corresponding to each human body image, and the converted third feature corresponding to each human body image are input into a neural network with an attention mechanism to obtain a fusion feature of the human body to be reconstructed output by the neural network with an attention mechanism.

[0015] In some embodiments, sampling spatial points in a space related to the human body to be reconstructed to obtain a set of spatial points includes:

[0016] Based on a first sampling radius, sampling spatial points in a first subspace in the space, wherein the first subspace corresponds to a visible part of the human body to be reconstructed, and the visible part is included in at least one human body image;

[0017] Based on a second sampling radius, spatial points are sampled in a second subspace in the space, wherein the second subspace corresponds to an invisible part of the human body to be reconstructed, the invisible part is not included in any human body image, and the first sampling radius is smaller than the second sampling radius.

[0018] In some embodiments, based on the feature corresponding to each spatial point in the set of spatial points, determining the spatial point in the set of spatial points for three-dimensional reconstruction includes:

[0019] For each spatial point in the set of spatial points, the feature of the spatial point is input into a multilayer perceptron to obtain the probability corresponding to the spatial point output by the multilayer perceptron, wherein the probability corresponding to the spatial point indicates the probability that the spatial point is on the surface of the human body to be reconstructed; when the probability corresponding to the spatial point is greater than a probability threshold, the spatial point is determined as a spatial point for three-dimensional reconstruction.

[0020] According to a second aspect of an embodiment of the present disclosure, there is provided a three-dimensional reconstruction device, comprising:

[0021] An acquisition module is configured to acquire a human body template corresponding to each human body image to be reconstructed, and to acquire a normal map of each human body image, wherein each human body image corresponds to a different viewing angle;

[0022] A feature extraction module is configured to extract, for each human body image, a first feature corresponding to the human body image from the human body image, extract a second feature corresponding to the human body image from a normal map of the human body image, extract an original third feature corresponding to the human body image from a human body template corresponding to the human body image, and convert the original third feature corresponding to the human body image into a converted third feature corresponding to the human body image;

[0023] A feature fusion module is configured to obtain a fusion feature of a human body to be reconstructed based on a first feature corresponding to each human body image, a second feature corresponding to each human body image, and a converted third feature corresponding to each human body image;

[0024] A sampling module is configured to sample spatial points in a space related to the human body to be reconstructed to obtain a set of spatial points, and determine a spatial point in the set of spatial points for three-dimensional reconstruction based on a feature corresponding to each spatial point in the set of spatial points, wherein the feature corresponding to the spatial point is a feature corresponding to the position of the spatial point in the fused features of the human body to be reconstructed;

[0025] The reconstruction module is configured to generate a three-dimensional model of the human body to be reconstructed based on each spatial point used for three-dimensional reconstruction.

[0026] In some embodiments, the acquisition module is further configured to input each human body image into a generative adversarial network to obtain a normal map of the human body image output by the generative adversarial network.

[0027] In some embodiments, the feature fusion module is further configured to input the first feature corresponding to each human body image, the second feature corresponding to each human body image, and the converted third feature corresponding to each human body image into a neural network with an attention mechanism to obtain the fused features of the human body to be reconstructed output by the neural network with an attention mechanism.

[0028] In some embodiments, the sampling module is further configured to sample spatial points in a first subspace in the space based on a first sampling radius, wherein the first subspace corresponds to the visible part of the human body to be reconstructed and the visible part is included in at least one human body image; and to sample spatial points in a second subspace in the space based on a second sampling radius, wherein the second subspace corresponds to the invisible part of the human body to be reconstructed and the invisible part is not included in any human body image, and the first sampling radius is smaller than the second sampling radius.

[0029] In some embodiments, the sampling module is further configured to input the features of each spatial point in the set of spatial points into a multilayer perceptron, and obtain the probability corresponding to the spatial point output by the multilayer perceptron, wherein the probability corresponding to the spatial point indicates the probability of the spatial point being on the surface of the human body to be reconstructed; when the probability corresponding to the spatial point is greater than a probability threshold, the spatial point is determined as a spatial point for three-dimensional reconstruction.

[0030] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0031] processor;

[0032] a memory for storing instructions executable by the processor;

[0033] The processor is configured to execute the instructions to implement the above three-dimensional reconstruction method.

[0034] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned three-dimensional reconstruction method.

[0035] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising a computer program, which enables the electronic device to perform the above-mentioned three-dimensional reconstruction method when the computer program is run on the electronic device.

[0036] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0037] When three-dimensional reconstruction of a human body to be reconstructed is performed using multiple human body images of the human body to be reconstructed, three-dimensional reconstruction can be performed without the user having to calibrate each camera in advance to obtain a three-dimensional model of the human body to be reconstructed. This avoids the problem of users having to perform tedious operations to perform three-dimensional reconstruction using multiple human body images, thereby improving the convenience of three-dimensional reconstruction.

[0038] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0040] Figure 1 is a flow chart of an embodiment of a three-dimensional reconstruction method according to an exemplary embodiment;

[0041] Figure 2 is a structural block diagram of a three-dimensional reconstruction device according to an exemplary embodiment;

[0042] Figure 3 The present invention is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0043] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0044] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0045] Figure 1 FIG. 1 is a flow chart of an embodiment of a three-dimensional reconstruction method according to an exemplary embodiment. The method comprises the following steps:

[0046] Step 101 , obtaining a human body template corresponding to each human body image to be reconstructed and a normal map of each human body image.

[0047] In the present disclosure, each of the multiple cameras may respectively photograph the human body to be reconstructed to obtain multiple original images of the human body to be reconstructed, and each original image includes the human body to be reconstructed.

[0048] For each original image of a human body to be reconstructed, a rectangular frame surrounding the human body to be reconstructed in the original image can be determined, and the original image can be cropped to obtain the portion of the original image surrounded by the rectangular frame. The portion of the original image surrounded by the rectangular frame is a sub-image in the original image, and the size of the portion of the original image surrounded by the rectangular frame is the size of the rectangular frame. The portion of the original image surrounded by the rectangular frame can be used as a human body image, or the size of the portion of the original image surrounded by the rectangular frame can be enlarged from the size of the rectangular frame to a preset size to obtain a human body image.

[0049] In the present disclosure, each human body image to be reconstructed corresponds to a different viewing angle. For each human body image to be reconstructed, the viewing angle corresponding to the human body image is the viewing angle used by the camera that captured the original image to which the human body image belongs when capturing the original image to which the human body image belongs.

[0050] In the present disclosure, the human body template can be a Skinned Multi-Person Linearmodel (SMPL for short), and the human body template can be regarded as a collection of parameters related to the human body in space. The human body template can include shape parameters, pose parameters of the human body to be reconstructed, the positions of the joints of the human body to be reconstructed, and the positions of the vertices of the human body to be reconstructed.

[0051] In the present disclosure, for each human body image, the human body image can be input into a neural network for generating a human body template, and the neural network for generating a human body template outputs a human body template corresponding to the human body image. The neural network for generating a human body template can be a graph convolutional neural network. The neural network for generating a human body template is pre-trained with multiple human body images for training and annotated human body templates corresponding to each human body image for training. In the process of training a neural network for generating a human body template, a human body image for training is input into the neural network for generating a human body template, and the neural network for generating a human body template outputs a predicted human body template corresponding to the human body image for training, and the parameters of the neural network for generating a human body template are updated according to the loss between the predicted human body template corresponding to the human body image for training and the annotated human body template corresponding to the human body image for training.

[0052] In the present disclosure, for a human body image, each human body image pixel point in the human body image corresponds to a spatial point in space, and each human body image pixel point in the human body image corresponds to a normal map pixel point in the normal map of the human body image. For each human body image pixel point in the human body image, the position of the human body image pixel point is the same as the position of the normal map pixel point corresponding to the human body image pixel point, and the pixel value of the normal map pixel point corresponding to the human body image pixel point is the normal vector of the spatial point corresponding to the human body image pixel point.

[0053] In the present disclosure, for a human body image, the normal vector of the spatial point corresponding to each human body image pixel in the human body image can be determined, and the normal map of the human body image can be generated based on the normal vector of the spatial point corresponding to each human body image pixel in the human body image.

[0054] In some embodiments, obtaining a normal map of each human body image includes: for each human body image, inputting the human body image into a generative adversarial network, and obtaining a normal map of the human body image output by the generative adversarial network.

[0055] In the present disclosure, a generative adversarial network is pre-trained using a plurality of human images for training and annotated normal maps of each human image for training. In a training process for a generative adversarial network, a human image for training is input into the generative adversarial network, and the generative adversarial network outputs a predicted normal map of the human image for training, and the parameters of the generative adversarial network are updated according to the loss between the predicted normal map of the human image for training and the annotated normal map of the human image for training.

[0056] In the present disclosure, when obtaining the normal map of a human body image, the human body image can be directly input into a generative adversarial network to obtain the normal map of the human body image output by the generative adversarial network, without having to determine the normal vector of the spatial point corresponding to each human body image pixel point in the human body image one by one. Thus, the normal map of the human body image can be obtained quickly, thereby improving the speed of obtaining the normal map of the human body image, and further improving the speed of three-dimensional reconstruction.

[0057] Step 102: For each human body image, obtain a first feature corresponding to the human body image, a second feature corresponding to the human body image, and a converted third feature corresponding to the human body image.

[0058] In the present disclosure, for each human body image to be reconstructed, a first feature corresponding to the human body image can be extracted from the human body image by a convolutional neural network for extracting a first feature. The convolutional neural network for extracting the first feature is pre-trained by using a plurality of human body images for training and annotated first features corresponding to each human body image for training. In a process of training the convolutional neural network for extracting the first feature, a human body image for training is input into the convolutional neural network for extracting the first feature, and the convolutional neural network for extracting the first feature outputs a predicted first feature corresponding to the human body image for training, and the parameters of the convolutional neural network for extracting the first feature are updated according to the loss between the predicted first feature corresponding to the human body image for training and the annotated first feature corresponding to the human body image for training.

[0059] In the present disclosure, for each human body image to be reconstructed, the second feature corresponding to the human body image can be extracted from the normal map of the human body image by a convolutional neural network for extracting the second feature. The convolutional neural network for extracting the second feature is pre-trained through a plurality of human body images for training and the annotated second features corresponding to each human body image for training. In the process of training the convolutional neural network for extracting the second feature, the normal map of a human body image for training is input into the convolutional neural network for extracting the second feature, and the convolutional neural network for extracting the second feature outputs the predicted second feature corresponding to the human body image for training, and the parameters of the convolutional neural network for extracting the second feature are updated according to the loss between the predicted second feature corresponding to the human body image for training and the annotated second feature corresponding to the human body image for training.

[0060] In the present disclosure, for each human body image to be reconstructed, the original third feature corresponding to the human body image can be extracted from the human body template corresponding to the human body image by a convolutional neural network for extracting the original third feature. The convolutional neural network for extracting the original third feature is pre-trained by a human body template corresponding to each human body image for training in a plurality of human body images for training and annotated original third features corresponding to each human body image for training. In a process of training the convolutional neural network for extracting the original third feature, a human body template corresponding to a human body image for training is input into the convolutional neural network for extracting the original third feature, and the convolutional neural network for extracting the original third feature outputs the predicted original third feature corresponding to the human body image for training, and the parameters of the convolutional neural network for extracting the original third feature are updated according to the loss between the predicted original third feature corresponding to the human body image for training and the annotated original third feature corresponding to the human body image for training.

[0061] In the present disclosure, the original third feature corresponding to the human body image includes a plurality of sub-features, and each sub-feature in the original third feature corresponding to the human body image corresponds to a spatial point.

[0062] In the present disclosure, for each human body image, the original third feature corresponding to the human body image is a feature in a three-dimensional coordinate system related to the viewing angle corresponding to the human body image, and the transformed third feature corresponding to each human body image is a feature in a target three-dimensional coordinate system related to a unified viewing angle.

[0063] For any original third feature corresponding to a human body image, the following formula can be used to transform the original third feature corresponding to the human body image into the target three-dimensional coordinate system to obtain the transformed third feature V corresponding to the human body image: t’ :

[0064] Vt’ =W(W -1 (V t ,J(β t ), θ t , W t ), J(β t’ ), θ t’ , W t )

[0065] β t represents the posture parameter of the human body template corresponding to the human body image at the viewing angle t, β t’ represents the posture parameters of the human body template corresponding to the human body image under the same viewing angle t', J(β t ) indicates that in β t The joint rotation matrix under t’ ) indicates that in β t’ The joint rotation matrix under t represents the shape parameter of the human body template corresponding to the human body image at viewing angle t, θ t’ represents the shape parameters of the human body template corresponding to the human body image under the same viewing angle t', W t Represents the skin drive weight parameter;

[0066] W() represents the skin driving function, W -1 () represents the skin drive inverse process function, V t Indicates the original third feature corresponding to the human body image;

[0067] W -1 The input parameters of () include: V t ,J(β t ), θ t , W t , W -1 () is used to convert V t Converted into the feature corresponding to the standard posture parameter under the unified perspective t', namely W -1 (V t ,J(β t ), θ t , W t );

[0068] The input parameters of W() include: -1 (V t ,J(β t ), θ t , W t ), J(β t’ ), θ t’ , W t , W() is used to convert W -1 (V t ,J(βt ), θ t , W t ) is converted to the corresponding posture parameter β under the unified perspective t' t’ The characteristic of V t’ .

[0069] W t Including the driving weight of each sub-feature in the original third feature corresponding to the human image, the driving weight W of the i-th sub-feature in the original third feature corresponding to the human image can be calculated by the following formula: t,i :

[0070]

[0071]

[0072]

[0073] Wj represents the inherent driving weight of the human body template corresponding to the human body image, p j , t represents the jth vertex in the human body template corresponding to the human body image, v i , t represents the spatial point corresponding to the i-th sub-feature, ||pj,tv i,t || represents the distance between the spatial point corresponding to the i-th sub-feature and the j-th vertex, and σ represents the pre-set The value of is within the preset value range.

[0074] Step 103, based on the first feature, the second feature and the converted third feature corresponding to each human body image, a fusion feature of the human body to be reconstructed is obtained.

[0075] In the present disclosure, when the fused features of the human body to be reconstructed are obtained based on the first feature, the second feature and the converted third feature corresponding to each human body image, the first feature corresponding to each human body image, the second feature corresponding to each human body image and the converted third feature corresponding to each human body image can be fused according to the prior relationship to obtain the fused features of the human body to be reconstructed.

[0076] The prior relationship may indicate how to fuse the given multiple first features, the given multiple second features, and the given multiple third features. When the fused features of the human body to be reconstructed are obtained based on the first features, the second features, and the converted third features corresponding to each human body image, the first features corresponding to each human body image are used as the given multiple first features, the second features corresponding to each human body image are used as the given multiple second features, and the converted third features corresponding to each human body image are used as the given multiple third features.

[0077] In some embodiments, based on the first feature, the second feature and the converted third feature corresponding to each human body image, obtaining the fusion feature of the human body to be reconstructed includes: inputting the first feature corresponding to each human body image, the second feature corresponding to each human body image, and the converted third feature corresponding to each human body image into a neural network with an attention mechanism, and obtaining the fusion feature of the human body to be reconstructed output by the neural network with an attention mechanism.

[0078] In the present disclosure, a neural network with an attention mechanism may refer to a neural network that uses the attention mechanism to process the input of the neural network. For example, the neural network with the attention mechanism may be a recurrent neural network with an attention mechanism, a Transformer network with an attention mechanism, etc.

[0079] The attention mechanism is suitable for fusing different types of features to obtain corresponding features. The first feature corresponding to each human image, the second feature corresponding to each human image, and the converted third feature corresponding to each human image are input into a neural network with an attention mechanism. The neural network with an attention mechanism can more accurately fuse the first feature corresponding to each human image, the second feature corresponding to each human image, and the converted third feature corresponding to each human image, and the obtained fused features of the human body to be reconstructed have a high accuracy.

[0080] In the present disclosure, a neural network with an attention mechanism is pre-trained so that the neural network with the attention mechanism learns in advance how to fuse a given plurality of first features, a given plurality of second features, and a given plurality of third features.

[0081] In the present disclosure, a neural network with an attention mechanism can be trained in advance by a training data set of the neural network with an attention mechanism and a labeled fusion feature corresponding to each training data in the training data set. The training data includes: a plurality of first features for training, a plurality of second features for training, and a plurality of third features for training.

[0082] In a process of training a neural network with an attention mechanism, a training data is input into the neural network with an attention mechanism, and the neural network with an attention mechanism outputs a predicted fusion feature corresponding to the training data. The parameters of the neural network with an attention mechanism are updated according to the loss between the predicted fusion feature corresponding to the training data and the labeled fusion feature corresponding to the training data.

[0083] Step 104 , sampling spatial points in the space related to the human body to be reconstructed to obtain a spatial point set, and determining the spatial points in the spatial point set for three-dimensional reconstruction based on the features corresponding to each spatial point in the spatial point set.

[0084] In the present disclosure, the space related to the human body to be reconstructed is a space in a three-dimensional space that surrounds the human body to be reconstructed. The space related to the human body to be reconstructed can be determined according to any human body image and any posture corresponding to the human body image.

[0085] For a human body image, the posture corresponding to the human body image is the posture of the camera when shooting the human body image. When the human body image and the posture corresponding to the human body image are used to determine the space related to the human body to be reconstructed, multiple pixel points on the contour of the human body to be reconstructed in the human body image can be determined. According to the posture corresponding to the human body image, the spatial point corresponding to each pixel point is determined, and the spatial points corresponding to each pixel point are connected to obtain the contour of the human body to be reconstructed in the three-dimensional space. The space within the contour of the human body to be reconstructed in the three-dimensional space is the space related to the human body to be reconstructed.

[0086] In the present disclosure, when sampling spatial points in a space related to a human body to be reconstructed to obtain a set of spatial points, multiple spatial points as sphere centers located in the space related to the human body to be reconstructed can be determined. For each spatial point as the sphere center, spatial points can be sampled in a sphere with the spatial point as the sphere center as the sphere center and a preset sampling radius as the radius to obtain multiple spatial points located in the sphere. All the obtained spatial points constitute the set of spatial points.

[0087] In some embodiments, sampling spatial points in a space related to a human body to be reconstructed to obtain a set of spatial points includes: based on a first sampling radius, sampling spatial points in a first subspace in the space related to the human body to be reconstructed, the first subspace corresponds to the visible part of the human body to be reconstructed, and the visible part of the human body to be reconstructed is included in at least one human body image; based on a second sampling radius, sampling spatial points in a second subspace in the space related to the human body to be reconstructed, the second subspace corresponds to the invisible part of the human body to be reconstructed, the invisible part is not included in any human body image, and the first sampling radius is smaller than the second sampling radius.

[0088] In the present disclosure, for any part of a human body to be reconstructed, if at least one human body image includes the part, the part is a visible part of the human body to be reconstructed. For example, for the face of a human body to be reconstructed, each human body image includes the face of the human body to be reconstructed, and the face of the human body to be reconstructed is a visible part.

[0089] In the present disclosure, for any part of the human body to be reconstructed, if each human body image does not include the part, then the part is an invisible part of the human body to be reconstructed. For example, for the back of the human body to be reconstructed, if each human body image does not include the part, then the part is an invisible part of the human body to be reconstructed.

[0090] In the present disclosure, each visible part of the human body to be reconstructed corresponds to a first subspace. For each visible part of the human body to be reconstructed, the first subspace corresponding to the visible part is the space occupied by the visible part in the space related to the human body to be reconstructed.

[0091] For each first subspace in the space related to the human body to be reconstructed, when sampling spatial points in the first subspace, multiple spatial points as sphere centers located in the first subspace can be determined. For each spatial point as the sphere center, spatial points can be sampled in a sphere with the spatial point as the sphere center as the sphere center and the first sampling radius as the radius, to obtain multiple spatial points located in the sphere.

[0092] In the present disclosure, each invisible part of the human body to be reconstructed corresponds to a second subspace. For each invisible part of the human body to be reconstructed, the second subspace corresponding to the invisible part is the space occupied by the invisible part in the space related to the human body to be reconstructed.

[0093] For each second subspace in the space related to the human body to be reconstructed, when sampling spatial points in the second subspace, multiple spatial points as sphere centers located in the second subspace can be determined. For each spatial point as the sphere center, spatial points can be sampled in a sphere with the spatial point as the sphere center as the sphere center and the second sampling radius as the radius, to obtain multiple spatial points located in the sphere.

[0094] In the present disclosure, all spatial points obtained by sampling in the first subspace and all spatial points obtained by sampling in the second subspace may constitute a spatial point set.

[0095] In the present disclosure, based on a first sampling radius, spatial points are sampled in a first subspace in a space related to a human body to be reconstructed; based on a second sampling radius, spatial points are sampled in a second subspace in a space related to a human body to be reconstructed, and the first sampling radius is smaller than the second sampling radius, so that the density of spatial points obtained by sampling in the first subspace is higher than the density of spatial points obtained by sampling in the second subspace. When performing three-dimensional reconstruction, the visible part of the human body to be reconstructed can be constructed using spatial points with higher density, thereby enhancing the details of the visible part of the human body to be reconstructed.

[0096] In the present disclosure, for each spatial point in the set of spatial points, the feature corresponding to the spatial point is the feature corresponding to the position of the spatial point in the fused features of the human body to be reconstructed.

[0097] The fused feature of the human body to be reconstructed includes a plurality of sub-features, and each sub-feature in the fused feature of the human body to be reconstructed corresponds to a position in a space related to the human body to be reconstructed.

[0098] For each spatial point in the set of spatial points, if the fused features of the human body to be reconstructed include a sub-feature corresponding to the position of the spatial point, the sub-feature can be used as the feature corresponding to the position of the spatial point. If the fused features of the human body to be reconstructed do not include a sub-feature corresponding to the position of the spatial point, the sub-feature whose corresponding position is closest to the position of the spatial point can be used as the feature corresponding to the position of the spatial point.

[0099] In the present disclosure, based on the features corresponding to each spatial point in the spatial point set, the spatial points in the spatial point set used for three-dimensional reconstruction are determined.

[0100] In the present disclosure, multiple different three-dimensional models of human bodies can be acquired in advance, and the acquired three-dimensional models of human bodies can be three-dimensional models established by existing three-dimensional reconstruction methods. For each acquired three-dimensional model, multiple spatial points belonging to the three-dimensional model can be determined, and the features of each of the multiple spatial points can be determined. The type of the feature of each spatial point is the same as the type of the feature corresponding to the spatial point in the spatial point set, and the features of each spatial point are used as preset features.

[0101] In the present disclosure, when determining the spatial points in a spatial point set for three-dimensional reconstruction based on the features corresponding to each spatial point in the spatial point set, for each spatial point in the spatial point set, the similarity between the features corresponding to the spatial point and each preset feature can be calculated. If the similarity between the features corresponding to the spatial point and a preset feature is greater than a similarity threshold, the spatial point can be determined as the spatial point for three-dimensional reconstruction.

[0102] In some embodiments, based on the features corresponding to each spatial point in the spatial point set, determining the spatial point in the spatial point set for three-dimensional reconstruction includes: for each spatial point in the spatial point set, inputting the features of the spatial point into a multi-layer perceptron, obtaining the probability corresponding to the spatial point output by the multi-layer perceptron, wherein the probability corresponding to the spatial point indicates the probability that the spatial point is located on the surface of the human body to be reconstructed; when the probability corresponding to the spatial point is greater than a probability threshold, the spatial point is determined as a spatial point for three-dimensional reconstruction.

[0103] In the present disclosure, a multi-layer perceptron (MLP) is pre-trained by a training data set of the multi-layer perceptron and a label of each training data in the training data set. The training data is a feature of a spatial point for training the multi-layer perceptron, and the label of the training data indicates whether the spatial point for training the multi-layer perceptron to which the training data belongs is located on the surface of the human body.

[0104] For each of a plurality of different human bodies, a plurality of spatial points located on the surface of the human body and a plurality of spatial points within the space occupied by the human body but not located on the surface of the human body can be determined in advance, each of the determined spatial points can be used as a spatial point for training a multi-layer perceptron, and the features of each of the determined spatial points can be used as training data to generate labels for the features of each of the determined spatial points.

[0105] In a process of training a multilayer perceptron, a training data in a training data set of the multilayer perceptron is input into the multilayer perceptron, and the multilayer perceptron outputs the probability corresponding to the training data, and the probability corresponding to the training data is the probability that the spatial point for training the multilayer perceptron to which the training data belongs is located on the surface of the human body. If the probability corresponding to the training data is greater than 50%, a prediction classification result indicating that the spatial point for training the multilayer perceptron to which the training data belongs is located on the surface of the human body can be generated; if the probability corresponding to the training data is less than or equal to 50%, a prediction classification result indicating that the spatial point for training the multilayer perceptron to which the training data belongs is not located on the surface of the human body can be generated; if the prediction classification result is consistent with the label of the training data, the parameters of the multilayer perceptron are not updated; if the prediction classification result is inconsistent with the label of the training data, the parameters of the multilayer perceptron are updated.

[0106] In the present disclosure, the features of a spatial point are input into a multilayer perceptron to obtain the probability corresponding to the spatial point output by the multilayer perceptron. When the probability corresponding to the spatial point is greater than a probability threshold, the spatial point is determined as a spatial point for three-dimensional reconstruction, without calculating the similarity between the features corresponding to the spatial point and each preset feature one by one and comparing the similarity between the features corresponding to the spatial point and each preset feature with the threshold one by one. Thus, the spatial point for three-dimensional reconstruction can be determined quickly, the speed of determining the spatial point for three-dimensional reconstruction can be improved, and the speed of three-dimensional reconstruction can be improved.

[0107] Step 105 : generating a three-dimensional model of the human body to be reconstructed based on each spatial point used for three-dimensional reconstruction.

[0108] In the present disclosure, each spatial point used for three-dimensional reconstruction can be regarded as a spatial point on the surface of the human body to be reconstructed. The three-dimensional model of the human body to be reconstructed is the surface of the human body to be reconstructed, and the three-dimensional model of the human body to be reconstructed can be generated based on each spatial point used for three-dimensional reconstruction.

[0109] Figure 2 is a structural block diagram of a three-dimensional reconstruction device according to an exemplary embodiment. Figure 2 The three-dimensional reconstruction device includes: an acquisition module 201, a feature extraction module 202, a feature fusion module 203, a sampling module 204, and a reconstruction module 205.

[0110] The acquisition module 201 is configured to acquire a human body template corresponding to each human body image to be reconstructed, and acquire a normal map of each human body image, wherein each human body image corresponds to a different viewing angle;

[0111] The feature extraction module 202 is configured to extract, for each human body image, a first feature corresponding to the human body image from the human body image, extract a second feature corresponding to the human body image from the normal map of the human body image, extract an original third feature corresponding to the human body image from a human body template corresponding to the human body image, and transform the original third feature corresponding to the human body image into a target three-dimensional coordinate system to obtain a transformed third feature corresponding to the human body image;

[0112] The feature fusion module 203 is configured to obtain a fusion feature of a human body to be reconstructed based on the first feature, the second feature and the converted third feature corresponding to each human body image;

[0113] The sampling module 204 is configured to sample spatial points in a space related to the human body to be reconstructed to obtain a set of spatial points, and determine a spatial point in the set of spatial points for three-dimensional reconstruction based on a feature corresponding to each spatial point in the set of spatial points, wherein the feature corresponding to the spatial point is a feature corresponding to the position of the spatial point in the fused feature of the human body to be reconstructed;

[0114] The reconstruction module 205 is configured to generate a three-dimensional model of the human body to be reconstructed based on each spatial point used for three-dimensional reconstruction.

[0115] In some embodiments, the acquisition module 201 is further configured to input each human body image into a generative adversarial network to obtain a normal map of the human body image output by the generative adversarial network.

[0116] In some embodiments, the feature fusion module 203 is further configured to input the first feature corresponding to each human body image, the second feature corresponding to each human body image, and the converted third feature corresponding to each human body image into a neural network with an attention mechanism to obtain the fusion feature of the human body to be reconstructed output by the neural network with an attention mechanism.

[0117] In some embodiments, the sampling module 204 is further configured to sample spatial points in a first subspace in the space based on a first sampling radius, wherein the first subspace corresponds to the visible part of the human body to be reconstructed and the visible part is included in at least one human body image; and to sample spatial points in a second subspace in the space based on a second sampling radius, wherein the second subspace corresponds to the invisible part of the human body to be reconstructed and the invisible part is not included in any human body image, and the first sampling radius is smaller than the second sampling radius.

[0118] In some embodiments, the sampling module 204 is further configured to input the features of each spatial point in the set of spatial points into a multilayer perceptron to obtain the probability corresponding to the spatial point output by the multilayer perceptron, wherein the probability corresponding to the spatial point indicates the probability of the spatial point being on the surface of the human body to be reconstructed; when the probability corresponding to the spatial point is greater than a probability threshold, the spatial point is determined as a spatial point for three-dimensional reconstruction.

[0119] Figure 3 is a structural block diagram of an electronic device according to an exemplary embodiment. Figure 3The electronic device includes a processing component 322, which further includes one or more processors, and a memory resource represented by a memory 332 for storing instructions, such as applications, that can be executed by the processing component 322. The application stored in the memory 332 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 322 is configured to execute instructions to perform the above method.

[0120] The electronic device may also include a power supply component 326 configured to perform power management of the electronic device, a wired or wireless network interface 350 configured to connect the electronic device to a network, and an input / output (I / O) interface 358. The electronic device may operate based on an operating system stored in the memory 532, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™ or the like.

[0121] The present disclosure also provides a computer-readable storage medium including instructions, such as a memory including instructions, and the above instructions can be executed by an electronic device to complete the above three-dimensional reconstruction method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0122] The present disclosure also provides a computer program product, including a computer program. When the computer program is run on an electronic device, the electronic device executes the above-mentioned three-dimensional reconstruction method.

[0123] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0124] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A three-dimensional reconstruction method, characterized in that: The method comprises: Acquire a human body template corresponding to each human body image to be reconstructed, and acquire a normal map of each human body image, wherein the image acquisition angle corresponding to each human body image is different; For each human body image, extract a first feature corresponding to the human body image from the human body image, extract a second feature corresponding to the human body image from a normal map of the human body image, extract an original third feature corresponding to the human body image from a human body template corresponding to the human body image, and transform the original third feature corresponding to the human body image into a target three-dimensional coordinate system to obtain a transformed third feature corresponding to the human body image; Based on the first feature, the second feature and the converted third feature corresponding to each human body image, a fusion feature of the human body to be reconstructed is obtained; Sampling spatial points in a space related to the human body to be reconstructed to obtain a spatial point set, and determining a spatial point in the spatial point set for three-dimensional reconstruction based on a feature corresponding to each spatial point in the spatial point set, wherein the feature corresponding to the spatial point is a feature corresponding to the position of the spatial point in the fused features of the human body to be reconstructed; Based on each spatial point used for three-dimensional reconstruction, generating a three-dimensional model of the human body to be reconstructed; The step of sampling spatial points in a space related to the human body to be reconstructed to obtain a set of spatial points includes: Based on a first sampling radius, sampling spatial points in a first subspace in the space, wherein the first subspace corresponds to a visible part of the human body to be reconstructed, and the visible part is included in at least one human body image; Based on a second sampling radius, spatial points are sampled in a second subspace in the space, wherein the second subspace corresponds to an invisible part of the human body to be reconstructed, the invisible part is not included in any human body image, and the first sampling radius is smaller than the second sampling radius.

2. The method according to claim 1, characterized in that Acquiring the normal map of each human body image includes: For each human body image, the human body image is input into a generative adversarial network to obtain a normal map of the human body image output by the generative adversarial network.

3. The method according to claim 1, characterized in that Based on the first feature, the second feature and the converted third feature corresponding to each human body image, the fusion feature of the human body to be reconstructed is obtained, including: The first feature corresponding to each human body image, the second feature corresponding to each human body image, and the converted third feature corresponding to each human body image are input into a neural network with an attention mechanism to obtain a fusion feature of the human body to be reconstructed output by the neural network with an attention mechanism.

4. The method according to claim 1, characterized in that: Based on the feature corresponding to each spatial point in the set of spatial points, determining the spatial point in the set of spatial points for three-dimensional reconstruction comprises: For each spatial point in the set of spatial points, the feature of the spatial point is input into a multilayer perceptron to obtain the probability corresponding to the spatial point output by the multilayer perceptron, wherein the probability corresponding to the spatial point indicates the probability that the spatial point is on the surface of the human body to be reconstructed; when the probability corresponding to the spatial point is greater than a probability threshold, the spatial point is determined as a spatial point for three-dimensional reconstruction.

5. A three-dimensional reconstruction device, characterized in that: The device comprises: An acquisition module is configured to acquire a human body template corresponding to each human body image to be reconstructed, and to acquire a normal map of each human body image, wherein each human body image corresponds to a different viewing angle; A feature extraction module is configured to extract, for each human body image, a first feature corresponding to the human body image from the human body image, extract a second feature corresponding to the human body image from a normal map of the human body image, extract an original third feature corresponding to the human body image from a human body template corresponding to the human body image, and convert the original third feature corresponding to the human body image into a converted third feature corresponding to the human body image; A feature fusion module is configured to obtain a fusion feature of a human body to be reconstructed based on a first feature corresponding to each human body image, a second feature corresponding to each human body image, and a converted third feature corresponding to each human body image; A sampling module is configured to sample spatial points in a space related to the human body to be reconstructed to obtain a set of spatial points, and determine a spatial point in the set of spatial points for three-dimensional reconstruction based on a feature corresponding to each spatial point in the set of spatial points, wherein the feature corresponding to the spatial point is a feature corresponding to the position of the spatial point in the fused features of the human body to be reconstructed; A reconstruction module, configured to generate a three-dimensional model of the human body to be reconstructed based on each spatial point used for three-dimensional reconstruction; The sampling module is further configured to sample spatial points in a first subspace in the space based on a first sampling radius, wherein the first subspace corresponds to the visible part of the human body to be reconstructed and the visible part is included in at least one human body image; and to sample spatial points in a second subspace in the space based on a second sampling radius, wherein the second subspace corresponds to the invisible part of the human body to be reconstructed and the invisible part is not included in any human body image, and the first sampling radius is smaller than the second sampling radius.

6. The device according to claim 5, characterized in that The acquisition module is further configured to input each human body image into a generative adversarial network to obtain a normal map of the human body image output by the generative adversarial network.

7. The device according to claim 5, characterized in that The feature fusion module is further configured to input the first feature corresponding to each human body image, the second feature corresponding to each human body image, and the converted third feature corresponding to each human body image into a neural network with an attention mechanism to obtain the fusion features of the human body to be reconstructed output by the neural network with an attention mechanism.

8. The device according to claim 5, characterized in that The sampling module is further configured to input the features of each spatial point in the set of spatial points into a multilayer perceptron to obtain the probability corresponding to the spatial point output by the multilayer perceptron, wherein the probability corresponding to the spatial point indicates the probability that the spatial point is on the surface of the human body to be reconstructed; when the probability corresponding to the spatial point is greater than a probability threshold, the spatial point is determined as a spatial point for three-dimensional reconstruction.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 4. 10 . A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to claim 1 .

11. A computer program product, comprising a computer program, which enables an electronic device to execute the method according to any one of claims 1 to 4 when the computer program is run on the electronic device.

Images