Human body three-dimensional reconstruction method and system based on deep learning

Through deep learning human feature extraction model and three-dimensional reconstruction technology, combined with texture acquisition model, the problems of high background requirements and equipment requirements of human body three-dimensional reconstruction in the existing technology are solved, improving the accuracy and quality of the model, and simplifying the process , suitable for complex backgrounds and non-frontal images.

CN119006742AActive Publication Date: 2024-11-22CHINA POST TIMES TELECOMMUNICATIONS TECHNOLOGY CO LTD +1

Patent Information

Application Number
CN202411007528.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-22
Estimated Expiration
2044-07-25

AI Technical Summary

Technical Problem

The existing three-dimensional reconstruction technology of human body has high background requirements, the recognition process is cumbersome and inaccurate, the equipment requirements are high, and it is difficult to apply on a large scale. The traditional self-attention network method with direct regression parameters does not learn enough discriminant features, resulting in low quality of the generated human body model. , long reconstruction time.

Method used

Using deep learning-based human feature extraction model, occupancy network and traveling cube modeling algorithm, combined with texture acquisition model, adaptive learning of features in the image through the perception model, channel selection model and feature interaction model, and extract accurate human features. And use UV mapping to combine textures to generate a textured three-dimensional model of the human body.

Benefits of technology

It improves the accuracy and quality of the three-dimensional model of the human body, reduces reconstruction time, is suitable for non-frontal and complex background images, simplifies the data processing process, and enhances the generalization performance of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006742A_ABST
    Figure CN119006742A_ABST
Patent Text Reader

Abstract

The invention relates to a human body three-dimensional reconstruction method and system based on deep learning, and belongs to the technical field of three-dimensional reconstruction, and the method comprises the steps: receiving a target image which is an image containing a to-be-reconstructed human body; processing the target image by adopting a human body feature extraction model to obtain target human body features, wherein the target human body features are used for reflecting a to-be-reconstructed human body in the target image; processing the target human body features by adopting the human body three-dimensional reconstruction model to obtain an initial human body three-dimensional model; processing the target image by adopting a texture acquisition model to obtain a two-dimensional human body texture; and combining the initial human body three-dimensional model and the human body textures by adopting a texture mapping model to obtain a target human body three-dimensional model, wherein the target human body three-dimensional model is a human body three-dimensional model with textures. According to the method, multiple models are set in advance, the target images which are not in front of the human body and have complex backgrounds can be received through mutual cooperation of the multiple models, and the human body three-dimensional model with the texture is generated based on the target images, so that the quality of the obtained human body three-dimensional model is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to a method and system for three-dimensional reconstruction of the human body based on deep learning. Background Art

[0002] Human body 3D reconstruction technology is a technology that uses computer vision and image processing technology to reconstruct the 3D geometry and surface texture of the human body by extracting information from 2D images or videos. It is widely used in computer graphics, human-computer interaction, virtual reality, augmented reality, medical imaging, virtual digital humans, and human comfort analysis.

[0003] At present, human body 3D reconstruction technology mainly includes: 1. Pre-build a human pose estimation and 3D reconstruction network, crop a sub-image containing the human body from the training image through the target detection and segmentation network, determine the global position parameters of the sub-image relative to the training image, and then predict the pose-related parameters of the human body in the sub-image through the position parameter prediction network. Then, use the human body model construction network to build a 3D human body model based on the predicted pose-related parameters, and then use the 2D projection model to perform 2D projection on the constructed 3D human body model. Finally, based on the obtained 2D projection results and the constructed 3D human body model, train the parameter prediction network, human body model construction network and 2D projection model. The disadvantage of this technology is that it has high requirements for the background of the training image. A complex background requires a cumbersome human body recognition process, and the accuracy of human body recognition cannot be guaranteed; 2. Construct a digital human (skinned multi-person linear, SMPL) model based on vertex offset modeling for the human body to obtain the length of the human limb bones. Then, add a new camera to the camera array, and iterate and optimize the calibration relationship between the new camera and the camera array according to the length of the limb bones and the 3D human joints obtained by the initial two-camera intersection and the 2D human joints under the new camera to obtain the optimized 3D human joints. Calculate the distance between the human joints in the optimized 3D human joints to optimize the morphological parameters and obtain the best morphological parameters. Secondly, calculate the posture parameters in the SMPL model based on the change in the relative rotation angle of the optimized 3D human joints to solve the complicated external calibration process of the multi-camera system and the error coupling problem in the human posture estimation, and estimate a more accurate human posture. The disadvantage of this technology is that it has high requirements for equipment, requires the construction of a complex camera array, is difficult to apply on a large scale, and the generation process requires continuous iterative updates, which takes a long time; 3. Input the human body image into the global encoder to obtain the first body parameter and the global two-dimensional feature, input the first hand feature, the first head feature and the human body image into the local decoder to obtain the first hand parameter and the first head parameter; the first hand feature and the first head feature are separated from the global two-dimensional feature. Then, input the first body parameter, the first hand parameter and the first head parameter into the component interaction module for component interaction to obtain the three-dimensional human body reconstruction result. The disadvantage of this technology is that its network model belongs to the traditional self-attention network method of direct regression parameters, which lacks the learning of discriminant features, resulting in poor results of the generated human body model and the need for cumbersome processing of different parts of the human body, resulting in a long generation time.

[0004] It can be seen that the current 3D reconstruction technology of the human body requires multiple high-precision frontal photos of the human body, or requires cumbersome human body recognition methods for complex background environments, which also requires large amounts of calculations and long reconstruction time. In addition, the traditional self-attention network method of direct regression parameters does not learn the discriminant features well, resulting in low quality of the generated human body model, which also makes it difficult to apply 3D reconstruction technology of the human body on a large scale. Summary of the invention

[0005] In order to solve the problem of low quality of human body models generated by human body 3D reconstruction technology, the present application provides a human body 3D reconstruction method and system based on deep learning.

[0006] In a first aspect of the present application, a method for three-dimensional reconstruction of a human body based on deep learning is provided. The method comprises: receiving a target image, wherein the target image is an image containing a human body to be reconstructed; Processing the target image with a human body feature extraction model to obtain target human body features, wherein the target human body features are used to reflect the human body to be reconstructed in the target image; Using a human body three-dimensional reconstruction model to process the target human body features to obtain an initial human body three-dimensional model; Processing the target image using a texture acquisition model to obtain a two-dimensional human body texture; The initial human body three-dimensional model and the human body texture are combined by using a texture mapping model to obtain a target human body three-dimensional model, wherein the target human body three-dimensional model is a human body three-dimensional model with texture.

[0007] By adopting the above technical solution, the human feature extraction model first receives the target image and extracts the target person's features, and then the human 3D reconstruction model is used to construct an initial human 3D model based on the target person's features. At the same time, the texture acquisition model is also used to process the target image to obtain the two-dimensional human texture of the human body to be reconstructed. Finally, the initial human 3D model and the two-dimensional human texture are combined to obtain the target human 3D model. It can be seen that the present application is provided with a variety of models in advance, and through the cooperation of a variety of models, it is possible to receive non-human frontal and complex background target images, and generate a textured human 3D model based on the target image, thereby improving the quality of the obtained human 3D model.

[0008] In a possible implementation: before using the human feature extraction model to process the target image to obtain the target human feature, the method includes: training the human feature extraction model; The method for training the human body feature extraction model comprises: Acquire a human body image set, the human body image set comprising a plurality of training images, wherein the training images are images containing human bodies; calculate training features of each training image, wherein the training features are data predicted by the human body feature extraction model and used to reflect the human body in the training image; Acquire actual features of each training image, where the actual features are data that truly reflect the human body in the training image; construct a loss function based on the training features and actual features of the plurality of training images; The human feature extraction model is trained according to the loss function until the value output by the loss function is less than a preset value, and the human feature extraction model is output.

[0009] By adopting the above technical scheme, a loss function is first constructed based on the training features and actual features of the training image, and then the human feature extraction model is trained according to the loss function until the value output by the loss function is less than the preset value, and the trained human feature extraction model is output to ensure the accuracy of the extraction results of the human feature extraction model.

[0010] In a possible implementation, the step of calculating the training features of each training image set includes: Processing the training image using a perceptual model to obtain a first feature; Linearly concatenate the training image and the first feature to obtain a second feature; Processing the second feature using a channel selection model to obtain a third feature; Linearly concatenate the first feature and the third feature to obtain a fourth feature; Processing the fourth feature using a feature interaction model to obtain a fifth feature; The third feature and the fifth feature are linearly concatenated to obtain a training feature.

[0011] By adopting the above technical scheme, the perception model, channel selection model and feature interaction model are used according to the context information of the training image, and multi-level and multi-angle feature extraction and fusion technology is also used to adaptively learn the important features related to the human body in the training image, thereby improving the accuracy of the extraction results of the trained human feature extraction model.

[0012] In a possible implementation: the perception model includes a convolution layer, an activation function layer and an attention layer; The convolution layer is used to access the training image and perform convolution on the training image; The attention layer is connected to the convolution layer, and the attention layer is used to extract key features in the training image, and the key features are used to reflect the human body in the training image; The activation function layer is arranged between the convolution layer and the attention layer, and the activation function layer is used to adjust the weights of key features in the training image.

[0013] By adopting the above technical solution, the perception model can increase the weight of key features in the training image, so that the key features can be captured more effectively in the subsequent processing flow, providing data support for promoting the training features to continuously approach the actual features, thereby improving the performance of the human feature extraction model.

[0014] In a possible implementation, the step of acquiring a human body image set includes: Receiving a plurality of the training images; Adding Gaussian noise to each of the training images and cutting out invalid areas to obtain a final training image, wherein the invalid areas are areas that do not contain a human body; The final set of training images is used as the human body image set.

[0015] By adopting the above-mentioned technical scheme, in order to improve the generalization performance of the trained human feature extraction model, the present application introduces the preprocessing steps of regional noise injection and cropping invalid areas, which helps the human feature extraction model to capture more local details and process occluded parts, so that even if the human feature extraction model receives training images that are not the front of the human body and have complex backgrounds, it can still extract training features from the training images.

[0016] In a possible implementation: the three-dimensional reconstruction model of the human body is configured with an occupancy network and a marching cubes modeling algorithm; The step of using the human body three-dimensional reconstruction model to process the target human body features to obtain an initial human body three-dimensional model includes: The occupancy network is used to calculate the occupancy probability of sampling points in a three-dimensional space according to the target human body features, wherein the three-dimensional space is a preset three-dimensional space to be reconstructed, and the sampling points are vertices of the grid units after the three-dimensional space is evenly divided into a plurality of grid units; Retaining sampling points whose occupancy probability is greater than a preset probability; The marching cubes modeling algorithm is used to construct an initial three-dimensional human body model based on the retained sampling points.

[0017] In a possible implementation, the occupancy probability of the sampling point is calculated by the following calculation formula: θ (F 目 ,V)=τ, where θ is the parameter of the occupied network, F 目 is the target human feature, V is the sampling point, and τ is the occupancy probability of the sampling point V.

[0018] In a possible implementation, the step of constructing an initial human body three-dimensional model based on the retained sampling points using the marching cubes modeling algorithm includes: Constructing a new three-dimensional space based on the retained sampling points; Segmenting the new three-dimensional space to obtain a plurality of cubic units, each of which contains a volume data point or a volume data value, wherein both the volume data point and the volume data value are used to reflect the target human body feature; Extracting an isosurface, wherein the isosurface is composed of a plurality of cubic units of the same volume data point, or a plurality of cubic units of the same volume data value; Locking the intersection of the isosurface and each cubic unit constituting the isosurface; Connecting the intersection points using a triangulation method to form a triangular mesh; The triangular meshes are combined to obtain the initial human body three-dimensional model.

[0019] By adopting the above technical solutions, the occupancy network has certain advantages in tasks such as 3D reconstruction and 3D object recognition, especially in 3D reconstruction tasks under a single perspective, which can simplify the data acquisition and processing process, and can better handle problems such as occlusion and perspective changes, providing technical support for obtaining an accurate 3D model of the target human body. The marching cube modeling algorithm is a classic and effective 3D surface reconstruction algorithm that can process various types of volume data and has good robustness and performance at different resolutions, further providing technical support for obtaining an accurate 3D model of the target human body.

[0020] In a possible implementation: the texture map model is configured with UV mapping; The method of combining the initial human body 3D model and the human body texture using a texture mapping model to obtain a target human body 3D model includes: Mapping the initial human body three-dimensional model onto UV coordinates using the UV mapping; Obtaining UV coordinates corresponding to the human body texture; Overlapping the UV coordinates corresponding to the initial human body three-dimensional model with the UV coordinates corresponding to the human body texture to form a reconstructed UV coordinate; The reconstructed UV coordinates are mapped onto a three-dimensional model using inverse UV mapping to obtain the target human body three-dimensional model.

[0021] By adopting the above technical solution, the initial human body 3D model and the human body texture are mapped to different UV coordinates respectively, and then the two UV coordinates are overlapped to obtain one UV coordinate, and finally the obtained UV coordinate is converted to the three-dimensional space by reverse UV mapping, so as to obtain a human body 3D model with texture.

[0022] In a second aspect of the present application, a human body three-dimensional reconstruction system based on deep learning is provided. The system comprises: A data acquisition module, used for receiving a target image, wherein the target image is an image containing a human body to be reconstructed; A first processing module, configured to process the target image using a human feature extraction model to obtain target human features, wherein the target human features are used to reflect a human body to be reconstructed in the target image; The second processing module is used to process the target human body features by using the human body three-dimensional reconstruction model to obtain an initial human body three-dimensional model; the third processing module is used to process the target image by using the texture acquisition model to obtain a two-dimensional human body texture; The data combination module is used to combine the initial human body three-dimensional model and the human body texture by using a texture mapping model to obtain a target human body three-dimensional model, wherein the target human body three-dimensional model is a human body three-dimensional model with texture.

[0023] In summary, the present application includes at least one of the following beneficial technical effects: First, the perception model, channel selection model and feature interaction model are used to adaptively learn the important features related to the human body on the training image according to the context information of the training image, and the multi-level and multi-angle feature extraction and fusion technology is used to improve the accuracy of the extraction results of the trained human feature extraction model. Based on the trained human feature extraction model, the human feature extraction model receives the target image and extracts the target person's features, and then uses the occupancy network and marching cube modeling algorithm in the human 3D reconstruction model to build an initial human 3D model according to the target person's features. Secondly, the texture acquisition model is also used to process the target image to obtain the two-dimensional human body texture of the human body to be reconstructed. Finally, the initial human body 3D model is mapped to the UV coordinate, the initial human body 3D model located on the UV coordinate and the two-dimensional human body texture are combined, and then the reverse UV mapping is performed to obtain the human body 3D model with texture. It can be seen that based on the human body feature extraction model with guaranteed accuracy, no matter the input target image is the non-frontal view of the human body to be reconstructed or the background is complex, it can capture more local details in the target image and process the occluded parts to obtain accurate target features, thereby providing data support for the subsequent high-quality target human body 3D model. In addition, a variety of currently mature models are used to reconstruct the target human body three-dimensional model based on target features, which reduces the analysis time, improves the reconstruction efficiency, and further improves the quality of the generated target human body three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flowchart of a method for three-dimensional reconstruction of a human body based on deep learning in an embodiment of the present application.

[0025] Figure 2 This is an example diagram of the composition structure of the perception model in the embodiment of the method of the present application.

[0026] Figure 3 It is an example diagram of the composition structure of the channel selection model in the embodiment of the method of the present application.

[0027] Figure 4 It is an example diagram of the composition structure of the feature interaction model in the embodiment of the method of the present application.

[0028] Figure 5 It is a block diagram of a human body three-dimensional reconstruction system based on deep learning in an embodiment of the present application.

[0029] Explanation of the accompanying drawings: 1. Data acquisition module; 2. First processing module; 3. Second processing module; 4. Third processing module; 5. Data combination module; 6. Data training module. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] In order to accurately capture human features when the input human image is non-frontal or has a complex background, this application introduces a deep learning algorithm, which uses deep neural networks and large-scale data sets to adaptively learn the relationship between the human area in the human image and the human features, thereby improving the accuracy of 3D reconstruction of the human body. Figure 1 As shown, an embodiment of the present application proposes a method for three-dimensional reconstruction of the human body based on deep learning, and the main process of the method is described as follows.

[0032] Step S100: receiving a target image, where the target image is an image containing a human body to be reconstructed.

[0033] In response to the user's demand for three-dimensional reconstruction of the human body, the image selected by the user and containing the human body to be reconstructed is set as the target image, the target image is received and the following steps are performed.

[0034] Step S200: using a human body feature extraction model to process a target image to obtain target human body features, where the target human body features are used to reflect the human body to be reconstructed in the target image.

[0035] The human feature extraction model is a pre-trained model that can automatically identify and extract target human features that reflect the human body to be reconstructed when receiving a target image. Specifically, the training process of the human feature extraction model is shown in steps S210 to S250.

[0036] Step S210, obtaining a human body image set, the human body image set including a plurality of training images, the training images being images containing human bodies. Specifically, the human body image set may be derived from an open source data set, such as the AFLW2000-3D database, the AFLW face database, or the Helen face database, or may be derived from a database organized or supplemented by experts.

[0037] Based on the obtained human image set, regional noise is injected into each training image in the human image set, specifically, Gaussian noise is added for data enhancement. Then, the training images with larger invalid areas in the human image set are screened, where the invalid areas are areas that do not contain human bodies, and the invalid areas of the training images are cropped, leaving only the areas where the human bodies are located in the training images. Finally, the same training images are merged as the final training images, so that the obtained training images contain more human information.

[0038] Step S220: Calculate the training features of each training image.

[0039] First, the perceptual model is used to process the training image to obtain the first feature. Figure 2 As shown, the perception model includes a convolution layer, an activation function layer, and an attention layer, wherein the convolution layer is used to access the training image and perform convolution operations on the training image. In this example, there are multiple convolution layers, that is, there are multiple convolution layers in the perception model, and each convolution layer is used to perform convolution operations on the training image. In this example, the activation function layer uses a nonlinear activation function (rectified linear unit, ReLU). The ReLU activation function is one of the commonly used activation functions in deep learning. It can introduce nonlinear factors into the neural network of deep learning, so that the neural network can learn and represent more complex functions. The ReLU activation function here is mainly used in conjunction with the attention layer. Specifically, the attention layer identifies and extracts the key features in the training image that reflect the human body, and the ReLU activation function is used to adjust the weights of the key features in the training image.

[0040] The attention layer uses the attention mechanism (convolutional block attention module, CBAM), also known as the CBAM attention layer. Among them, the CBAM attention layer is divided into the channel attention module (CAM) and the spatial attention module (SAM). The channel attention module aims to extract the correlation between different channels in the training image. The channel represents the output dimension of the training image. The working process of the channel attention module is: the global information carried by each channel is captured by the global average pooling (GAP) technology, and then the weight of each channel is adjusted using the ReLU activation function according to the global information carried by each channel. For example, if the importance of the global information carried by channel a is higher than that of the global information carried by channel b, the weight of channel a is increased. The importance of global information is defined in advance. The weights are then used to perform weighted summation on the training image to generate a feature representation with stronger correlation on the training image. The spatial attention module is connected in series with the channel attention module. Based on the processing results of the channel attention module, the spatial attention module highlights the correlation between different positions on the training image. The position represents the pixel unit on the training image, also known as the pixel position. The working process of the spatial attention module is: use the channel weights output by the channel attention module to perform a one-dimensional convolution operation on the training image to capture the correlation between different positions on the training image. Then, use the built-in activation function to normalize the convolution output to the range of [0,1] to obtain the weight of each pixel position on the training image. It should be noted that the built-in activation function of the spatial attention module uses the Sigmoid activation function.

[0041] Combining the channel attention module and the spatial attention module can simultaneously focus on the correlation between multiple channels and the correlation between spatial positions on the training image, so that the subsequent human feature extraction model can more effectively capture the important features related to the human body on the training image and improve the model performance. In this example, the feature identified and extracted by the attention layer is called the first feature, and the first feature is represented by F1. The first feature F1 is the feature identified after adjusting the channel weight and spatial weight. In short, the training image passes through the convolution layer, and the convolution layer extracts local features and outputs them as feature out11. The ReLU activation function layer promotes the nonlinear transformation of feature out11 to obtain feature out12. The CBAM attention layer identifies the global information and position information of the channel on the training image, and then adjusts the channel weight of the training image through the ReLU activation function, and adjusts the spatial weight of the training image through the built-in activation function to change the feature out12 to generate the first feature F1. It should be noted that feature out11-feature out12 means that the features extracted from the training image are in different processing stages, and are not used to limit the data output by the perception model.

[0042] The first feature F1 is composed of F ij For example, i≤2, j≤2, then F=(F 11 ,F 12 ,F 21 ,F 22 ), the combination can also be in matrix form. Specifically, F ij The calculation formula is as follows: Among them, Q and K refer to the query and key in the attention layer respectively, and both Q and K are in matrix form. The Exp(,) function means that for each query, the similarity between the query and all keys is calculated, and the similarity is normalized to obtain the attention weight. ij It indicates that the CABM attention layer learns the attention weight between the i-th eigenvector of Q and the j-th eigenvector of K based on the feature out12, and N indicates the number of eigenvectors in the feature matrix of K.

[0043] Then, based on the obtained first feature F1, the training image and the first feature F1 are linearly concatenated to obtain the second feature F2. In this example, linear concatenation is a feature fusion method, which means simply connecting multiple features together to form a larger feature vector or feature matrix.

[0044] Then, based on the obtained second feature F2, the channel selection model is used to process the second feature F2 to obtain the third feature F3. Figure 3As shown, the channel selection model is composed of the first convolution layer, the average pooling layer, the second convolution layer, the first activation layer, the third convolution layer, the first splicing layer and the second activation layer from the input end to the output end, wherein the first activation layer and the second activation layer both use the ReLU activation function. Specifically, the working process of the channel selection model is as follows: the first convolution layer performs a convolution operation on the second feature F2 to obtain feature out21, feature out21 is processed by the average pooling layer, the second convolution layer and the first activation layer in turn to obtain feature out22, the third convolution layer convolves feature out21 and feature out22 to obtain feature out23, and the first splicing layer uses a linear splicing method to linearly splice feature out21 and feature out23 to obtain feature out24. Finally, the second activation function processes feature out24 to obtain the third feature F3.

[0045] It should be noted that the above-mentioned features out21 to out24 refer to the second feature F2 being in different processing stages, and are not used to limit the data output by the channel selection model.

[0046] In addition, based on the obtained third feature F3, the first feature F1 and the third feature F3 are linearly concatenated to obtain the fourth feature F4. The linear concatenation used here is also a feature fusion method, which also means simply connecting multiple features together to form a larger feature vector or feature matrix.

[0047] Finally, based on the fourth feature F4, the feature interaction model is used to process the fourth feature F4 to obtain the fifth feature F5. Figure 4 As shown in the figure, the feature interaction model is composed of the fourth convolution layer, the fifth convolution layer, the sixth convolution layer, the second splicing layer, the normalization layer and the third splicing layer from the input end to the output end, wherein the second splicing layer and the third splicing layer both use the linear splicing method. Specifically, the working process of the feature interaction model is as follows: the fourth convolution layer performs a convolution operation on the fourth feature F4 to obtain three identical feature matrices, which are represented by X, Y, and Z respectively. The calculation formulas of the feature matrices X, Y, and Z all use the following formulas: Among them, H and W are the height and width of the training image respectively, and X ijis the value at position (i, j) on the training image. The fifth convolutional layer convolves the feature matrix Y with the feature matrix Z, and the sixth convolutional layer convolves again on the basis of the fifth convolutional layer, that is, after the feature matrix Y and the feature matrix Z are convolved, they are convolved with the feature matrix X. The second concatenation layer linearly concatenates the feature matrix X and the data output by the sixth convolutional layer to obtain the attention map, which is represented by A, then A=X+(Y×Z)×X. The normalization layer uses the Softmax function to decompose the attention map A into the horizontal attention map and the vertical attention map, which are represented by AW and AH respectively. The third concatenation layer performs weighted calculation on the fourth feature F4, the horizontal attention map AW, and the vertical attention map AH to obtain the fifth feature F5, that is, F5=F4+AW×ω1+AH×ω2, where ω1 and ω2 are the weights of the horizontal attention map AW and the vertical attention map AH, respectively, and ω1 and ω2 are both defined in advance.

[0048] Based on the obtained fifth feature F5, the third feature F3 and the fifth feature F5 are linearly concatenated to obtain the training feature. The training feature is denoted by F 训 The linear concatenation used here is also a feature fusion method, which means simply connecting multiple features together to form a larger feature vector or feature matrix.

[0049] It should be noted that, through the above calculation process, the training features of each training image in the human image set can be calculated, and only after the training features of all training images are calculated, step S240 is entered. In actual training, the more training images are used, the higher the accuracy of the human feature extraction model obtained by subsequent training, so this example uses all the training images in the human image set. In other examples, the number of training images used can also be set according to needs.

[0050] Step S230, obtaining the actual features of each training image. When the human image set is established in step S210, each of the collected training images has real human features, also called actual features. The actual features are denoted by F 实 Indicates that, so in the case of actual characteristics F 实 When needed, it can be directly called from the human body image set.

[0051] Step S240: construct a loss function based on the training features and actual features of the multiple training images. Specifically, the calculation formula of the loss function is: The superscript T is used to indicate the transpose operation of the matrix, which is used to interchange rows and columns.

[0052] Step S250: Train the human feature extraction model according to the loss function until the value output by the loss function is less than a preset value, and then output the human feature extraction model. Specifically, when the value output by the loss function is less than the preset value, it means that the training features obtained by the human feature extraction model have approached or are equal to the actual features, so the trained human feature extraction model is output.

[0053] Based on the trained human feature extraction model, the target image is input into the human feature extraction model, and the human feature extraction model automatically identifies and extracts the target human features reflecting the human body to be reconstructed, and then enters step S300.

[0054] Step S300: using the human body 3D reconstruction model to process the target human body features to obtain an initial human body 3D model.

[0055] The 3D reconstruction model of the human body uses an occupancy network, which is a method based on deep learning for directly reconstructing 3D objects from a single or multiple 2D images. Traditional 3D reconstruction methods for the human body usually require image sequences taken from multiple perspectives or 3D point cloud data acquired using sensors such as laser scanning. However, the occupancy network attempts to learn the representation of 3D objects directly from a single or multiple 2D images without explicitly reconstructing from multiple perspectives or using other sensors. The basic idea of ​​the occupancy network is to divide the 3D space into a grid (usually a cubic grid), and then use a neural network to predict whether each grid cell is occupied by an object. In this way, a representation of a 3D object can be established in 3D space without explicitly reconstructing the geometry or surface.

[0056] Therefore, after the occupancy network receives the target human features, the specific working process is: pre-set the three-dimensional space to be reconstructed, divide the three-dimensional space into multiple identical grid units, and use a vertex of each grid unit as a sampling point. The sampling point is represented by V, so a grid unit has four sampling points. Then, based on the target human features, the occupancy probability of the sampling point V in the three-dimensional space is predicted. The calculation formula of the occupancy probability is as follows: f θ (F 目 ,V)=τ, where θ is the parameter of the occupied network, F 目 is the target human feature, τ is the occupancy probability of the sampling point V, τ is between 0 and 1, and the closer τ is to 1, the greater the possibility that the sampling point is occupied.

[0057] Then, the sampling points whose occupancy probability τ is greater than the preset probability are retained, and the preset probability is a probability value defined in advance.

[0058] Finally, the sample points that are retained are collected and the Marching Cube modeling algorithm is used to construct a human body 3D model based on the sample point set. The human body 3D model obtained here is called the initial human body 3D model. The Marching Cube modeling algorithm is a classic and effective 3D surface reconstruction algorithm that can process various types of human body data and has good robustness and performance at different resolutions. Due to its simple and intuitive implementation and good results, the Marching Cube modeling algorithm has been widely used in the field of 3D modeling and visualization.

[0059] The working principle of the above-mentioned marching cube modeling algorithm is as follows: first, segment the three-dimensional space: segment the three-dimensional space into a series of cubic units. Since this step is based on the retained sampling points to further generate a three-dimensional human body model, when segmenting the three-dimensional space, it is based on the re-segmentation of the three-dimensional space composed of multiple grids whose occupancy probability of the sampling points is greater than the preset probability. Therefore, each cubic unit contains a volume data point or volume data value, which is a representation of the target human body feature. Then, extract the isosurface: traverse each cubic unit, and determine the isosurface (i.e., the surface whose value is equal to or exceeds the threshold) by comparing the value inside the cubic unit with the preset threshold. The interpolation method is usually used to estimate the position of the isosurface. Secondly, triangulate: for each isosurface, determine the multiple cubic units that constitute the isosurface, and lock the intersection of the isosurface and each cubic unit that constitutes the isosurface, and use the triangulation method to connect the intersections to form a triangular mesh. The obtained triangular mesh finally constitutes the mesh representation of the entire surface. Finally, merge the surface mesh: merge the triangular meshes in the space into an overall surface mesh to obtain the initial human body three-dimensional model. Simply put, the marching cube modeling algorithm connects the equal-value surfaces in multiple small cubes and the intersection points of the small cubes to form one or more triangular meshes, and the triangular meshes in all the small cubes are connected to form the initial three-dimensional model of the human body.

[0060] Step S400: Use the texture acquisition model to process the target image to obtain a two-dimensional human body texture.

[0061] The texture acquisition model is configured with UV mapping, where UV mapping is to map the two-dimensional target image onto the surface of the initial three-dimensional human body model. Specifically, the texture acquisition model first identifies the texture of the human body to be reconstructed on the target image, removes impurities or noise to obtain a two-dimensional human body texture, and then uses UV mapping to map the two-dimensional human body texture onto the UV coordinates.

[0062] Step S500: Using a texture mapping model to combine the initial human body 3D model and the human body texture to obtain a target human body 3D model, where the target human body 3D model is a human body 3D model with texture.

[0063] The texture mapping model is also configured with UV mapping, where the UV mapping maps each vertex on the multiple cubic units constituting the initial three-dimensional human body model to a UV coordinate on a two-dimensional plane, thereby obtaining a flattened human body, that is, a two-dimensional human body image.

[0064] Specifically, the working process of the texture mapping model is as follows: first, the initial human body three-dimensional model is mapped to UV coordinates using UV mapping, that is, the triangular mesh obtained by the above-mentioned marching cube modeling algorithm is used to map the UV coordinates of the vertices of each triangular mesh to a two-dimensional plane to form a UV coordinate. Then, the UV coordinates corresponding to the initial human body three-dimensional model and the UV coordinates corresponding to the two-dimensional human body texture are overlapped to form a reconstructed UV coordinate, and the reconstructed UV coordinate contains the human body and human body texture to be reconstructed. Finally, the inverse UV mapping is used to map the reconstructed UV coordinates to the three-dimensional model, thereby obtaining a human body three-dimensional model with texture. In this example, the human body three-dimensional model with texture is referred to as the target human body three-dimensional model.

[0065] In summary, the implementation principle of a method for three-dimensional reconstruction of the human body based on deep learning in an embodiment of the present application is as follows: First, a perception model, a channel selection model and a feature interaction model are used according to the contextual information of the training image, while focusing on multiple dimensions (channel information and spatial information, etc.) in the training image, and multi-level and multi-angle feature extraction and fusion technology are used to adaptively learn important features related to the human body on the training image, so as to improve the accuracy of the extraction results of the human body feature extraction model obtained by training. Based on the trained human body feature extraction model, the human body feature extraction model receives the target image and extracts the target person's features, and then uses the occupancy network and marching cube modeling algorithm in the human body three-dimensional reconstruction model to construct an initial human body three-dimensional model based on the target person's features. Secondly, a texture acquisition model is also used to process the target image to obtain the human body texture in a two-dimensional form of the human body to be reconstructed. Finally, the initial human body three-dimensional model is mapped to the UV coordinate, the initial human body three-dimensional model located on the UV coordinate and the two-dimensional human body texture are combined, and then the reverse UV mapping is performed to obtain the human body three-dimensional model with texture. It can be seen that based on the human feature extraction model with guaranteed accuracy, no matter the input target image is the non-frontal view of the human body to be reconstructed or the background is complex, it is possible to capture more local details in the target image and process the occluded parts to obtain accurate target features, thereby providing data support for the subsequent high-quality target human 3D model. In addition, the target human 3D model is reconstructed based on the target features using the current mature technology, which reduces the analysis time, improves the reconstruction efficiency, and further improves the quality of the generated target human 3D model.

[0066] Figure 5A block diagram of a deep learning-based three-dimensional human body reconstruction system according to an embodiment of the present application is shown, the system comprising a data acquisition module 1, a first processing module 2, a second processing module 3, a third processing module 4 and a data combination module 5.

[0067] The data acquisition module 1 is used to receive a target image, where the target image is an image containing a human body to be reconstructed.

[0068] The first processing module 2 is used to process the target image using a human feature extraction model to obtain target human features, and the target human features are used to reflect the human body to be reconstructed in the target image.

[0069] The second processing module 3 is used to process the target human body features by using the human body 3D reconstruction model to obtain an initial human body 3D model.

[0070] The third processing module 4 is used to process the target image using the texture acquisition model to obtain a two-dimensional human body texture.

[0071] The data combination module 5 is used to combine the initial human body 3D model and the human body texture by using the texture mapping model to obtain a target human body 3D model, where the target human body 3D model is a human body 3D model with texture.

[0072] In addition, the human body three-dimensional reconstruction system based on deep learning also includes a data training module 6, which is used to train a human body feature extraction model based on a human body image set. The trained human body feature extraction model can be called by the first processing module 2.

[0073] The modules involved in the embodiments described in the present application may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a data acquisition module 1, a first processing module 2, a second processing module 3, a third processing module 4, a data combination module 5 and a data training module 6. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the data acquisition module 1 may also be described as a "module for receiving a target image".

[0074] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0075] In order to better execute the program of the above method, the present application also provides a device, which includes a memory and a processor.

[0076] The memory may be used to store instructions, programs, codes, code sets or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the above-mentioned method for three-dimensional reconstruction of a human body based on deep learning, etc.; the data storage area may store data involved in the above-mentioned method for three-dimensional reconstruction of a human body based on deep learning, etc.

[0077] The processor may include one or more processing cores. The processor calls the data stored in the memory by running or executing the instructions, programs, code sets or instruction sets stored in the memory, performs various functions of the present application and processes data. The processor may be at least one of a special purpose integrated circuit, a digital signal processor, a digital signal processing device, a programmable logic device, a field programmable gate array, a central processing unit, a controller, a microcontroller and a microprocessor. It is understandable that for different devices, the electronic device used to implement the above-mentioned processor function can also be other, and the embodiments of the present application are not specifically limited.

[0078] The present application also provides a computer-readable storage medium, for example, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes. The computer-readable storage medium stores a computer program that can be loaded by a processor and execute the above-mentioned deep learning-based three-dimensional human body reconstruction method.

[0079] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for three-dimensional reconstruction of human body based on deep learning, characterized in that: include: receiving a target image, wherein the target image is an image containing a human body to be reconstructed; Processing the target image with a human body feature extraction model to obtain target human body features, wherein the target human body features are used to reflect the human body to be reconstructed in the target image; Using a human body three-dimensional reconstruction model to process the target human body features to obtain an initial human body three-dimensional model; Processing the target image using a texture acquisition model to obtain a two-dimensional human body texture; The initial human body three-dimensional model and the human body texture are combined by using a texture mapping model to obtain a target human body three-dimensional model, wherein the target human body three-dimensional model is a human body three-dimensional model with texture.

2. The method for three-dimensional reconstruction of a human body based on deep learning according to claim 1, characterized in that: Before the human body feature extraction model is used to process the target image to obtain the target human body features, the method includes: training the human body feature extraction model; The method for training the human body feature extraction model comprises: Acquire a human body image set, the human body image set comprising a plurality of training images, wherein the training images are images containing human bodies; calculate training features of each training image, wherein the training features are data predicted by the human body feature extraction model and used to reflect the human body in the training image; Acquire actual features of each training image, where the actual features are data that truly reflect the human body in the training image; construct a loss function based on the training features and actual features of the plurality of training images; The human feature extraction model is trained according to the loss function until the value output by the loss function is less than a preset value, and the human feature extraction model is output.

3. The method for three-dimensional reconstruction of a human body based on deep learning according to claim 2, characterized in that: The calculating of the training features of each training image set includes: Processing the training image using a perceptual model to obtain a first feature; Linearly concatenate the training image and the first feature to obtain a second feature; Processing the second feature using a channel selection model to obtain a third feature; Linearly concatenate the first feature and the third feature to obtain a fourth feature; Processing the fourth feature using a feature interaction model to obtain a fifth feature; The third feature and the fifth feature are linearly concatenated to obtain a training feature.

4. The method for three-dimensional reconstruction of a human body based on deep learning according to claim 3, characterized in that: The perception model includes a convolution layer, an activation function layer and an attention layer; The convolution layer is used to access the training image and perform convolution on the training image; The attention layer is connected to the convolution layer, and the attention layer is used to extract key features in the training image, and the key features are used to reflect the human body in the training image; The activation function layer is arranged between the convolution layer and the attention layer, and the activation function layer is used to adjust the weights of key features in the training image.

5. The method for three-dimensional reconstruction of a human body based on deep learning according to claim 2, characterized in that: The step of acquiring a human body image set comprises: Receiving a plurality of the training images; Adding Gaussian noise to each of the training images and cutting out invalid areas to obtain a final training image, wherein the invalid areas are areas that do not contain a human body; The final set of training images is used as the human body image set.

6. The method for three-dimensional reconstruction of a human body based on deep learning according to claim 1, characterized in that: The human body three-dimensional reconstruction model is configured with an occupancy network and a marching cubes modeling algorithm; The step of using the human body three-dimensional reconstruction model to process the target human body features to obtain an initial human body three-dimensional model includes: The occupancy network is used to calculate the occupancy probability of sampling points in a three-dimensional space according to the target human body features, wherein the three-dimensional space is a preset three-dimensional space to be reconstructed, and the sampling points are vertices of the grid units after the three-dimensional space is evenly divided into a plurality of grid units; Retaining sampling points whose occupancy probability is greater than a preset probability; The marching cubes modeling algorithm is used to construct an initial three-dimensional human body model based on the retained sampling points.

7. The method for three-dimensional reconstruction of a human body based on deep learning according to claim 6, characterized in that: The occupancy probability of the sampling point is calculated by the following formula: f θ (F 目 ,V)=τ, Where θ is the parameter of the occupancy network, F 目 is the target human feature, V is the sampling point, and τ is the occupancy probability of the sampling point V.

8. The method for three-dimensional reconstruction of a human body based on deep learning according to claim 6, characterized in that: The step of using the marching cubes modeling algorithm to construct an initial human body three-dimensional model based on the retained sampling points includes: Constructing a new three-dimensional space based on the retained sampling points; Segmenting the new three-dimensional space to obtain a plurality of cubic units, each of which contains a volume data point or a volume data value, wherein both the volume data point and the volume data value are used to reflect the target human body feature; Extracting an isosurface, wherein the isosurface is composed of a plurality of cubic units having the same volume data points, or a plurality of cubic units having the same volume data values; Locking the intersection of the isosurface and each cubic unit constituting the isosurface; Connecting the intersection points using a triangulation method to form a triangular mesh; The triangular meshes are combined to obtain the initial human body three-dimensional model.

9. The method for three-dimensional reconstruction of a human body based on deep learning according to claim 1, characterized in that: The texture map model is configured with UV mapping; The method of combining the initial human body 3D model and the human body texture using a texture mapping model to obtain a target human body 3D model includes: Mapping the initial human body three-dimensional model onto UV coordinates using the UV mapping; Obtaining UV coordinates corresponding to the human body texture; Overlapping the UV coordinates corresponding to the initial human body three-dimensional model with the UV coordinates corresponding to the human body texture to form a reconstructed UV coordinate; The reconstructed UV coordinates are mapped onto a three-dimensional model using inverse UV mapping to obtain the target human body three-dimensional model.

10. A human body three-dimensional reconstruction system based on deep learning, characterized in that: include: A data acquisition module (1) is used to receive a target image, wherein the target image is an image containing a human body to be reconstructed; A first processing module (2) is used to process the target image using a human feature extraction model to obtain target human features, wherein the target human features are used to reflect the human body to be reconstructed in the target image; A second processing module (3) is used to process the target human body features using a human body three-dimensional reconstruction model to obtain an initial human body three-dimensional model; A third processing module (4) is used to process the target image using a texture acquisition model to obtain a two-dimensional human body texture; The data combination module (5) is used to combine the initial human body three-dimensional model and the human body texture by using a texture mapping model to obtain a target human body three-dimensional model, wherein the target human body three-dimensional model is a human body three-dimensional model with texture.

Citation Information

Patent Citations

  • Three-dimensional face model reconstruction method and device, storage medium and computer equipment

    CN114037802A

  • Joint modeling method and device for enhancing local features of pedestrians

    CN115240121A

  • Human body three-dimensional modeling method, and method and device for training human body structure generation model

    CN116645468A

  • Texture transfer and synthesis using aligned maps in image generation systems and applications

    US20230274492A1

  • Three-dimensional model reconstruction method and apparatus, and three-dimensional reconstruction model training method and apparatus

    WO2022057526A1

Cited By

  • Texture generation method and system of three-dimensional character model and storage medium

    CN119359882A