A 3D human body reconstruction method and system based on Transformer model
By using the Transformer model in the three-dimensional human body reconstruction technology, combining image features and key point information, the problems of low efficiency and low accuracy in the existing technology are solved, and efficient and accurate three-dimensional human body model reconstruction is achieved.
Patent Information
- Application Number
- CN202111314592.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-08
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-11-08
AI Technical Summary
The existing three-dimensional human body reconstruction technology has problems of low efficiency and low accuracy, especially the model-free method based on convolutional neural networks has shortcomings in terms of complexity and lack of detailed information.
Using a three-dimensional human body reconstruction method based on the Transformer model, through image feature extraction, local feature extraction, global feature extraction and spatial position extraction, combined with the information of key points and grid vertices, the multi-layer Transformer model is used to decouple and fusion information. Realize the precise reconstruction of three-dimensional mannequin.
It improves the efficiency and accuracy of three-dimensional human body reconstruction, can accurately predict human body models from a single RGB image, reduces the amount of calculation, enhances anti-interference ability and flexibility, and can effectively extract detailed information.
Smart Images

Figure CN114049435B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of three-dimensional human body model construction, and in particular to a three-dimensional human body reconstruction method and system based on a Transformer model. Background Art
[0002] Currently, the fields of VR (virtual reality) / VR (augmented reality), 3D film and animation production, interactive game design, and virtual fitting all require the reconstruction of accurate 3D human models. For the reconstruction of 3D human models, most of them are currently implemented using high-precision scanners or motion capture systems. Although this type of method has high modeling accuracy, it takes a lot of time and cost.
[0003] In the prior art, three-dimensional human body reconstruction algorithms can be mainly divided into optimization-based methods and learning-based methods. Among them, the optimization-based method usually uses traditional optimization methods to reconstruct the three-dimensional human body model in an iterative manner. However, since it is very difficult to directly restore the 3D model of the person in the image from the 2D image, this method needs to directly estimate a set of sparse parameters, such as SCAPE, SMPL, etc. to reduce the difficulty of the reconstruction task, so that an accurate three-dimensional human body model can be reconstructed to a certain extent. However, due to the drawbacks of traditional optimization algorithms, this type of method has high requirements for the initialization of the reconstruction algorithm and is prone to fall into a local optimal solution. In addition, the optimization-based human body reconstruction method generally takes a lot of time to iteratively optimize a reasonable model.
[0004] Compared with optimization-based methods, learning-based 3D human reconstruction methods have obvious advantages in efficiency. Learning-based 3D human reconstruction methods can generally be divided into model-based (model-related) methods and model-free (model-independent) methods. Model-based methods are similar to optimization-based methods, except that they use deep learning technology to directly predict SMPL parameters. Although this direct prediction of template parameters can make the reconstruction task simpler, the reconstruction results are often limited by the template itself, such as lack of details. Model-free methods usually use deep learning technology to directly predict the mesh vertices of the 3D human model, so that the reconstruction results are not limited by the performance of the template.
[0005] Currently, most model-free methods use convolutional neural networks to directly extract a global feature, and then use methods such as graph convolution to directly predict mesh vertices. However, this type of method is not only complex to implement and requires a lot of measurement, but also because the pure global feature will lack detailed information, so the final reconstruction accuracy is actually not high. Summary of the invention
[0006] The technical problem to be solved by the present invention is: in view of the technical problems existing in the prior art, the present invention provides a three-dimensional human body reconstruction method and system based on the Transformer model with simple implementation method, high reconstruction efficiency and accuracy.
[0007] A three-dimensional human body reconstruction method based on a Transformer model, comprising the following steps:
[0008] S01. Image feature extraction: input a human body image, and use a convolutional neural network to extract features of the input human body image to obtain image features F;
[0009] S02. Local feature extraction: upsampling the extracted image feature F to obtain a sampled feature F′, inputting the sampled feature F′ into a network composed of a plurality of residual blocks, extracting local features of key points; and generating a local feature related to human body posture for each vertex in the mesh, to obtain local features of mesh vertices;
[0010] S03. Global feature extraction: performing global feature extraction on the extracted image feature F;
[0011] S04. Spatial position extraction: using the image feature F to predict the parameterized human body model, obtaining a human body mesh and key point positions corresponding to the human body mesh;
[0012] S05. The extracted local features, global features, human body mesh, and key point positions are spliced together, and the feature vector sequence obtained after splicing is input into the Transformer model to predict the final three-dimensional human body mesh vertices and key point positions to achieve three-dimensional human body reconstruction.
[0013] Furthermore, the Transformer model includes two layers, each layer including a Transformer block. The lower layer is used to decouple key points and mesh vertices. The key points and human body mesh vertices are respectively encoded using the Transformer block in the lower layer to obtain encoded key points and mesh vertices. The encoded key points and mesh vertices are used together as inputs of the Transformer block in the upper layer, and the Transformer block in the upper layer fuses the information of the key points and mesh vertices.
[0014] Furthermore, in the step S02, the local features of the key points are linearly combined by linearly blending skin weights to obtain the local features of the mesh vertices.
[0015] Furthermore, the step S02 also includes regressing the key point position from the sampled feature F′, and using the key point error function to supervise the network to extract local features of the key points, so that the obtained local features encode the key point position information.
[0016] Furthermore, in step S03, the extracted image feature F is subjected to maximum pooling processing and then input into a multi-layer perceptron to be encoded to obtain the global feature.
[0017] Furthermore, in the step S04, the parameterized human body model is predicted using an error feedback iterative model to predict a three-dimensional human body model that is preliminarily aligned with the input human body image, and the human body mesh and the corresponding key point positions are obtained from the predicted three-dimensional human body model; in the step S05, the human body mesh and the corresponding key point positions are used as position coding information of the Transformer model to align the position coding information with the input human body image.
[0018] Furthermore, in the step S05, the global features, local features of key points, and local features of mesh vertices are spliced, and the feature vector sequence S0 includes a first vector sequence k0 corresponding to the key points and a second vector sequence v0 corresponding to the mesh vertices. The spatial positions of the key points and the mesh vertices are used to position encode the first vector sequence k0 and the second vector sequence v0, respectively.
[0019] Furthermore, the step S05 also includes, in the Transformer model, masking the key points and mesh vertices according to the segmentation information of the human body, wherein the key points are randomly masked, and the mesh vertices of the human body language segmentation corresponding to the key points are masked.
[0020] A three-dimensional human body reconstruction device based on a Transformer model, comprising:
[0021] An image feature extraction module is used to input a human body image and use a convolutional neural network to extract features of the input human body image to obtain image features F;
[0022] A local feature extraction module is used to upsample the extracted image feature F to obtain a sampled feature F′, input the sampled feature F′ into a network composed of a plurality of residual blocks, and extract the local features of the key points; and generate a local feature related to the human body posture for each vertex in the mesh to obtain the local features of the mesh vertex;
[0023] A global feature extraction module, used for performing global feature extraction on the extracted image feature F;
[0024] A spatial position extraction module, used to predict a parameterized human body model using the image feature F to obtain a human body mesh and key point positions corresponding to the human body mesh;
[0025] The Transformer model module is used to splice the extracted local features, global features with the human body mesh and key point positions, and input the feature vector sequence obtained after splicing into the Transformer model to predict the final three-dimensional human body mesh vertices and key point positions to achieve three-dimensional human body reconstruction.
[0026] A computer device comprises a processor and a memory, wherein the memory is used to store a computer program, the processor is used to execute the computer program, and the processor is used to execute the computer program to perform the above method.
[0027] In order to solve the above technical problems, the technical solution proposed by the present invention is:
[0028] Compared with the prior art, the advantages of the present invention are:
[0029] 1. The present invention realizes key point guided local feature extraction by adding key point guided information in the feature encoding middle layer, and combines the global feature self-perception method to extract rich detail information. At the same time, the local features of the key points, the local features and global features of the mesh vertices and the key point positions are input into the Transformer model to predict the mesh vertices and key point positions. It can predict an accurate human three-dimensional model from a single RGB image by combining the relationship between local features and global features, thereby realizing accurate three-dimensional human body reconstruction.
[0030] 2. The present invention further utilizes a linear hybrid skinning method to linearly combine the local features of the key points based on the local features related to the key points, thereby extracting a local feature for each mesh vertex. This can not only extract rich detail information, but also greatly reduce the amount of calculation compared to the traditional method of directly encoding local features for each vertex. There is no need to rely on various complex models, which can reduce the complexity of implementation and effectively improve efficiency and accuracy.
[0031] 3. The present invention further decouples key point information and vertex information in the lower layer by using a layered Transformer model, and fuses the encoded information again in the upper layer. Through the operation of first decoupling and then fusing, useful information between key points and mesh vertices can be fused for human body mesh prediction, thereby further improving the accuracy of three-dimensional human body model reconstruction.
[0032] 4. The present invention further performs masking based on prior information of human body segmentation, and fully considers the semantic information of key point tokens and mesh vertex tokens in the masking process. Compared with the traditional method of directly performing random masking on human key points and mesh vertices, it can match the characteristics of human body three-dimensional reconstruction and achieve more effective masking. When certain key points and corresponding mesh vertices are masked, the positions of the masked vertices can be inferred based on the local and global features of the input. Therefore, when the model is used, even if the hands or other parts of the human body are blocked in the input image, the blocked parts can still be effectively predicted, thereby effectively improving the anti-interference and flexibility of the reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 It is a schematic diagram of the implementation flow of the three-dimensional human body reconstruction method based on the Transformer model in this embodiment.
[0034] Figure 2 It is a schematic diagram of the implementation process of local feature extraction in this embodiment.
[0035] Figure 3 It is a schematic diagram of the implementation process of global feature extraction in this embodiment.
[0036] Figure 4 It is a schematic diagram of the structure of the Transformer model used in this embodiment.
[0037] Figure 5 It is a flowchart of implementing three-dimensional human body reconstruction based on the Transformer model in this embodiment.
[0038] Figure 6 It is a detailed schematic diagram of the implementation principle of three-dimensional human body reconstruction based on the Transformer model in this embodiment.
[0039] Figure 7 It is a detailed implementation flow chart of the three-dimensional human body reconstruction based on the Transformer model in this embodiment.
[0040] Figure 8 It is a schematic diagram of the reconstruction result obtained in a specific application embodiment of the present invention. DETAILED DESCRIPTION
[0041] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.
[0042] like Figures 1 to 7 As shown, the steps of the 3D human body reconstruction method based on the Transformer model in this embodiment include:
[0043] S01. Image feature extraction: Input a human RGB image, use a convolutional neural network to extract features from the input human RGB image, and obtain image features F.
[0044] Specifically, a convolutional neural network is used to extract features from the input RGB image of the human body. ResNet deep residual network can be used.
[0045] S02. Local feature extraction: Upsample the extracted image feature F to obtain the sampled feature F′, input the sampled feature F′ into a network composed of multiple superimposed residual blocks, and extract the local features of the key points; and generate a local feature related to the human body posture for each vertex in the mesh to obtain the local features of the mesh vertices.
[0046] After extracting the image feature F, upsampling is performed to obtain the feature The upsampling process can be achieved by stacking three layers of deconvolution operations, that is, the feature map output by ResNet is first upsampled through deconvolution. By introducing local features related to the positions of key points of the human body, detailed information can be extracted, so that the reconstructed human body model has detailed information and the reconstruction accuracy is improved.
[0047] like Figure 2 As shown, in this embodiment, the sampled feature F′ is input into a network with four superimposed residual blocks to encode the local features of the key points. That is, the residual block outputs a 24*1024 local feature, where 24 is the number of key points, and a local feature is extracted for each of the 24 key points. At the same time, the mixed skin weight in the SMPL template is used to transform the local feature (431*24) is linearly combined to generate an independent local feature related to human posture for each vertex in the mesh. The local features of the mesh vertices are obtained so that a local feature (431*1024) can be encoded for each mesh vertex, where 431 is the number of mesh vertices.
[0048] The local features of this embodiment are specifically to first upsample the feature map output by resnet through deconvolution, and then output the local features related to the key points through the residual block, and the local features of the encoded key points are linearly combined by using a linear mixed skinning method, so that a local feature is also extracted for each mesh vertex, which can not only extract rich detail information, but also greatly reduce the amount of calculation compared to the traditional method of directly encoding local features for each vertex, without relying on various complex models, can reduce the complexity of implementation, and can effectively improve efficiency and accuracy.
[0049] In the above-mentioned local feature extraction process, this embodiment also includes a process of regressing the key point position from the sampled feature F′, and extracting the local features of the key points using the key point error function supervision network, so that the obtained local features encode the key point position information, that is, by regressing the human body key points by predicting the heat map from F′, the feature F′ encodes the features related to the human body posture. By introducing the learning process of the key point error supervision network, a key point position is added to supervise the extraction of local features, so that the local features are highly related to the key point information, that is, the local features highly encode the key point position information, so that the correct key point position information can be extracted from the local features. Based on the accurate key point position information, the accuracy of human body reconstruction can be further ensured.
[0050] S03. Global feature extraction: Perform global feature extraction on the extracted image feature F.
[0051] like Figure 3 As shown, in this embodiment, the image feature F extracted by the convolutional neural network is subjected to maximum pooling processing and then input into the multi-layer perceptron MLP to encode the global feature The MLP consists of two fully connected layers with 1024 neurons in total, each followed by a dropout layer.
[0052] In this embodiment, both local features and global features are directly obtained from the input RGB image, and useful information in the image pixels will not be lost.
[0053] S04. Spatial position extraction: Use the image feature F to predict the parameterized human body model to obtain the human body mesh and the key point positions corresponding to the human body mesh.
[0054] Specifically, the SMPL parameterized human body model is predicted from the image feature F using the IEF (error feedback iteration) model through the SMPL parameters parameter You can get the human body mesh Then, the key point position corresponding to the human body mesh M0 can be obtained using the key point regression function f.
[0055] The above IEF module is specifically composed of two fully connected layers with 1024 neurons, each of which is connected to a dropout layer. At the same time, in order to reduce the amount of model calculation and to make the model better for training, the human body mesh M0 in this private model is downsampled from 6890 vertices to 431 vertices, and the θ parameter in the SMPL model uses the 6dof representation method.
[0056] S05. Transformer model reconstruction: The extracted local features, global features, human body mesh, and key point positions are spliced together, and the resulting feature vector sequence is input into the Transformer model to predict the final three-dimensional human body mesh vertices and key point positions to achieve three-dimensional human body reconstruction.
[0057] In this embodiment, the global features and local features are first spliced, and the spliced results are then spliced with the key point positions and the human body mesh to obtain a feature vector sequence. The feature vector sequence S0 includes the first vector sequence corresponding to the key point and a second sequence of vectors corresponding to the mesh vertices Finally, sequence S0 is used as the input of the Transformer model to predict the final 3D human body mesh vertices. And the key point location The specific input of the Transformer module is the extracted key point features (24*2048, 24 key points, each key point has 1024-dimensional local features and 2014-dimensional global features and mesh vertex features (431*2048).
[0058] Transformer can encode input sequences well, fully integrate input sequence information, and use the Transformer model to reconstruct the human body, which can reduce the complexity of implementation. This embodiment uses the above-mentioned characteristics of Transformer and uses the Transformer model to achieve human body reconstruction. The multi-head self-attention mechanism is used in the Transformer model to better encode the relationship between vertices in the mesh. By extracting the local features of key points, the local features and global features of mesh vertices, and the key point positions, the Transformer model is used to predict mesh vertices and key point positions. The Transformer model can combine the relationship between local features and global features to predict a more accurate three-dimensional human body model, thereby achieving accurate three-dimensional human body reconstruction.
[0059] The Transformer model of this embodiment adopts a layered Transformer. Each Transformer encoder is a layered Transformer model, specifically including two layers, each layer including a Transformer block. The lower layer is used to decouple key points and mesh vertices. The lower layer includes two Transformer blocks for encoding the key point input sequence k and the human mesh vertex input sequence v respectively, to obtain the encoded key points and mesh vertices; then, the key point input sequence and the human mesh vertex input sequence are concatenated in the upper layer as the input of the upper layer Transformer block, and the Transformer block in the upper layer fuses the information of the key points and mesh vertices. That is, in the lower layer, the relationship between key points and key points and the relationship between vertices and vertices are encoded for the key points and mesh vertices respectively through two Transformer blocks, and then the encoded key points and mesh vertices are used together as the input of the Transformer block in the upper layer, so that the useful information between the key points and mesh vertices can be fused for direct prediction of the human mesh.
[0060] This embodiment uses a hierarchical Transformer model to first decouple the key point information and the mesh vertex information and encode them separately, so as to learn the optimal information of the key point information and the mesh vertex information respectively. Then, the key point information and the mesh vertex information are simultaneously considered in the upper-level Transformer block, so as to effectively fuse the key point information and the mesh vertex information.
[0061] like Figure 4 , 5 As shown in the figure, in this embodiment, the feature vector sequence S0 is input into the Transformer Encoder through a simple linear mapping, and the output is directly the predicted mesh vertex. The Transformer encoder model is divided into two layers, each layer is implemented by a Transformer block, and each Transformer block contains two LN layers, a multi-head self-attention module (MHSA) and an MLP layer. The key points and mesh vertices are decoupled by the lower layer, and the input key point tokenk is respectively used by the Transformer block. l-1 and mesh vertex token v l-1 Encoding is performed, where l represents the lth layer Transformer encoder (in this embodiment, l is a maximum of 3). When l = 1, k0, v0 is obtained by converting the global feature vector F global Separate splicing At the same time, the key points and the position encoding of the mesh vertices are added. The key point token k decoupled from the lower layer is obtained in the upper layer l and mesh vertex token v l As the input of the Transformer block, the information of the key points and the grid is fused. In order to directly predict the 3D coordinates of the grid, each Transformer encoder is connected to a linear mapping layer to reduce the dimension of the features output by the Transformer encoder.
[0062] In the above step S04, the error feedback iteration model is specifically used to predict the parameterized human body model, and a three-dimensional human body model that is preliminarily aligned with the input human body image is predicted, and the human body mesh and the corresponding key point positions are obtained from the predicted three-dimensional human body model; in step S05, the human body mesh and the corresponding key point positions are used as the position coding information of the Transformer model, so that the position coding information is aligned with the input human body image. That is, the position coding used in the input token of the Transformer model is obtained by the IEF estimation in step S04. First, a three-dimensional human body model that is preliminarily aligned with the image is predicted through an error feedback iteration model, and then the spatial positions (x, y, z) of each key point and mesh vertex of the model are spliced as the Transformer position coding, so that the Transformer model position coding can be aligned with the input image.
[0063] The above step S05 also includes masking the key points and mesh vertices according to the segmentation information of the human body in the Transformer model, wherein the key points are randomly masked, and the mesh vertices of the human body language segmentation corresponding to the key points are masked. That is, the mesh vertices are first divided into 24 parts according to the correspondence with the key points. When masking, the key point tokens are first randomly masked, and then the corresponding mesh vertex tokens are masked according to the correspondence. For example, the key points of the left hand are masked first, and then all the mesh vertices corresponding to the left hand in the human body mesh vertices are masked accordingly.
[0064] In human body reconstruction, the human body structure has certain rules, that is, each key point corresponds to some mesh vertices. This embodiment performs masking based on prior information of human body segmentation, and fully considers the semantic information of key point tokens and mesh vertex tokens in the masking process. Compared with the traditional method of directly performing random masking on human body key points and mesh vertices, it can match the characteristics of human body three-dimensional reconstruction and achieve more effective masking. When certain key points and corresponding mesh vertices are masked, the position of the masked vertices can be inferred based on the local and global features of the input. Therefore, when the model is used, even if the hands or other parts of the person are blocked in the input image, these blocked parts can still be effectively predicted, thereby effectively improving the anti-interference and flexibility of reconstruction.
[0065] In a specific application embodiment, the deconvolution used in the above step S01 is shown in the decoder block part in Table 1, and the residual block configuration used in step S02 is shown in the stacked residual block part in Table 1.
[0066] Table 1 Deconvolution and residual block configuration parameters
[0067]
[0068] like Figure 6 , 7 As shown, this embodiment first extracts image features from the input image, and then extracts a global feature and a local feature, wherein the local features of the key points are output by firstly upsampling the features in the image and then outputting the local features of the key points through a specific residual block model, and then the local features of the human body model mesh are obtained by linearly combining the extracted key point features using linear mixed skinning weights (such as Figure 2 As shown in the figure), the global features are Figure 3 The fully connected network shown in the figure is used for extraction. At the same time, in order to make the extracted local features closely related to the reconstruction task, a specific key point supervision is added in the middle step of the entire feature extraction process; then the obtained global features are spliced with the local features of the key points and the local features of the mesh vertices to form a series of feature sequences, and the three-dimensional spatial positions of the key points and mesh vertices are used to respectively encode the positions of the corresponding feature sequences, and at the same time, an error feedback iterative model is used to predict a SMPL model that is roughly aligned with the image, so that the position encoding can be aligned with the input image, and then the predicted SMPL model is used to encode the position information of the feature sequence, and finally the feature sequence encoded with the position information is used as the input of the Transformer model, and finally the predicted three-dimensional human body model mesh and key point positions are obtained. The reconstruction effect obtained by using the above method in a specific application embodiment is as follows: Figure 8 As shown, Figure 8 (a) corresponds to the original image, (b) to (d) correspond to the reconstruction effects under different viewing angles.
[0069] The present invention realizes the local feature extraction guided by the key point by adding a key point guided information in the feature coding middle layer, and combines the global feature self-perception mode to extract rich detail information. At the same time, the local features of the key points, the local features of the mesh vertices, the global features and the key point positions are input into the Transformer model to predict the mesh vertices and the key point positions. It can combine the relationship between the local features and the global features to predict the accurate human body three-dimensional model from a single RGB image, thereby realizing accurate three-dimensional human body reconstruction. Further local feature encoding only needs to encode the key point features first, and then linearly combine them through the skin weights, so that a reasonable local feature can be encoded for each mesh. Since the number of key points is much smaller than the number of mesh vertices, the local feature encoding efficiency is higher than the traditional encoding method. Further, by using a layered Transformer model, the key point information and vertex information are decoupled in the lower layer, and the encoded information is fused again in the upper layer. Through the operation of first decoupling and then fusion, the useful information between the key points and the mesh vertices can be fused to predict the human body mesh, further improving the accuracy of three-dimensional human body model reconstruction.
[0070] The three-dimensional human body reconstruction device based on the Transformer model in this embodiment includes:
[0071] An image feature extraction module is used to input a human body image and use a convolutional neural network to extract features of the input human body image to obtain image features F;
[0072] A local feature extraction module is used to upsample the extracted image feature F to obtain a sampled feature F′, input the sampled feature F′ into a network composed of multiple residual blocks, extract the local features of the key points; and generate a local feature related to the human body posture for each vertex in the mesh to obtain the local features of the mesh vertex;
[0073] A global feature extraction module, used for performing global feature extraction on the extracted image feature F;
[0074] A spatial position extraction module is used to predict the parameterized human body model using the image feature F to obtain a human body mesh and key point positions corresponding to the human body mesh;
[0075] The Transformer model module is used to splice the extracted local features, global features with the human body mesh and key point positions. The feature vector sequence obtained after splicing is input into the Transformer model to predict the final three-dimensional human body mesh vertices and key point positions to achieve three-dimensional human body reconstruction.
[0076] The three-dimensional human body reconstruction device based on the Transformer model in this embodiment corresponds to the three-dimensional human body reconstruction method based on the Transformer model mentioned above, and will not be described in detail here.
[0077] The computer device of this embodiment includes a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program. The processor is used to execute the computer program to perform the above method.
[0078] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.
Claims
1. A three-dimensional human body reconstruction method based on a Transformer model, characterized in that the steps include: S01. Image feature extraction: Input a human body image, and use a convolutional neural network to extract features from the input human body image to obtain image features. ; S02. Local feature extraction: extracting image features Upsampling is performed to obtain the sampled features , the sampled features Input into a network composed of multiple residual blocks, extract the local features of key points; and generate a local feature related to human posture for each vertex in the mesh to obtain the local features of the mesh vertex; S03. Global feature extraction: extracting the image features Perform global feature extraction; S04. Spatial position extraction: using the image features Predicting a parameterized human body model to obtain a human body mesh and key point positions corresponding to the human body mesh; S05. Splicing the extracted local features, global features, human body mesh, and key point positions, and inputting the feature vector sequence obtained after splicing into the Transformer model to predict the final three-dimensional human body mesh vertices and key point positions to achieve three-dimensional human body reconstruction; The Transformer model includes two layers, each layer including a Transformer block. The lower layer is used to decouple key points and mesh vertices. The key points and human body mesh vertices are encoded using the Transformer block in the lower layer to obtain encoded key points and mesh vertices. The encoded key points and mesh vertices are used as inputs of the Transformer block in the upper layer, and the Transformer block in the upper layer fuses the information of the key points and mesh vertices. In the step S04, the parameterized human body model is predicted using an error feedback iterative model to predict a three-dimensional human body model that is preliminarily aligned with the input human body image, and the human body mesh and the corresponding key point positions are obtained from the predicted three-dimensional human body model; in the step S05, the human body mesh and the corresponding key point positions are used as position coding information of the Transformer model to align the position coding information with the input human body image.
2. The three-dimensional human body reconstruction method based on the Transformer model according to claim 1, characterized in that: In the step S02, the local features of the key points are linearly combined by linearly blending skin weights to obtain the local features of the mesh vertices.
3. The three-dimensional human body reconstruction method based on the Transformer model according to claim 1, characterized in that: In step S02, the method further includes: The key point positions are regressed, and a key point error function is used to supervise the network in extracting local features of the key points, so that the obtained local features encode the key point position information.
4. The three-dimensional human body reconstruction method based on the Transformer model according to claim 1, characterized in that: In step S03, the extracted image features After the maximum pooling process, the data is input into a multi-layer perceptron and encoded to obtain the global features.
5. The three-dimensional human body reconstruction method based on the Transformer model according to any one of claims 1 to 4, characterized in that: In step S05, the global features, the local features of the key points, and the local features of the mesh vertices are concatenated, and the feature vector sequence Contains the first vector sequence corresponding to the key point , using the spatial positions of key points and mesh vertices to respectively , Perform position encoding.
6. The three-dimensional human body reconstruction method based on the Transformer model according to any one of claims 1 to 4, characterized in that: The step S05 also includes masking the key points and mesh vertices in the Transformer model according to the segmentation information of the human body, wherein the key points are randomly masked, and the mesh vertices of the human body language segmentation corresponding to the key points are masked.
7. A 3D human body reconstruction system based on a Transformer model, characterized in that: include: The image feature extraction module is used to input a human body image and use a convolutional neural network to extract features from the input human body image to obtain image features. ; Local feature extraction module is used to extract image features Upsampling is performed to obtain the sampled features , the sampled features Input into a network composed of multiple residual blocks, extract the local features of key points; and generate a local feature related to human posture for each vertex in the mesh to obtain the local features of the mesh vertex; A global feature extraction module is used to extract the image features Perform global feature extraction; A spatial position extraction module is used to use the image features Predicting a parameterized human body model to obtain a human body mesh and key point positions corresponding to the human body mesh; The Transformer model module is used to splice the extracted local features, global features with the human body mesh and key point positions, and input the feature vector sequence obtained after splicing into the Transformer model to predict the final three-dimensional human body mesh vertices and key point positions to achieve three-dimensional human body reconstruction.
8. A computer device comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program, wherein: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 6.
