A 3D human body model reconstruction method based on the Transformer decoder

Abstract

The present invention discloses a 3D human body model reconstruction method based on a Transformer decoder. The original RGB image is preprocessed and input into a convolutional neural network for feature extraction to obtain image grid features. The vertex coordinates of the template 3D human body model are input into a linear layer to obtain encoded template vertex features. The image grid features are incorporated into the vertex features of the current state, and finally transformed vertex features are formed. The transformed vertex features are upsampled through an upsampling layer, and the vertex features are decoded into the vertex coordinates of the finally predicted 3D human body model through an inverse linear layer. The present invention makes more full use of the local information in the image grid features, and at the same time imposes strict semantic consistency constraints on the vertex feature transformation process, helping the network to ignore unnecessary semantic conversions and focus on predicting a more accurate human body model.

Description

Technical Field

[0001] This application relates to the field of computer vision, and particularly to a 3D human model reconstruction method based on a Transformer decoder. Background Art

[0002] 3D human model reconstruction (Human Mesh Reconstruction, HMR) aims to predict the 3D human mesh model of the target human body through visual image signals, which is an important task in computer vision. Human model reconstruction provides rich information such as the pose and shape of the target human body for many downstream tasks, such as human action recognition and virtual fitting, etc., so it has received more and more attention from researchers in recent years. At the same time, there are still many challenges to be solved in this field, such as the high dynamicity of human joints, the ambiguity of 2D image to 3D prediction, occlusion, etc.

[0003] In recent years, 3D human model reconstruction methods based on RGB images can be mainly divided into two categories: model-based and model-free. The model-based methods regard the parameters of 3D human models such as SMPL as the prediction targets, and then input the predicted parameters into SMPL to obtain the final 3D human model. Among them, some methods iteratively optimize the model parameters from the initial state to match the 2D visual information, and some directly regress through neural networks to obtain the model parameters. However, due to the huge difference in the representation between 2D images and model parameters, in order to obtain accurate results, these methods generally require a large amount of optimization time or training data.

[0004] Although the model-based and iterative optimization methods are too time-consuming, these methods have created a large number of 3D human model pseudo-labels offline for many datasets without 3D annotations. Recently, with the help of these datasets with pseudo-labels, many methods have tried model-free methods to reconstruct 3D human models. These methods fix the 3D human model mesh topology and directly predict the mesh vertex coordinates through neural networks. These methods generally adopt network structures with strong local modeling capabilities such as convolutional neural networks and graph convolutional neural networks, but they cannot fully express the relationships between long-distance components or vertices of the human body.

[0005] A Transformer is a network structure based on the attention mechanism that can softly model the relationship between two sequences. Due to its powerful non-local interaction modeling ability, the Transformer has achieved excellent results in many recent visual tasks (such as image recognition, object detection, etc.). In the field of human model reconstruction, there are methods that extract image feature vectors through a convolutional neural network, obtain the features of each vertex through repeated operations, and use a Transformer encoder to regress the 3D coordinates of each vertex. However, the existing methods only model the internal relationship between vertices through the self-attention mechanism, do not construct a mapping from image features to vertex features with sufficient expressive power, and do not make full use of image features. Summary of the Invention

[0006] To solve the above problems, this application proposes a 3D human model reconstruction method based on a Transformer decoder, which can make full use of the information input by RGB images and predict an accurate target 3D human model.

[0007] To achieve the above purpose, the technical solution of this application is as follows:

[0008] A 3D human model reconstruction method based on a Transformer decoder, including:

[0009] Preprocess the original RGB image to obtain an image containing the target human body, and input it into the image feature extraction module for feature extraction to obtain image grid features;

[0010] Input the vertex coordinates of a template 3D human model into a linear layer to obtain encoded template vertex features;

[0011] Input the image grid features and the template vertex features into the vertex feature transformation module, and after fusion, obtain transformed vertex features;

[0012] Upsample the transformed vertex features through an upsampling module, and decode the upsampled vertex features into the vertex coordinates of the finally predicted 3D human model through an inverse linear layer.

[0013] Further, the image feature extraction module uses a ResNet network.

[0014] Further, the template 3D human model uses a genderless, standard pose SMPL human model.

[0015] Further, the template 3D human model is a low-resolution template 3D human model processed by a mesh downsampling algorithm.

[0016] Further, the vertex feature transformation module includes a preset number of Transformer decoders. The input of the first-layer Transformer decoder is the image grid feature and the template vertex feature, and the input of other Transformer decoders is the image grid feature and the vertex feature output by the previous-layer Transformer decoder.

[0017] Further, the Transformer decoder includes a self-attention layer, a cross-attention layer, and a feed-forward network layer connected in sequence. Normalization layers are respectively provided before the self-attention layer, the cross-attention layer, and the feed-forward network layer. The input of each layer of the self-attention layer, the cross-attention layer, and the feed-forward network layer is accumulated with the output of this layer and then input to the next layer; the input of the self-attention layer is the template vertex feature or the vertex feature output by the previous-layer Transformer decoder, and the input of the cross-attention layer further includes the image grid feature.

[0018] Further, the upsampling layer consists of two identical MLP networks. Each MLP consists of two linear layers and a non-linear activation layer therebetween, and converts the low-resolution vertex feature into a high-resolution vertex feature in stages.

[0019] Further, the linear layer and the inverse linear layer are strictly inverse to each other. The linear layer is expressed as:

[0020]

[0021] where represents the vertex coordinates of the template 3D human body model, z0 represents the template vertex feature, and W and b respectively represent the weight matrix and the bias vector of the linear layer;

[0022] The inverse linear layer is expressed as:

[0023] V 3D =(z V -b)W -1 ;

[0024] where, V 3D represents the vertex coordinates of the 3D human body model output by the inverse linear layer, z V represents the vertex feature output by the vertex feature transformation module, W -1 represents the generalized right inverse matrix of W, W -1 =W T (WW T ) -1 .

[0025] Further, the image feature extraction module, linear layer, vertex feature transformation module, upsampling module, and inverse linear layer constitute a network model. When the network model is trained, the loss function used is as follows:

[0026]

[0027] Among them, represents the loss of the vertex coordinates of the finally predicted 3D human body model, represents the loss of the coordinates of the predicted 3D human body key points, represents the loss of the coordinates of the predicted 2D human body key points; the predicted 3D human body key point coordinates and the predicted 2D human body key point coordinates are calculated from the vertex coordinates of the finally predicted 3D human body model, and λ1, λ2, and λ3 are hyperparameters.

[0028] A 3D human body model reconstruction method based on a Transformer decoder proposed in this application has the following beneficial effects:

[0029] 1) Using the cross-attention layer of the multi-stage Transformer decoder to construct a fusion mapping relationship from image grid features to vertex features. Compared with simple mapping relationships such as linear mapping, the method of the present invention can make more full use of the local information in the image features and improve the accuracy of the predicted human body model.

[0030] 2) Using a strictly reciprocal linear mapping layer to process the bidirectional mapping between vertex features and vertex coordinates, and at the same time applying the loss function to the outputs of all layers inside the vertex feature transformation module, ensuring the semantic consistency inside and on both sides of the vertex feature transformation module, and prompting the network to focus on learning more valuable information rather than paying attention to unnecessary semantic conversions.

[0031] 3) Providing a two-stage training method, and the additionally introduced model smoothing loss can make full use of the model capacity and representation ability of the upsampling module itself to help the network output a more smooth and real 3D human body model.

[0032] 4) The provided embodiments adopt a relatively lightweight network design. Compared with other methods in the field, the model of this application has a smaller capacity and faster speed while ensuring the prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flowchart of the 3D human body model reconstruction method based on the Transformer decoder of this application;

[0034] Figure 2 is a schematic diagram of the network model structure of this application;

[0035] Figure 3Schematic diagram of the vertex feature transformation module of the present application;

[0036] Figure 4 Schematic diagram of the upsampling module of an embodiment of the present application;

[0037] Figure 5 Schematic diagram of the linear layer and the inverse linear layer of an embodiment of the present invention. Detailed implementation manners

[0038] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be emphasized that the following described embodiments are only used to explain the present application, rather than to limit the scope and application of the present invention.

[0039] In one embodiment, as Figure 1 shown, a 3D human body model reconstruction method based on a Transformer decoder is provided, including:

[0040] Step S1: Preprocess the original RGB image to obtain an image containing the target human body, and input it into the image feature extraction module for feature extraction to obtain image grid features.

[0041] In this step, to extract the image grid features, first preprocess through the preprocessing module, and then extract the image grid features through the image feature extraction module.

[0042] As Figure 2 shown, the preprocessing module is responsible for cropping and occluding data augmentation of the input RGB picture to obtain an image that fully contains the target portrait. Among them, for the up-cropping process, a human body detection method can be used to estimate the rectangular bounding box in the original image where the target portrait is fully contained, and then crop the image accordingly. In one embodiment, models such as YOLO v3 can be used for human body detection. The detected rectangular bounding box will be expanded according to the longest side to form a square bounding box, and the original RGB picture will be cropped accordingly to obtain a square preprocessed picture, and the size will be adjusted to a resolution of 224×224. When the cropped area exceeds the boundary of the original image, the pixel intensity of the exceeded part is set to zero. Other methods can also be used for image cropping, which will not be elaborated here.

[0043] The above-mentioned occluding data augmentation can further superimpose noise on the cropped image, randomly combine the noise patterns of simple geometric figures with the instance object patterns taken from the segmentation dataset, and randomly occlude the local area of the target image to provide a wider range of training samples for the training process. In this example, in addition to the random geometric pattern noise, instance patterns unrelated to the human body in the Pascal VOC 2012 dataset can also be selected. When using the trained model to reconstruct the 3D human body model for the RGB image to be processed, only cropping is required.

[0044] The image feature extraction module is responsible for extracting features from the preprocessed image to obtain image grid features. In this example, the ResNet50 model is used to extract image features, and models such as ResNet101 and HRNet-W64 can also be used to extract image features. The resolution of the extracted image grid features is reduced to 1 / 32 of the original image, which is 7×7. At the same time, its spatial dimension is flattened to form an image grid feature sequence with a length of 7×7 = 49, and the spatial dimension of the image grid features is mapped to 512.

[0045] Step S2: Input the vertex coordinates of a template 3D human body model into the linear layer to obtain the encoded template vertex features.

[0046] In this step, the linear layer encodes the vertex coordinates of the template 3D human body model into corresponding vertex features. In this example, the template 3D human body model uses a gender-neutral, standard pose SMPL human body model, and is pre-compressed by a mesh model downsampling algorithm to compress the original 6980 vertices into a low-resolution template 3D human body model with 431 vertices.

[0047] Pre-compressing the mesh in advance can reduce the sequence length processed by the Transformer, thereby reducing the memory usage. In this example, the linear layer encodes the 3D vertex coordinate space into a 512-dimensional vertex feature space.

[0048] It should be noted that the linear layer and the inverse linear layer are often used for adjusting the feature space dimension and are relatively mature technologies, which will not be elaborated here.

[0049] Step S3: Input the image grid features and the template vertex features into the vertex feature transformation module, and after fusion, obtain the transformed vertex features.

[0050] In this step, the vertex feature transformation module includes a Transformer decoder with a preset number L of layers. As Figure 3 shown, the input of the first-layer Transformer decoder is the image grid features and the template vertex features, and the input of other Transformer decoders is the image grid features and the vertex features output by the previous layer of the Transformer decoder.

[0051] As Figure 3As shown, the Transformer decoder includes a self-attention layer, a cross-attention layer, and a feed-forward network layer connected in sequence. Normalization layers are respectively provided before the self-attention layer, the cross-attention layer, and the feed-forward network layer. The input of each layer of the self-attention layer, the cross-attention layer, and the feed-forward network layer is added to the output of this layer and then input to the next layer; the input of the self-attention layer is the template vertex feature or the vertex feature output by the previous layer of the Transformer decoder, and the input of the cross-attention layer further includes the image grid feature. The input of the cross-attention layer not only includes the image grid feature, but also includes the vertex feature after adding the output of the self-attention layer and the input of the self-attention layer.

[0052] The self-attention layer can model the association between vertices, and the cross-attention layer models the association between the image grid feature and the vertex feature. In each cross-attention layer of the vertex feature transformation module, the image grid feature and the vertex feature will be used as Key and Query respectively to calculate a cross-attention map, so as to softly extract the features concerned by the network in Value (image grid feature) accordingly, and map and superimpose them in the vertex feature space to form the transformed vertex feature.

[0053] Throughout the process, the length of the input vertex feature sequence remains unchanged from beginning to end, and the vertex feature dimension is the same as the image grid feature dimension.

[0054] The layer-by-layer and content-related feature transformation and fusion method of this application can make more full use of the local information in the image grid feature. In this example, the length of the vertex feature sequence remains the number of model vertices after downsampling, that is, 431, from beginning to end. There are L = 6 Transformer decoders with exactly the same structure in the vertex feature transformation module.

[0055] Step S4: Upsample the transformed vertex features through the upsampling module, and decode the upsampled vertex features into the vertex coordinates of the final predicted 3D human model through the inverse linear layer.

[0056] This step mainly processes the transformed vertex features through the upsampling module and the inverse linear layer.

[0057] The upsampling module is used to perform upsampling operations on the transformed vertex features to expand the low-resolution vertex features into high-resolution vertex features. In this example, as Figure 4As shown, the upsampling module consists of two identical MLP networks. Each MLP consists of two linear layers and a non-linear activation layer therebetween. The non-linear activation layer uses the ReLU activation function. The low-resolution vertex features are transformed into high-resolution vertex features in stages. The topological structure of the 3D human mesh model is completely fixed, and the vertex coordinates of the decoded 3D human model are also arranged in order, thus reconstructing the 3D human model.

[0058] The inverse linear layer is used to decode the high-resolution vertex features output by the upsampling module to high-resolution vertex coordinates. To ensure the semantic consistency of the vertex features, this inverse linear layer is strictly inverse to the linear layer in step S2. In this example, as Figure 5 shown, the linear layer of step S2 is first defined and can be expressed as:

[0059]

[0060] where represents the vertex coordinates of the template 3D human model, z0 represents the template vertex features, and W and b represent the weight matrix and bias vector of the linear layer respectively. Then, the inverse linear layer of step S4 shares the parameters of the linear layer in step S2 and can be expressed as:

[0061] V 3D =(z V -b)W -1 ;

[0062] where V 3D represents the vertex coordinates of the 3D human model output by the inverse linear layer, z V represents the vertex features output by the vertex feature transformation module, and W -1 represents the generalized right inverse matrix of W, which can be expressed as:

[0063] W -1 =W T (WW T ) -1 ;

[0064] Since W -1 is inverse to W, the above two linear layers are strictly inverse, and the initial input and final output of the entire method are three-dimensional vertex coordinates, thus ensuring the semantic consistency of the vertex features on both sides of the Transformer decoder. In this example, such a design can avoid the network focusing on unnecessary semantic conversions and instead focus on learning more valuable information.

[0065] In a specific embodiment, the image feature extraction module, linear layer, vertex feature transformation module, upsampling module, and inverse linear layer of the present application constitute the entire network. When training this network model, a combination of multiple loss terms is used to supervise the human body model output by the network.

[0066] In this embodiment, the loss function is as follows:

[0067]

[0068] Among them, represents the loss of the vertex coordinates of the finally predicted 3D human body model, represents the loss of the predicted 3D human body key point coordinates, represents the loss of the predicted 2D human body key point coordinates; the predicted 3D human body key point coordinates and the predicted 2D human body key point coordinates are calculated from the vertex coordinates of the finally predicted 3D human body model, and λ1, λ2, and λ3 are hyperparameters.

[0069] Specifically, calculate the L1 distance loss between the vertex coordinates of the finally predicted 3D human body model and the ground truth [[ID=2 O]]Calculate the L1 distance loss between the predicted 3D human body key point coordinates and the ground truth Calculate the L1 distance loss between the predicted 2D human body key point coordinates and the ground truth

[0070] The predicted 3D human body key point coordinates J 3D are obtained by multiplying the predicted 3D model vertex coordinates V 3D ] by a fixed key point regression matrix G:

[0071] J 3D = GV 3D

[0072] The predicted 2D human body key point coordinates J 2D are obtained by weak perspective projection of J 3D :

[0073] J 2D =(∏ x,y J 3D + t)s

[0074] where t and s are weak perspective projection camera parameters, representing the offset parameter and the scaling parameter respectively. The loss terms are defined as follows:

[0075]

[0076]

[0077]

[0078] wherein respectively represent the true values of the 3D human body model vertex coordinates, 3D human body key point coordinates, and 2D human body key point coordinates.

[0079] In another embodiment, the training is divided into two stages. In the first stage, the above loss function is used to train all trainable network parameters except the upsampling module. During the training of this stage, the upsampling module is replaced by a sampling matrix (i.e., a fixed linear interpolation module) generated by the model grid downsampling algorithm described in step S2. In the second stage, based on the foregoing loss function, an additional loss term is introduced to train only the upsampling module to improve its ability to output a smooth 3D human body model.

[0080] Specifically, the following additional loss terms are introduced:

[0081]

[0082]

[0083] where f is the number of any patch in the grid, i and j are the numbers of any two different vertices in the patch numbered f, and v and v GT respectively represent the predicted coordinates and true coordinates of a certain vertex, and n GT represents the normal direction vector of a certain patch. It should be noted that the topological structures of the human body models predicted in this application and the true human body model are exactly the same, both fixed as the SMPL model, and both have 6890 fixed-number vertices and 13776 fixed-number patches; the vertex coordinates output in step S4 are arranged in an orderly manner according to the preset vertex numbers, so a pair of vertices, edges, or patches with the same number in the prediction model and the true model correspond one by one. In the above two additional loss terms, measures the length difference between each pair of edges with the same number in the predicted human body model and the true human body model, measures the normal direction difference between each pair of patches with the same number in the predicted human body model and the true human body model. The total loss function after introducing the additional smoothing term is:

[0084]

[0085] where λ4 and λ5 are also hyperparameters.

[0086] In the above two examples, all loss terms act not only on the output of the Transformer decoder at the last layer (i.e., l = 6) of the vertex feature transformation module, but also on the outputs of the Transformer decoders at other layers (l ∈ {1, 2, 3, 4, 5}). Such a training strategy forces the vertex features output by the internal layers of the vertex feature transformation module to have the same semantics as the final output, further ensuring the semantic consistency of the vertex features; macroscopically, the entire network gradually transforms the vertex features of the template human model into the output vertex features layer by layer with the help of the image grid features, and finally decodes them into the output human model vertex coordinates.

[0087] The above embodiments are further detailed descriptions of the present invention in combination with specific implementation manners, but it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. It should be noted that for those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, several alternatives or modifications can be made to these described implementation manners, and these alternative or modified manners should all be regarded as belonging to the protection scope of the present invention. Therefore, the protection scope of the patent application of the present application shall be subject to the appended claims.

Claims

1. A 3D human body model reconstruction method based on Transformer decoder, characterized in that: The 3D human body model reconstruction method based on the Transformer decoder comprises: Preprocess the original RGB image to obtain an image containing the target human body, and input it into the image feature extraction module for feature extraction to obtain image grid features; Input the vertex coordinates of a template 3D human body model into the linear layer to obtain the encoded template vertex features; The image mesh features and template vertex features are input into the vertex feature transformation module, and the transformed vertex features are obtained after fusion; The transformed vertex features are upsampled through the upsampling module, and the upsampled vertex features are decoded into the final predicted 3D human body model vertex coordinates through the inverse linear layer; The linear layer and the inverse linear layer are strictly inverse to each other, and the linear layer is expressed as: in, represents the vertex coordinates of the template 3D human body model, z0 represents the template vertex features, W and b represent the weight matrix and bias vector of the linear layer respectively; The inverse linear layer is expressed as: V 3D =(z V -b)W -1 ; Among them, V 3D Represents the vertex coordinates of the 3D human body model output by the inverse linear layer, z V Represents the vertex features output by the vertex feature transformation module, W -1 represents the generalized right inverse matrix of W, W -1 =W T (WW T ) -1 .

2. The 3D human body model reconstruction method based on Transformer decoder according to claim 1, characterized in that: The image feature extraction module uses the ResNet network.

3. The 3D human body model reconstruction method based on Transformer decoder according to claim 1, characterized in that The template 3D human body model uses a gender-neutral, standard-pose SMPL human body model.

4. The 3D human body model reconstruction method based on Transformer decoder according to claim 3, characterized in that: The template 3D human body model is a low-resolution template 3D human body model processed by a grid downsampling algorithm.

5. The 3D human body model reconstruction method based on Transformer decoder according to claim 1, characterized in that: The vertex feature transformation module includes a preset number of Transformer decoders. The input of the first-layer Transformer decoder is the image mesh features and the template vertex features. The input of other Transformer decoders is the image mesh features and the vertex features output by the previous layer Transformer decoder.

6. The 3D human body model reconstruction method based on Transformer decoder according to claim 5, characterized in that: The Transformer decoder includes a self-attention layer, a cross-attention layer and a feedforward network layer connected in sequence. The self-attention layer, the cross-attention layer and the feedforward network layer are respectively provided with their own normalization layers. The input of each layer of the self-attention layer, the cross-attention layer and the feedforward network layer is accumulated with the output of the current layer and then input to the next layer; the input of the self-attention layer is the template vertex feature or the vertex feature output by the previous layer of the Transformer decoder, and the input of the cross-attention layer also includes the image grid feature.

7. The 3D human body model reconstruction method based on Transformer decoder according to claim 1, characterized in that: The upsampling module consists of two identical MLP networks, each of which consists of two linear layers and a nonlinear activation layer in between, which converts low-resolution vertex features into high-resolution vertex features in stages.

8. The 3D human body model reconstruction method based on Transformer decoder according to claim 1, characterized in that: The image feature extraction module, linear layer, vertex feature transformation module, upsampling module and inverse linear layer constitute a network model. When training the network model, the loss function used is as follows: in, represents the final predicted 3D human body model vertex coordinate loss, represents the predicted 3D human key point coordinate loss, Represents the predicted 2D human body key point coordinate loss; the predicted 3D human body key point coordinates and the predicted 2D human body key point coordinates are calculated from the final predicted 3D human body model vertex coordinates, and λ1, λ2, and λ3 are hyperparameters.

Images