A heuristic three-dimensional human body mesh reconstruction method based on pose prior constraint
Patent Information
- Application Number
- CN202311449598.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-02
- Publication Date
- 2026-09-22
- Estimated Expiration
- 2043-11-02
AI Technical Summary
[0004]本发明的目的在于提供一种基于姿态先验约束的启发式三维人体网格重建方法，解决了现有技术中计算开销过多和模型收敛缓慢的问题，同时克服了现有技术中模型对人体特征的理解不精确的问题
[0037]本发明提供的一种基于姿态先验约束的启发式三维人体网格重建方法，提出了一种全新的基于先验约束的注意力标记和一种全新的启发式微调策略。首先，传统的方法之一是利用可学习的位置编码充当注意力令牌，另一种最近提出的方法是利用双向映射，通过线性映射将人体模板映射到高维空间，同时将输出的结果通过逆映射更新参数，保证了输入输出的语义一致性。与先前工作不同的是，本发明对注意力标记进行了两个改进：一个改进从特征图中提取综合先验约束向量，它将自我注意力集中在图像中人体的身体部分，避免了特征冗余。另一个改进将骨架信息合并到一个互逆映射层中，增强模型修正人体网格姿态能力的同时保证了输入输出的语义一致性。此外，本发明提取的区域先验约束通过将交叉注意力指向顶点周围的关键点来微调局部顶点。然后，在微调过程中，利用启发式微调策略确定当前查询顶点的最优偏移量。和传统基于Transformer的方法相比，本发明所提出的启发式Transformer有效地减少了交叉注意力查询数量，在保证模型轻量级的同时加速模型的收敛速度。本申请提高了单目三维人体重建网格的精度，在两个流行的数据集上比最近的方法取得了领先的性能。
Smart Images
Figure CN117372651B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of monocular 3D human body reconstruction technology, specifically involving a heuristic 3D human body mesh reconstruction method based on posture prior constraints. Background Technology
[0002] In the field of computer vision, human-like mesh reconstruction (HMR) from a single image is a popular research area. The goal is to estimate sparse skeletons or dense mesh models using visual signals. However, this task is challenging due to depth blur in monocular environments and occlusion caused by various human poses.
[0003] There are two main types of Human Morphological Mesh Reconstruction (HMR) methods: the first is model-based, characterized by regressing and calculating the shape and pose parameters of a multi-person skin linear model (SMPL). The second is model-free, characterized by directly regressing the 3D coordinates of all vertices. With the introduction of the Detection Transformer Framework (DETR), Transformer-based model-free HMR methods have recently become mainstream. End-to-end Deformable Human Mesh Reconstruction (METRO) was the first to use Transformer to solve the HMR problem. Subsequent work added graph convolution modules to METRO to improve local interactions between adjacent vertices and joints. Meanwhile, Fast End-to-End Deformable Human Mesh Reconstruction (FastMETRO) proposed a transformer encoder-decoder architecture and achieved good results. These methods directly input image feature information extracted from the backbone network into the self-attention module, using the Transformer mechanism to solve the problem of long-distance vertex interactions. However, the area covered by the human body in an image is relatively small. Directly using the features of the entire image for training leads to the inclusion of a lot of useless background information, resulting in inaccurate understanding of human features by the model. Meanwhile, to adjust the vertex coordinates of local parts of the human body, some models use cross-attention modules to detect local locations in the feature map. This is also susceptible to interference from local background information, and attention queries require traversing every feature point in the feature map, resulting in excessive computational overhead and slow model convergence. Summary of the Invention
[0004] The purpose of this invention is to provide a heuristic 3D human body mesh reconstruction method based on posture prior constraints, which solves the problems of excessive computational overhead and slow model convergence in the prior art, and also overcomes the problem of inaccurate understanding of human body features in the prior art.
[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0006] A heuristic 3D human body mesh reconstruction method based on attitude prior constraints, comprising:
[0007] A single RGB image is extracted and input into the comprehensive pose prior generator. Multi-scale feature maps and high-dimensional feature maps are extracted through the backbone network in the comprehensive pose prior generator. The comprehensive prior module in the comprehensive pose prior generator is used to output the spatial confidence matrix and comprehensive prior constraint vector of the high-dimensional feature map.
[0008] The multi-scale feature map is input into the query region partitioner to obtain the region prior constraint list, and the constrained region is generated according to the region prior constraint list.
[0009] A heuristic Transformer encoder is introduced, which includes a reciprocal mapping layer, a multi-head self-attention module, a multi-head cross-attention module, and an upsampling module;
[0010] The human template mesh is transformed to a high-dimensional space through forward mapping in the inverse mapping layer. The mapped human template mesh is then fused with the comprehensive prior constraint vector and the spatial confidence matrix to obtain the attention label.
[0011] The attention markers are input into the multi-head self-attention module to reconstruct the vertices of the global human template mesh, generating a preliminary target network;
[0012] The initial target network is input into the multi-head cross-attention module. Based on the constraint region, the multi-head cross-attention module performs heuristic fine-tuning on the vertices of the local human body template mesh to obtain the reconstructed human body output mesh.
[0013] The human body output mesh is reverse mapped through the inverse mapping layer, and the output of the inverse mapping is then passed through the upsampling module to obtain the final reconstructed 3D human body mesh.
[0014] Several alternative methods are provided below, but they are not intended as additional limitations on the overall solution above. They are merely further additions or optimizations. Provided there are no technical or logical contradictions, each alternative method can be combined individually with respect to the overall solution above, or multiple alternative methods can be combined with each other.
[0015] Preferably, the backbone network is a convolutional neural network, and the extraction of multi-scale feature maps and high-dimensional feature maps through the backbone network in the integrated pose prior generator includes:
[0016] A single RGB image is input into the backbone network, which then outputs multi-scale feature maps. and high-dimensional feature maps Where n represents the scale of the multi-scale feature map, i represents the i-th scale, and f i Let C represent the feature map at the i-th scale. i H represents the number of channels in the i-th scale feature map. i and W i Let C represent the height and width of the i-th scale feature map, and let C, H, and W represent the number of channels, height, and width of the high-dimensional feature map, respectively.
[0017] Preferably, the comprehensive prior module includes a 1×1 convolutional layer, linear dimensionality reduction, and a softmax operation. The step of using the comprehensive prior module in the comprehensive pose prior generator to output a spatial confidence matrix and a comprehensive prior constraint vector from the high-dimensional feature map includes:
[0018] High-dimensional feature maps Input a 1×1 convolutional layer to extract multi-channel spatial pose features. H represents the height of the high-dimensional feature map and the multi-channel spatial pose feature, W represents the width of the high-dimensional feature map and the multi-channel spatial pose feature, C represents the number of channels of the high-dimensional feature map, and V+J represents the number of channels of the multi-channel spatial pose feature, where V and J represent the number of vertices and the number of joints of the human body mesh, respectively.
[0019] Multi-channel spatial attitude features Flattening and performing linear dimensionality reduction yields the comprehensive prior constraint vector. Where D1 is the dimension of the comprehensive prior constraint vector;
[0020] Multi-channel spatial attitude features Performing the Softmax operation yields the spatial confidence matrix.
[0021] Preferably, the step of inputting the multi-scale feature map into the query region partitioner to extract the region prior constraint list and constraint region includes:
[0022] Based on the obtained multi-scale feature maps After encoding using a scale-corresponding multi-scale encoder, a multi-scale feature map with a uniform number of channels is obtained. Where n represents the scale, i represents the i-th scale, and f i Y represents the feature map at the i-th scale. i Let C represent the scaled feature map after encoding the i-th element. i N represents the number of channels, H i and W i Indicates height and width;
[0023] Flatten the encoded multi-scale feature map and store it in the region prior constraint list, and record the position of each feature map in the region prior constraint list.
[0024] Take the side length K of the preset query box, and determine the constrained region based on the region prior constraint list.
[0025] Preferably, the process of transforming the human body template mesh to a high-dimensional space through forward mapping in the reciprocal mapping layer, and fusing the mapped human body template mesh with the comprehensive prior constraint vector and spatial confidence matrix to obtain attention labels includes:
[0026] 3D human body template mesh The high-dimensional human body template mesh is converted through forward mapping in the reciprocal mapping layer. D2 is the dimension of the high-dimensional human body template mesh, and V+J represents the number of channels, where V and J represent the number of vertices and joints of the human body mesh, respectively.
[0027] High-dimensional human body template mesh With the integrated prior constraint vector The mesh is then spliced together, where D1 is the dimension of the combined prior constraint vector.
[0028] The stitched grid and spatial confidence matrix The attention labels are obtained by applying the Hadamard product operation. D1+D2 represents the dimension of the attention label.
[0029] Preferably, the input to the multi-head self-attention module includes a keypoint query sequence.
[0030] The shape of each joint is as follows: This represents the Jth query node;
[0031] The input to the multi-head self-attention module also includes a vertex query sequence. The shape of each vertex is: Let V be the Vth query vertex, D1+D2 be the dimension, and V and J represent the number of vertices and the number of joints in the human body mesh, respectively.
[0032] The multi-head self-attention module uses the self-attention module to perform vertex queries and complete global attention interaction on the vertices of the human template mesh; at the same time, it uses key point queries to calculate auxiliary loss during training to correct the pose of the human template network.
[0033] Preferably, the heuristic fine-tuning strategy adopted by the heuristic cross-attention module when performing heuristic fine-tuning is as follows:
[0034] F refine =Q refine +ΔH refine
[0035] Among them, F refine For heuristic fine-tuning functions, Q refine Indicates from the query vertex To the current vertex The offset cost, k∈[1,V], m∈[1,V], ΔH refine Indicates starting from the current vertex To the next nearest vertex The optimal offset;
[0036] Based on the constrained region, use the current vertex. As the center of the constrained region, and using the probe to predict relative to the current vertex. The offset is used to obtain the optimal offset ΔH within the constrained region. refine .
[0037] This invention provides a heuristic 3D human mesh reconstruction method based on pose prior constraints, proposing a novel attention token based on prior constraints and a novel heuristic fine-tuning strategy. First, one traditional method uses learnable positional encoding as attention tokens; another recently proposed method utilizes bidirectional mapping, mapping the human template to a high-dimensional space via linear mapping, while updating parameters through inverse mapping of the output, ensuring semantic consistency between input and output. Unlike previous work, this invention improves the attention token in two ways: one improvement extracts a comprehensive prior constraint vector from the feature map, focusing self-attention on the body parts of the human in the image, avoiding feature redundancy; the other improvement incorporates skeleton information into a reciprocal mapping layer, enhancing the model's ability to correct human mesh pose while maintaining semantic consistency between input and output. Furthermore, the region prior constraints extracted in this invention fine-tune local vertices by directing cross-attention to keypoints around the vertices. Then, during fine-tuning, a heuristic fine-tuning strategy is used to determine the optimal offset of the currently queried vertex. Compared to traditional Transformer-based methods, the heuristic Transformer proposed in this invention effectively reduces the number of cross-attention queries, accelerating the model's convergence speed while maintaining its lightweight nature. This application improves the accuracy of monocular 3D human body reconstruction meshes and achieves leading performance over recent methods on two popular datasets. Attached Figure Description
[0038] Figure 1This is a flowchart of the heuristic 3D human body mesh reconstruction method based on posture prior constraints proposed in this application;
[0039] Figure 2 This is a schematic diagram illustrating the process of dividing the query area and the local fine-tuning process in this application;
[0040] Figure 3 This is a schematic diagram comparing the attention tagging based on comprehensive prior constraints in this application with two traditional generation methods. Detailed Implementation
[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.
[0043] In one embodiment, such as Figure 1 As shown, a heuristic 3D human body mesh reconstruction method based on posture prior constraints is proposed, including:
[0044] Step S1: Extract a single RGB image and input it into the comprehensive pose prior generator. Extract multi-scale feature maps and high-dimensional feature maps through the backbone network in the comprehensive pose prior generator, and use the comprehensive prior module in the comprehensive pose prior generator to output the spatial confidence matrix and comprehensive prior constraint vector of the high-dimensional feature map.
[0045] The backbone network used in this embodiment is a convolutional neural network, which can be a ResNet network or an HrNet network. In this embodiment, an HrNet network is used to extract multi-scale feature maps from a single RGB image. and high-dimensional feature maps Where n represents the scale of the feature map, i represents the i-th scale, and f i Let C represent the feature map at the i-th scale. i H represents the number of channels in the i-th scale feature map. i and W i Let C represent the height and width of the i-th scale feature map, and let C, H, and W represent the number of channels, height, and width of the high-dimensional feature map, respectively.
[0046] The comprehensive prior module of this embodiment comprises a 1×1 convolutional layer, a linear dimensionality reduction operation and a Softmax operation. After obtaining the high-dimensional feature map, the high-dimensional feature map is input to the 1×1 convolutional layer to extract multi-channel spatial pose features at this time, the number of channels has been changed from C to V+J, where V and J represent 431 human vertices and 14 human joint points respectively.
[0047] Subsequently, the multi-channel spatial pose features are flattened, and a linear dimensionality reduction operation is performed thereon to obtain a comprehensive prior constraint vector wherein the dimension after dimensionality reduction becomes D1. Meanwhile, for the multi-channel spatial pose features, a Softmax operation is performed to obtain a spatial confidence matrix the spatial confidence matrix retains the spatial position information of vertices.
[0048] The comprehensive prior module adjusts the number of output channels to 431 dimensions, and each number of channels represents a feature of each human coordinate point. It should be noted that, when one letter represents one characteristic, it represents the characteristic of the corresponding feature or matrix on different features or matrices. For example, V+J represents the number of channels, so when referring to the multi-channel spatial pose features it represents the number of channels of the multi-channel spatial pose features, and in the spatial confidence matrix it represents the number of channels of the spatial confidence matrix.
[0049] Step S2: inputting the obtained multi-scale feature maps into a constrained region divider to obtain a region prior constraint list L feature and a constrained region A, as Figure 2 shown.
[0050] The present application subjects the multi-scale feature maps obtained in step S1 to respective multi-scale encoders (during design, the input parameters of the encoders are determined by the size of the feature maps) to obtain encoded multi-scale feature maps wherein after passing through the encoders of respective scales, the number of channels of the feature maps of each scale is unified, which is changed from C i to N uniformly.
[0051] Subsequently, these multi-scale feature maps are flattened and stored in the region prior constraint list L feature , and the position subscript corresponding to each feature map is recorded.
[0052] Subsequently, the side length K of a query box is determined to ensure that each attention query is kept within the specified constrained region K is a hyperparameter; setting different K values achieves different levels of accuracy. The region prior constraint list is the state in which the computer stores these prior constraints. When the model queries, it is equivalent to defining a K×K range within the N×N feature map, and attention interactions are performed within this range.
[0053] This example differs from the traditional method (FastMETRO) in that, in order to expand the receptive field, the traditional method utilizes depthwise convolutions to obtain feature maps at a smaller scale. However, this method also captures a large amount of unnecessary information from the surrounding areas. In contrast, the constrained region divider in this example restricts the attention of the cross-attention module to a small subset of sampling points around the vertices. Therefore, this significantly reduces the number of subsequent attention queries required.
[0054] Step S3: Introduce a heuristic Transformer encoder, which includes a reciprocal mapping layer, a multi-head self-attention module, a multi-head cross-attention module, and an upsampling module. Input the spatial confidence matrix generated in Steps 1 and 2, the integrated prior constraint vector, and the constraint region into the heuristic Transformer encoder to obtain the final reconstructed 3D human body mesh.
[0055] Step S301: Use linear mapping to map the human body template mesh. The transformation is mapped to a high-dimensional space with dimension D². Next, the transformed template mesh is concatenated with the integrated prior constraint vector X. Furthermore, the Hadamard product operation is applied between the concatenated mesh and the spatial confidence matrix S to obtain the attention labels. The entire calculation process can be represented as:
[0056] Z = Hadamard(Concat(T) V+J W+b,X),S)
[0057] in and Let W represent the weights and biases of the forward mapping in the linear mapping, respectively. Then, the right inverse matrix of W is used in the linear mapping... and transpose matrix W T A reverse mapping is established, specifically represented as follows:
[0058]
[0059] The attention tags generated in this example Each channel represents a feature of one of the human body's vertices, which means that all the trained features are related to human body features.
[0060] like Figure 3As shown, one traditional method for calculating attention tokens is to use learnable positional codes as attention tokens. Another recently proposed method uses bidirectional mapping, which maps the human template to a high-dimensional space through linear mapping, while updating parameters through inverse mapping of the output, ensuring semantic consistency between input and output. Compared with traditional feature extraction methods, this invention filters out redundant background information, allowing the model to avoid wasting time and computing power.
[0061] Step S302: The attention tags obtained in step S301 are... Input a multi-head self-attention module to perform global attention interactions on human vertices. The input to multi-head self-attention is a sequence of keypoint query sequences. The shape of each joint is as follows: This represents the first query node; the others can be understood similarly. The query sequence for the other vertex is... The shape of each vertex is The first vertex is queried, and the others are similarly understood. Vertex queries utilize a self-attention module to construct global interactions between mesh vertices. Simultaneously, keypoint queries gradually modify the pose of the humanoid mesh during training. Ultimately, the human's mesh is gradually transformed into the initial target mesh.
[0062] Step S303: Input the preliminary target mesh generated in step S302 into the multi-head cross-attention module, and use a heuristic fine-tuning strategy to complete the final fine-tuning of local vertices. Fine-tuning function F refine It can be represented as:
[0063] F refine =Q refine +ΔH refine
[0064] Among them, Q refine Indicates from the query vertex To the current vertex The offset cost, where index k represents one of the 431 query vertices, and m represents the offset index of that vertex. ΔH refine Indicates starting from the current vertex To the next nearest vertex The optimal offset.
[0065] When obtaining the optimal offset, the constrained region A obtained in step S2 is used, along with the currently queried vertex. As the center of the constraint region A, and predicting relative to the probe... The offset. In each iteration, Interact with surrounding vertices to obtain attention weights and optimal offset ΔHrefine Then, add the current offset cost Q. refine This yields an estimate that assesses whether offset optimization should be performed.
[0066] Next, after fine-tuning all vertices and performing dimensionality reduction, the final human body output mesh was obtained. It should be noted that the network structures of the multi-head self-attention module and the multi-head cross-attention module in this embodiment are conventional. The corresponding inputs and outputs and the heuristic fine-tuning strategies used are specific to this invention. Therefore, the conventional network structures will not be described in detail.
[0067] Step S304: The output mesh obtained in step S303 The inverse mapping established in step S301 is used to obtain the coarse mesh of the human body.
[0068]
[0069] The reverse mapping process updates the weights in the forward mapping based on factors such as the loss function. and bias This embodiment combines the human body template vertices with skeleton vertex information through a reciprocal mapping layer, increasing the dimension of each vertex through forward linear mapping. The output human body mesh M is then reverse-mapped to predict the 3D coordinates of each vertex, forming a coarse human body mesh description. The weight matrix and bias vector in the forward mapping are updated synchronously to ensure semantic consistency between input and output.
[0070] Next, apply the coarse mesh to the human body. The upsampling module is used to obtain the final reconstructed 3D human body mesh.
[0071] Experimental data from this application demonstrate that the method described herein achieves better reconstruction accuracy compared to other existing methods. The experimental data are shown in Tables 1 and 2 below:
[0072] Table 1 shows the performance of our method compared to other methods on the Human 3.6M dataset.
[0073]
[0074] Table 1 compares the performance of our method with other methods on the Human3.6M dataset. Experimental results are reported using two metrics: MPJPE (mean error per joint position) and PA-MPJPE (mean error after rigid alignment of predicted joints with ground truth joints). Lower prediction accuracy indicates higher reconstruction accuracy. Our method achieves state-of-the-art results on the Human3.6M dataset, reducing MPJPE by 3.2 and PA-MPJPE by 1.7 compared to the current state-of-the-art method (PointHMR). Furthermore, compared to the baseline method, MPJPE accuracy is reduced by as much as 13.6%. These experimental results demonstrate that our method can more effectively utilize human prior constraints to achieve higher-precision human mesh reconstruction.
[0075] Table 2 shows the performance of the proposed method compared to other methods on the 3DPW dataset.
[0076]
[0077] As shown in Table 2, the 3DPW dataset is a dataset with a smaller amount of data, with the number of training images less than 10% of Human3.6M. The newly added metric MPVE represents the average error per vertex, and total params represents the number of parameters required by the entire model. The experimental results show that the proposed method achieves the best accuracy in both MPJPE and MPVE metrics. Specifically, in terms of PA-MPJPE, the proposed method differs from the best method (FastMETRO) by only 0.3, but the number of parameters required by the proposed method is reduced by as much as 34.6% compared to FastMETRO. These results demonstrate that even on datasets with smaller data volumes, the proposed method achieves the best mesh reconstruction results with fewer parameters.
[0078] The full names corresponding to the abbreviations of other existing methods in Tables 1 and 2 are as follows: DSR stands for differential semantic rendering; METRO stands for mesh reconstruction with transformers; Graphormer stands for Mesh graphormer; FastMETRO stands for fast differential semantic rendering; and PointHMR is generally known as the point-guided three-dimensional human body reconstruction method.
[0079] This invention addresses the issues of feature redundancy and model inefficiency in 3D human body mesh reconstruction. The method of this invention obtains comprehensive prior constraints and region prior constraints regarding human pose, which serve as signals for the model throughout the entire human body mesh reconstruction process. To simplify feature information in the image, a multi-head self-attention module, guided by the comprehensive prior signal, focuses on the region where the human body is located. Similarly, a heuristic multi-head cross-attention module, guided by the region prior signal, focuses on key sampling points around vertices. Furthermore, after generating key query regions through the region prior signal, a heuristic fine-tuning strategy is proposed, which helps the model effectively refine the local human body mesh.
[0080] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0081] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.
Claims
1. A heuristic method for reconstructing 3D human body meshes based on posture prior constraints, characterized in that, The heuristic 3D human body mesh reconstruction method based on attitude prior constraints includes: A single RGB image is extracted and input into the comprehensive pose prior generator. Multi-scale feature maps and high-dimensional feature maps are extracted through the backbone network in the comprehensive pose prior generator. The comprehensive prior module in the comprehensive pose prior generator is used to output the spatial confidence matrix and comprehensive prior constraint vector of the high-dimensional feature map. The multi-scale feature map is input into the query region partitioner to obtain the region prior constraint list, and the constrained region is generated according to the region prior constraint list. A heuristic Transformer encoder is introduced, which includes a reciprocal mapping layer, a multi-head self-attention module, a multi-head cross-attention module, and an upsampling module; The human template mesh is transformed to a high-dimensional space through forward mapping in the inverse mapping layer. The mapped human template mesh is then fused with the comprehensive prior constraint vector and the spatial confidence matrix to obtain the attention label. The attention markers are input into the multi-head self-attention module to reconstruct the vertices of the global human template mesh, generating a preliminary target network; The initial target network is input into the multi-head cross-attention module. Based on the constraint region, the multi-head cross-attention module performs heuristic fine-tuning on the vertices of the local human body template mesh to obtain the reconstructed human body output mesh. The human body output mesh is reverse mapped through the inverse mapping layer, and the output of the inverse mapping is upsampled through the upsampling module to obtain the final reconstructed 3D human body mesh. The process of transforming the human body template mesh to a high-dimensional space through forward mapping in the reciprocal mapping layer, and then fusing the mapped human body template mesh with the comprehensive prior constraint vector and spatial confidence matrix to obtain attention labels includes: 3D human body template mesh The high-dimensional human body template mesh is converted through forward mapping in the reciprocal mapping layer. , The dimensions of the high-dimensional human body template mesh. Indicates the number of channels, where and These represent the number of vertices and the number of joints in the human body mesh, respectively. High-dimensional human body template mesh With the integrated prior constraint vector By piecing together the elements, we obtain the pieced-together mesh. To determine the dimension of the comprehensive prior constraint vector; The stitched grid and spatial confidence matrix The attention labels are obtained by applying the Hadamard product operation. , This indicates the dimension of the attention label.
2. The heuristic 3D human body mesh reconstruction method based on attitude prior constraints as described in claim 1, characterized in that, The backbone network is a convolutional neural network. The extraction of multi-scale feature maps and high-dimensional feature maps through the backbone network in the integrated pose prior generator includes: A single RGB image is input into the backbone network, which then outputs multi-scale feature maps. and high-dimensional feature maps ,in The scale of the multi-scale feature map is represented by... Indicates the first One scale, Indicates the first Each scale feature map Indicates the first The number of channels in a scale feature map. and Indicates the first The height and width of the feature map at each scale. , , These represent the number of channels, height, and width of the high-dimensional feature map, respectively.
3. The heuristic 3D human body mesh reconstruction method based on attitude prior constraints as described in claim 1, characterized in that, The comprehensive prior module includes a 1×1 convolutional layer, linear dimensionality reduction, and a softmax operation. The comprehensive prior module in the comprehensive pose prior generator outputs a spatial confidence matrix and a comprehensive prior constraint vector from the high-dimensional feature map, including: High-dimensional feature maps Input a 1×1 convolutional layer to extract multi-channel spatial pose features. , This represents the height of the high-dimensional feature map and multi-channel spatial pose features. The width represents the high-dimensional feature map and the multi-channel spatial pose features. This represents the number of channels in a high-dimensional feature map. The number of channels represents the multi-channel spatial pose feature, where and These represent the number of vertices and the number of joints in the human body mesh, respectively. Multi-channel spatial attitude features Flattening and performing linear dimensionality reduction yields the comprehensive prior constraint vector. ,in To determine the dimension of the comprehensive prior constraint vector; Multi-channel spatial attitude features Performing the Softmax operation yields the spatial confidence matrix. .
4. The heuristic 3D human body mesh reconstruction method based on attitude prior constraints as described in claim 1, characterized in that, The step of inputting the multi-scale feature map into the query region partitioner to extract the region prior constraint list and constraint region includes: Based on the obtained multi-scale feature maps After encoding using a scale-corresponding multi-scale encoder, a multi-scale feature map with a uniform number of channels is obtained. ,in Indicates scale. Indicates the first One scale, Indicates the first Each scale feature map Indicates the first Each encoded scale feature map , Indicates the number of channels. and Indicates height and width; Flatten the encoded multi-scale feature map and store it in the region prior constraint list, and record the position of each feature map in the region prior constraint list. Get the side length of the preset query box Based on the list of prior constraints for the region, determine the constraint region. .
5. The heuristic 3D human body mesh reconstruction method based on attitude prior constraints as described in claim 1, characterized in that, The input to the multi-head self-attention module includes a keypoint query sequence. Each joint has the following shape: , Indicates the first One query key point; The input to the multi-head self-attention module also includes a vertex query sequence. Each vertex has the following shape: , For the first Query vertex As a dimension, and These represent the number of vertices and the number of joints in the human body mesh, respectively. The multi-head self-attention module uses the self-attention module to perform vertex queries and complete global attention interaction on the vertices of the human body template mesh; Meanwhile, during training, joint query is used to calculate auxiliary loss in order to correct the pose of the human template network.
6. The heuristic 3D human body mesh reconstruction method based on attitude prior constraints as described in claim 5, characterized in that, When performing heuristic fine-tuning, the heuristic fine-tuning strategy adopted by the heuristic cross-attention module is as follows: in, For heuristic fine-tuning functions, Indicates from the query vertex To the current vertex The offset cost, , , Indicates starting from the current vertex To the next nearest vertex The optimal offset; Based on the constrained region, use the current vertex. As the center of the constrained region, and using the probe to predict relative to the current vertex. The offset is used to obtain the optimal offset within the constrained region. .