A hand-drawing-based three-dimensional human body reconstruction method

Abstract

The embodiment of the application discloses a kind of three-dimensional human reconstruction methods based on hand drawing, including making a new dataset Sketch3D, mapping relationship is established between human hand sketch and three-dimensional model, using the end-to-end method based on convolutional neural network combined with SMPL model, according to the characteristics of hand drawing, our network is reformed, and a new training method is created: step by step training (step by step training, SST).Using the application, the ability of feature extraction of the model to hand-drawn images is improved by creating a multi-channel branch attention network, the function of reconstructing a three-dimensional human model from a hand-drawn human sketch is achieved, and this function is not only suitable for users with a drawing basis, but also suitable for users without a drawing basis.The efficiency of three-dimensional human modeling is greatly improved.

Description

Technical Field

[0001] This invention relates to a three-dimensional reconstruction method, and more particularly to a three-dimensional human body reconstruction method based on hand-drawing. Background Technology

[0002] The history of 3D reconstruction can be traced back to the origins of computer graphics, with the earliest 3D reconstruction techniques proposed in the 1960s and 70s. However, because the initial methods relied heavily on scanners and cameras to detect real-world objects, they were extremely labor-intensive and resource-intensive, making them unsuitable for the general public.

[0003] The earliest 3D reconstruction technology was proposed by Ivan Sutherland[1], a pioneer of computer graphics in the United States. In the following decades, with the development of computers and the progress of hardware devices, especially the emergence of GPUs, the 3D reconstruction technology has developed rapidly. David Marr proposed the concept of "visual intelligence" and put forward a theoretical framework for extracting 3D information from images. Marc Levoy et al. proposed a method to scan the surface of a real object using a laser scanner and reconstruct a 3D model, which is one of the earliest 3D reconstruction methods based on real data. Chen and Medioni proposed a 3D reconstruction method based on visual geometry, which uses images from multiple perspectives for reconstruction. After the 20th century, due to the emergence of GPUs, 3D reconstruction technology has been developed and popularized. Among them, the most representative methods include: point cloud-based reconstruction methods, stereo vision-based reconstruction methods, light field-based reconstruction methods, image-based reconstruction methods, etc.

[0004] However, the development of 3D human reconstruction methods based on hand-drawn sketches is still very limited. Most of these methods use neural networks to predict the mesh vertices of the human model from a hand-drawn sketch. Brodt and Bessmeltsev transferred the sketch pose to a standard 3D human model by predicting three key factors (2D bone tangents, self-contact, and pre-shortening), which can directly obtain a standard 3D human mesh from a hand-drawn sketch.

[0005] Existing 3D human reconstruction techniques based on hand-drawn sketches still rely on annotating joints and self-connected regions within images. This annotation process is extremely resource-intensive, and creating a large dataset is very difficult. Furthermore, considering the abstract and random nature of hand-drawn sketches, previous methods cannot extract crucial information about the human body from sketches of any style. This approach is fundamentally a 3D reconstruction based on single-view images, but directly applying current methods to sketches still presents many problems.

[0006] (1) Previous networks were designed for RGB images with rich image features. Users needed to take a photo in a specific pose and then input the photo into the network to obtain the reconstructed 3D human body. This method seems convenient, but if a large number of human body models with different poses are needed, a lot of posing and taking photos is required. Some of these actions can only be performed by professionals, which causes great inconvenience.

[0007] (2) As an abstract and sparse form of expression, sketches not only result in messy lines and disproportionate shapes, but also the style and level of detail of sketches drawn by different users are unpredictable. Image features extracted using previous methods may not be able to obtain key information about the sketch.

[0008] (3) The information contained in the human body includes posture, shape, and position of the human body in the image. Previous methods used a single branch to directly predict these three parameters of the human body. This approach causes the posture, shape, and projection position in the sketch to affect each other. Simple sketches tend to pass posture information to the 3D model, which causes shape information to also have some influence on posture. More importantly, hand-drawn sketches do not mark the positions of 2D joints when they are drawn. Using the previous methods, we cannot obtain the accurate projection of the 3D model onto the human body in 2D space. Therefore, recognizing the specific information expressed by the sketch is a very challenging task for computers.

[0009] (4) The reconstruction methods based on hand-drawn sketches that have emerged so far still use networks based on real images and rely on a large amount of data labeling, such as sketch joint annotations and sketch self-contact annotations, which makes the workload of obtaining large datasets huge. Summary of the Invention

[0010] The technical problem to be solved by the embodiments of the present invention is to provide a method for reconstructing a three-dimensional human body based on a hand-drawn sketch. A three-dimensional human body model can be reconstructed from a hand-drawn sketch.

[0011] To address the aforementioned technical problems, this invention provides a method for reconstructing a three-dimensional human body based on hand-drawing, comprising the following steps:

[0012] S1: Create a dataset corresponding to the hand-drawn sketches and 3D human body models. The dataset includes composite images, hand-drawn images, and corresponding 3D human body models.

[0013] S2: Use ResNet50 as the encoder of the convolutional neural network to allow information to flow directly from the input to the output.

[0014] The decoder structure is designed as a multi-branch structure, including a pose decoder, a shape decoder, and a projection position decoder. Each decoder is composed of an attention module plus a stack of linear layers. The special tensor vector extracted by the convolutional neural network is optimized for attention through different branches.

[0015] For the motion-based decoding method used in the decoder, the feature enhancement part of the decoder, including the pose decoder, shape decoder, and projection position decoder, all employ a self-attention model to enhance the feature vectors obtained from the encoder, optimizing the feature vectors obtained from the encoder. Subsequently, based on the different dimensions of the pose, shape, and projection parameters, the optimized feature vectors are dimensionally scaled.

[0016] S3: The two-dimensional joint data of the synthetic data of the dataset used for pre-training are also used as weak supervision in the training process. The SMPLify method is used for optimization, and the incorrectly labeled three-dimensional targets in the dataset are corrected. Hand-drawn sketch data is used for stepwise fine-tuning training to fine-tune the pose.

[0017] Specifically, S1 includes the following steps:

[0018] S11: Obtain two-dimensional joint information of the sketch using two-dimensional projection parameters trained with a synthetic dataset;

[0019] S12: Obtain the position information of the human body in the sketchboard from the coordinates of the two joints.

[0020] S12 further includes the following steps: using a preprocessing program, directly detecting the top, bottom, right and left boundaries of the human body in the sketch to obtain the boundary of the sketch in the target detection canvas, and then using the boundary to calculate the center coordinates and scaling ratio of the sketch after it is redefined as a shape of (224, 224), to obtain a set of data representing the position information of the human body in each sketch, storing the obtained information in a file, and reading it out during training.

[0021] The decoding method of the motion mode includes establishing topological relationships between different joints of the human body and constructing a 24-dimensional list. Each dimension represents 24 joints of the SMPL human body model from top to bottom, and each dimension of the list stores a sublist containing the indexes of other joints that have the greatest impact on each joint.

[0022] The step of enhancing the feature vector obtained by the encoder using a self-attention model includes:

[0023] First, input the image. Image features are obtained through the encoder. Then, the image features are pooled to reduce the dimensionality. Feature vector ,Will Input three self-attention mechanism networks;

[0024] In the three self-attention mechanism networks, feature vectors are used. The query parameters are obtained through a linear layer. Key parameters and its own value Then it is input into the self-attention mechanism for calculation. Global attention for the three branches Where i = 1, 2, 3, and finally... and The optimized global features are obtained by summing. Where i represents the branch, the specific calculation process is as follows:

[0025]

[0026]

[0027]

[0028] in, , , These are three different parameters obtained by processing the feature vector generated by the encoder through the neural network. These three parameters represent the value of the feature vector, the index of the feature vector after being encoded by the neural network, and the value after encoding, respectively. It is composed of self-attention modules, which correspond to , , This corresponds to the Attention pipeline and requires the use of... , , The input is used to compute the self-attention mechanism, and the final output has the same dimension as the output produced by the encoder. , This is the final result after feature enhancement, and the dimension is also [missing information]. .

[0029] The decoding process of the decoder is as follows:

[0030]

[0031]

[0032]

[0033] In the above formula, , `cam` represents the final predicted SMPL pose parameters, shape parameters, and projection position parameters by the network, respectively. `PoseDecoder`, `ShapDecoder`, and `CamDecoder` represent the pose decoder, shape decoder, and projection position decoder, respectively. `concat` represents the concatenation operation. , , represents the feature vector obtained through feature enhancement operations, which is a vector with a dimension of 2048. init_pose, init_shape, and init_cam represent the SMPL parameters, which are the initial human pose, shape, and projection parameters of the model without any deformation.

[0034] This also includes the following steps:

[0035] Will , , The pose parameters, shape parameters, and projection parameters corresponding to the SMPL initialization model are respectively stitched together. , , Separately and , , In the first dimension, a splicing operation is performed to produce three new shapes, namely... , , The three vectors are then input into their respective multilayer perceptrons to obtain the final result, where the output of the pose decoder is... The output of the shape decoder is The output of the projection position decoder is Output and The parameters are passed to SMPL to generate a 3D mesh file, and then the 3D mesh is passed through... The parameters are projected onto the original image.

[0036] S3 further includes the calculation of the objective function:

[0037]

[0038]

[0039]

[0040]

[0041]

[0042] Where L represents a loss function used for parameter updates in deep learning training, shape3D is the 3D human body mesh, 3D represents the 3D human body joints, 2D represents the 2D human body joints, θ represents the pose parameters of the SMPL, and β represents the shape parameters of the SMPL. Mesh representing network prediction, Represents the target mesh. Represents three-dimensional joints, Represents two-dimensional joints. , These represent the attitude and shape parameters of the SMPL, respectively. Indicates batches during the training process;

[0043] The goal used in the pre-training phase is:

[0044]

[0045] The objective used in fine-tuning training is:

[0046] .

[0047] Implementing the embodiments of the present invention has the following beneficial effects: The present invention realizes the function of reconstructing a three-dimensional human body model from a hand-drawn human body sketch, and this function is applicable not only to users with drawing skills, but also to users without drawing skills. It greatly improves the efficiency of three-dimensional human body modeling. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of a hand-drawn sketch dataset that does not include two-dimensional joint information;

[0049] Figure 2 This is a schematic diagram illustrating the visualization effect of projecting a 3D model onto a 2D plane;

[0050] Figure 3 This is a schematic diagram of the SketchbodyNet network structure;

[0051] Figure 4 This is the final result of the network design. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0053] In this embodiment of the invention, a three-dimensional human body reconstruction method based on hand-drawing mainly includes the following steps:

[0054] (1) Based on the requirements, a dataset corresponding to hand-drawn sketches and 3D human models was created. Using the existing large public dataset mpi-inf-3dhp, a large dataset of human composite sketches and corresponding models was created. Then, the outlines in this dataset were used to draw sketches and 25k hand-drawn sketches were collected. This dataset was named sketch3D, which contains composite images, hand-drawn images, and corresponding 3D models of the human body.

[0055] (2) Based on the characteristics of the sketch, a deep neural network based on CNN and Transformer was created. Using an end-to-end method based on convolutional neural network combined with SMPL model, an image is input and a three-dimensional human body mesh is output.

[0056] (3) Training method: Since there are big differences between hand-drawn drawings and photos, the network was modified according to the characteristics of hand-drawn drawings, and a new training method was created: step by step training (SST). A model that performs well on hand-drawn sketches was trained based on the new dataset and applied to Web applications.

[0057] The following provides a detailed explanation of the above steps.

[0058] 1. Create a Sketch3D dataset.

[0059] This embodiment's dataset contains various human postures, including subjects performing everyday activities such as yoga, fencing, running, golf, and basketball from fourteen different camera angles. The dataset contains a wide range of posture information. A sketch corresponding to the 3D model is drawn by observing the 2D projection surface of the 3D model. Based on the dataset size, this embodiment divides the dataset into 40 groups, each containing 600 to 800 texture images. Depending on the different standards of the sketches and the varying drawing skills of the volunteers, the following requirements were set for the illustrators: 1) Volunteers with drawing skills should strive to reproduce as many details of the human body as possible from the texture images, including facial details and key projection wrinkles (e.g., wrinkles at joints); 2) Volunteers with average drawing skills are required to roughly reproduce the information of the human body's edges, without focusing on details; 3) Volunteers with very poor drawing skills are only required to draw the general outline of the human body's posture. Each person's drawing style is very different, which is consistent with the stylistic diversity of sketches. Figure 1 As can be seen, our final sketch dataset contains sketches with relatively slender shapes, sketches with disproportionate parts, and sketches lacking detail and with messy textures. This proves that sketches vary greatly among different users, but it is also a necessary attribute of the sketch dataset.

[0060] The dataset, named Sketch3D, contains 22,000 sketches and their corresponding 3D models, making it one of the largest sketch datasets available. While the dataset does not include 2D joint information, 2D projection parameters trained using a synthetic dataset are used to obtain the 2D joint information for the sketches. However, these positions are not absolutely accurate, as absolutely accurate projection positions require human annotation. This invention uses machine annotation and trains the synthetic sketches to determine these 2D projection parameters. Due to different users' drawing habits, sketches in the dataset may not be centered within the image. For example, some human figures may be located at the corners of the image, while others may be in the center. Previous methods for determining the human figure's position in human figure datasets involved labeling the top-left coordinates and center point, and the position of the human figure in the sketch canvas could be obtained from the coordinates of the 2D joints. However, the sketch dataset lacks 2D joint annotation information. To address this, this invention uses a preprocessing procedure to directly detect the top, bottom, and right boundaries of the human figure in the sketch to obtain the sketch's boundary in the target detection canvas. This boundary is then used to calculate the center coordinates and scaling ratio of the sketch after it is redefined to a shape of (224, 224). This yields a dataset representing the positional information of the human body in each sketch. This data is preprocessed and stored in a file, ready to be retrieved during training. The purpose of this is to further optimize the dataset. In this embodiment, the input image to the SketchBodyNet network is a well-cropped, accurately positioned human sketch. This approach serves two purposes: first, it makes the sketch features more prominent by only inputting the relevant cut-out parts of the human body; second, it speeds up the training computation, allowing for multiple training batches.

[0061] like Figure 2 As shown, the synthetic data contains human joint information, and the visualization effect of projecting the 3D model onto a 2D plane is shown in the figure. This dataset is designed to enable the network to learn prior knowledge of predicting the pose and projection of the human model. Since the body shapes of the subjects in the original dataset are relatively similar, the prior knowledge of shape is not much different from the shape initialized by SMPL. However, the network of this invention can also learn it without fixing the shape parameters. This is also to ensure that new sketch data can be obtained in the future that involves shape.

[0062] 2. Create the SketchBodyNet network structure based on the characteristics of the sketch.

[0063] The SketchBodyNet method of this invention is a novel network architecture formed through multi-branching and feature optimization based on Convolutional Neural Networks (CNNs). Unlike previous networks, this invention employs a novel optimization strategy for sketches. This strategy encompasses not only the overall network structure but also the network training process. For the network architecture, previous methods using a single branch to predict all parameters cannot optimize individual performance in pose, shape, or projection position. To reduce the mutual influence of these three parameters, this invention proposes a multi-branch attention method based on the Multi-Head-Attention module of Trasnformer [Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017, 30]. Details are as follows:

[0064] (1) Encoder structure based on ResNet50: Considering the fuzziness and randomness of sketches, this invention needs to select a network that can effectively normalize the sketch graphic features. Deep residual networks perform well in extracting image features. Therefore, this invention selects ResNet50 [He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.] (Residual Network) as the encoder of the network in this invention. ResNet proposes the idea of ​​residual learning, which skips a part of the neural network connection, so that information can flow directly from the input to the output, thereby solving the gradient vanishing problem. The main advantage of ResNet is that it can train very deep networks, which improves the accuracy and generalization performance of the model. The architecture of ResNet is very flexible. Networks of different depths can be constructed by stacking different numbers and sizes of ResNet modules. The network in this invention adopts the stacked structure of ResNet50. The ResNet module contains multiple residual blocks, each consisting of two convolutional layers and skip connections. These residual structures better preserve the feature information of the sketch. ResNet also uses techniques such as batch normalization and global average pooling to improve the training speed and accuracy of the model.

[0065] (2) Multi-branch attention enhancement mechanism: Considering that the human pose and shape information contained in the sketch is affected by the different drawing skills of the artist, some problems may occur if the pose and shape in the human body reconstruction are confused. For example, if the human body contained in a randomly drawn sketch has obvious shape deviations, these deviations will affect the pose reconstruction to some extent. To solve this problem, this embodiment designs the Decoder structure as a multi-branch structure, and enhances the attention to specific feature regions in each branch. These branch structures are as follows: Figure 3 As shown, the method of this invention applies the attention mechanism to multiple branches independently. The network decoding layer of this invention is divided into three branches (pose decoder, shape decoder, and projection position decoder). , , These decoders are all made by Composed of modules and stacked linear layers, this invention optimizes the attention of the feature vectors extracted by CNNs through different branches. This optimization makes the feature vectors exhibit different attention in different tasks, thereby enabling the task of this invention to be performed differently on the pose parameters. Shape parameters and projection parameters We will perform separate optimizations on it.

[0066] For the pose decoder, this invention adopts a motion-based decoding method [Wan Z, Li Z, Tian M, et al. Encoder-decoder with multi-level attention for 3D human shape and pose estimation[C] / / Proceedings of the IEEE / CVF International Conferenceon Computer Vision. 2021: 13033-13042.]. This method establishes topological relationships between different joints of the human body. Specifically, it constructs a 24-dimensional list, where each dimension represents 24 joints of the SMPL human model from top to bottom. Each dimension of the list stores a sublist containing the indices of other joints that have the greatest impact on each joint. This method can effectively improve the pose parameters. The reconstruction effect.

[0067] For the feature enhancement part of the Decoder, the pose decoder, shape decoder, and projection position decoder all use self-attention [Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017, 30] to enhance the feature vectors obtained by the Encoder.

[0068] The specific formula and explanation are as follows.

[0069] First, input the image. (The input image is a three-channel RGB image with a height and width of 224, such as...) Figure 1 Image features are obtained from Freehand Sketch and Synthetic Sketch by the Encoder. (A 7×7 feature map with 2048 dimensions generated after computation using ResNet50), then the image features are pooled to reduce the dimensionality. Feature vector ( Figure 3 Feature Tensor in the middle), Input into three self-attention networks, such as Figure 3 (The multi-head-attention network in CamDecoder, ShapeDecoder, and PoseDecoder). In these three self-attention mechanism networks, this invention uses feature vectors. The query parameters are obtained through a linear layer. Key parameters and its own value Then it is input into the self-attention mechanism for calculation. Global attention for the three branches (i=1,2,3). Finally, and The optimized global features are obtained by summing. (i represents the branch). The specific calculation process is as follows.

[0070] (1)

[0071] (2)

[0072] (3)

[0073] in , , These are three different parameters obtained by passing the feature vector generated by the Encoder through the neural network. These three parameters represent the value of the feature vector (Query), the index of the feature vector after being encoded by the neural network (Key), and the value after encoding (Value). It is composed of self-attention modules, which correspond to , , This corresponds to the Attention pipeline and requires the use of... , , The input is used to compute the self-attention mechanism, and the final output has the same dimension as the output produced by the Encoder. . The final result after feature enhancement also has dimensions. .

[0074] The pose decoder, shape decoder, and projection position decoder all use this enhancement mechanism to optimize the feature vectors obtained from the encoder. Then, based on the different dimensions of the pose, shape, and projection parameters, we scale the dimensions of their respective optimized feature vectors. The decoding process is as follows:

[0075] (4)

[0076] (5)

[0077] (6)

[0078] In the above formula, , `cam` represents the final predicted SMPL pose parameters, shape parameters, and projection position parameters, respectively. `PoseDecoder`, `ShapDecoder`, and `CamDecoder` represent the pose decoder, shape decoder, and projection position decoder, respectively. `concat` represents the concatenation operation. , , These represent the feature vectors we obtained through the feature enhancement operations described above; they are all vectors of dimension 2048. `init_pose`, `init_shape`, and `init_cam` represent the SMPL parameters, which are the initial human pose, shape, and projection parameters of the model without any deformation. The specific operation process is as follows:

[0079] Global pose features were obtained through feature enhancement. Global features of shape and projection global features This invention will , , The attitude parameters corresponding to the SMPL initialization model are respectively ( ), shape parameters ( ), projection parameters ( Perform a splicing operation to... , , Separately and , , In the first dimension, a splicing operation is performed to produce three new shapes, namely... , , The three vectors are then input into their respective multilayer perceptrons. Figure 3The final result is obtained by using the MLP network in CamDecoder, ShapeDecoder, and PoseDecoder. This is done to have standard data for comparative supervision during training, which can accelerate the convergence process. The output of the pose decoder is... The output of the shape decoder is The output of the projection position decoder is Output and The parameters are passed to SMPL to generate a 3D mesh file, and then the 3D mesh is passed through... The parameters are projected onto the original image, and the resulting effect is as follows: Figure 4 .

[0080] III. Training using the Step-by-Step Training (SST) method of transfer learning.

[0081] The training process includes pre-training and stepwise fine-tuning (SST) phases. The pre-trained model is trained on synthetic data (such as... Figure 1 As shown in the figure, the hand-drawn sketch lacks 2D joint annotations, rendering some existing methods for improving human pose accuracy unusable. Therefore, this invention uses 2D joint data from synthetic data as weak supervision during training. The SMPLify method [Bogo F, Kanazawa A, Lassner C, et al. Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image[C] / / Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, TheNetherlands, October 11-14, 2016, Proceedings, Part V 14. Springer International Publishing, 2016: 561-578] is used to optimize 3D target data using an iterative method during training. After optimization by SMPLify, some incorrectly labeled 3D targets in the original dataset are corrected, thereby improving the training effect of this invention.

[0082] The model of this invention, pre-trained, can be directly applied to hand-drawn sketches, but the reconstruction performance is generally poor. Therefore, in the second step, this invention uses hand-drawn sketch data for progressive fine-tuning training, ultimately obtaining a model with good performance. The specific objective function formula is as follows:

[0083] (7)

[0084] (8)

[0085] (9)

[0086] (10)

[0087] (11)

[0088] In these formulas, L represents a loss function, also known as the objective function, used for parameter updates during deep learning training. During training, the network's output and the actual standard results are input into L to obtain a final value, which is used to update the network's parameters using gradient descent. The table below represents their categories. This embodiment defines five categories of loss functions: 3D human mesh (shape3D), 3D human joints (3D), 2D human joints (2D), and pose parameters of SMPL. SMPL shape parameters The shape target calculation uses the L1 paradigm, which is the sum of the absolute values ​​of the differences between the predicted value and the target value. Other target calculations use the L2 paradigm, which is the average of the sum of the squares of the differences between the predicted value and the target value. Mesh representing network prediction, Represents the target mesh. Represents three-dimensional joints, Represents two-dimensional joints. , These represent the attitude and shape parameters of SMPL, respectively. This represents a batch during the training process. The objective used in the pre-training phase of this embodiment is:

[0089] (11)

[0090] In the second stage of fine-tuning training, since this embodiment focuses on pose fine-tuning, shape targets in the 3D Mesh target are removed. Furthermore, since hand-drawn sketches lack true values ​​for 2D joints, 2D losses are also removed. The final target used is:

[0091] (12).

[0092] This invention primarily proposes a multi-branch attention network, SketchBodyNet, suitable for 3D human reconstruction from synthetic sketches and hand-drawn sketches. This invention also creates a large dataset, Sketch3D, corresponding to synthetic and hand-drawn sketches and 3D models. Among multiple human regression networks, this embodiment demonstrates the best performance and evaluation results. The final model performance of this embodiment is as follows: Figure 4 As shown.

[0093] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A three-dimensional human body reconstruction method based on hand-drawing, characterized in that, Includes the following steps: S1: Create a dataset corresponding to the hand-drawn sketches and 3D human body models. The dataset includes composite images, hand-drawn images, and corresponding 3D human body models. S2: Use ResNet50 as the encoder of the convolutional neural network to allow information to flow directly from the input to the output. The decoder structure is designed as a multi-branch structure, including a pose decoder, a shape decoder, and a projection position decoder. Each decoder is composed of an attention module plus a stack of linear layers. The special tensor vector extracted by the convolutional neural network is optimized for attention through different branches. For the decoder using a motion-based decoding method, for the feature enhancement part of the decoder, the pose decoder, shape decoder, and projection position decoder all use a self-attention model to enhance the feature vector obtained from the encoder, optimize the feature vector obtained from the encoder, and then scale the optimized feature vector according to the different dimensions of pose, shape, and projection parameters. S3: The two-dimensional joint data of the synthetic data of the dataset used for pre-training are also used as weak supervision in the training process. The SMPLify method is used to optimize and correct the incorrectly labeled three-dimensional targets in the dataset. Hand-drawn sketch data is used for stepwise fine-tuning training to fine-tune the pose.

2. The three-dimensional human body reconstruction method based on hand-drawing according to claim 1, characterized in that, S1 specifically includes the following steps: S11: Obtain two-dimensional joint information of the sketch using two-dimensional projection parameters trained with a synthetic dataset; S12: Obtain the position information of the human body in the sketchboard from the coordinates of the two joints.

3. The three-dimensional human body reconstruction method based on hand-drawing according to claim 2, characterized in that, S12 further includes the following steps: using a preprocessing program, directly detecting the top, bottom, right boundaries of the human body in the sketch to obtain the boundary of the sketch in the target detection canvas, and then using the boundary to calculate the center coordinates and scaling ratio of the sketch after it is redefined as a shape of (224, 224), to obtain a set of data representing the position information of the human body in each sketch, storing the obtained information in a file, and reading it out during training.

4. The three-dimensional human body reconstruction method based on hand-drawing according to claim 1, characterized in that, The decoding method of the motion mode includes establishing topological relationships between different joints of the human body and constructing a 24-dimensional list. Each dimension represents 24 joints of the SMPL human body model from top to bottom, and each dimension of the list stores a sublist containing the indexes of other joints that have the greatest impact on each joint.

5. The three-dimensional human body reconstruction method based on hand-drawing according to claim 1, characterized in that, The step of enhancing the feature vector obtained by the encoder using a self-attention model includes: First, input the image. Image features are obtained through the encoder. Then, the image features are pooled to reduce the dimensionality. Feature vector ,Will Input three self-attention mechanism networks; In the three self-attention mechanism networks, feature vectors are used. The query parameters are obtained through a linear layer. Key parameters and its own value Then it is input into the self-attention mechanism for calculation. Global attention for the three branches Where i = 1, 2, 3, and finally... and The optimized global features are obtained by summing. Where i represents the branch, the specific calculation process is as follows: in, , , These are three different parameters obtained by processing the feature vector generated by the encoder through the neural network. These three parameters represent the value of the feature vector, the index of the feature vector after being encoded by the neural network, and the value after encoding, respectively. It is composed of self-attention modules, which correspond to , , This corresponds to the Attention pipeline and requires the use of... , , The input is used to compute the self-attention mechanism, and the final output has the same dimension as the output produced by the encoder. , This is the final result after feature enhancement, and the dimension is also [missing information]. .

6. The three-dimensional human body reconstruction method based on hand-drawing according to claim 1, characterized in that, The decoding process of the decoder is as follows: In the above formula, , `cam` represents the final predicted SMPL pose parameters, shape parameters, and projection position parameters by the network, respectively. `PoseDecoder`, `ShapDecoder`, and `CamDecoder` represent the pose decoder, shape decoder, and projection position decoder, respectively. `concat` represents the concatenation operation. , , represents the feature vector obtained through feature enhancement operations, which is a 2048-dimensional vector. init_pose, init_shape, and init_cam represent the SMPL parameters, namely the initial human pose, shape, and projection parameters of the model without any deformation.

7. The three-dimensional human body reconstruction method based on hand-drawing according to claim 6, characterized in that, It also includes the following steps: Will , , Perform stitching operations with the pose parameters, shape parameters, and projection parameters corresponding to the SMPL initialization model respectively, and then... , , Separately and , , In the first dimension, a splicing operation is performed to produce three new shapes, namely... , , The three vectors are then input into their respective multilayer perceptrons to obtain the final result, where the output of the pose decoder is... The output of the shape decoder is The output of the projection position decoder is Output and The parameters are passed to SMPL to generate a 3D mesh file, and then the 3D mesh is passed through... The parameters are projected onto the original image.

8. The three-dimensional human body reconstruction method based on hand-drawing according to any one of claims 1-7, characterized in that, S3 also includes the calculation of the objective function: Where L represents a loss function used for parameter updates in deep learning training, shape3D is the 3D human body mesh, 3D represents the 3D human body joints, 2D represents the 2D human body joints, θ represents the pose parameters of the SMPL, and β represents the shape parameters of the SMPL. Mesh representing network prediction, Represents the target mesh. Represents three-dimensional joints, Represents two-dimensional joints. , These represent the attitude and shape parameters of the SMPL, respectively. Indicates batches during the training process; The goal used in the pre-training phase is: The objective used in fine-tuning training is: 。

Images