Three-dimensional human body grid estimation method and system based on clothes scissors and human body vision model

Abstract

The invention discloses a three-dimensional human body grid estimation method and system based on clothes scissors and a human body vision model. The method comprises the following steps: extracting a first human body image containing clothes from an image to be reconstructed; identifying a boundary between a costume area and a body area in the first human body image, and removing the costume area from the first human body image to obtain a second human body image; the second human body image is input into a three-dimensional human body grid estimation module, and the three-dimensional human body grid estimation module is used for extracting three key intermediate representations including joint information, depth information and contour information from the second human body image; and aligning the intermediate representation of the preliminarily generated three-dimensional human body grid with the intermediate representation extracted from the second human body image, minimizing the difference between the intermediate representation and the intermediate representation, and finally outputting a high-precision three-dimensional human body grid. According to the method, the stability of the algorithm under the complex clothing condition can be improved, and the high fidelity of the reconstructed grid is ensured.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and computer graphics, and more specifically, relates to a three-dimensional human body mesh estimation method and system based on clothing shears and human body visual models. Background Art

[0002] 3D human mesh estimation refers to the process of reconstructing a complete and accurate 3D human body model from a monocular image or video. This technology accurately estimates the 3D shape and pose parameters of the human body to generate a 3D digital model based on the SMPL (Skinned Multi-Person Linear Model), which can accurately reflect the geometric structure and movement posture of the human body. Specifically, 3D human body mesh recovery is usually based on a parameterized human body model. Through optimization or regression methods, the parameters of the model are estimated from the input image to generate a 3D human body mesh. With the development of deep learning, 3D human body mesh methods can be mainly divided into two categories: regression-based methods and optimization-based methods. Regression-based methods directly regress parameters from the network and improve the accuracy of human pose and shape estimation through various forms of supervision. Optimization-based methods focus on estimating the 3D human body mesh by minimizing the error between the mesh and 2D image information.

[0003] Due to the lack of annotated datasets covering diverse clothing, existing 3D human mesh reconstruction methods struggle to adequately address clothing occlusion. State-of-the-art methods, such as PyMAF, PyMAF-X, Virtual Markers, and TokenHMR, often suffer from shape misestimation or failure to accurately capture complex poses when working with loose clothing. Even for scenes with loose clothing, existing methods still suffer from insufficient accuracy in pose and shape estimation.

[0004] Existing methods typically do not directly recover the 3D human mesh from the original RGB image, but instead use an intermediate representation to indirectly complete the reconstruction. Introducing an intermediate representation helps alleviate the data scarcity problem, thereby improving the accuracy of 3D human mesh reconstruction. However, inferring the intermediate representation from the original RGB image remains an open problem that has not been fully solved. Any error in the inferred intermediate representation may cause the 3D human mesh reconstruction to fail. To address this issue, many methods attempt to combine multiple intermediate representations. However, the intermediate representations used by these methods are usually predicted separately by multiple deep neural networks, resulting in inconsistencies between different representations, which in turn affects the accuracy of 3D human mesh recovery.

[0005] In summary, although single-image 3D human mesh estimation methods have made significant progress in recent years, the following key issues still exist: (1) Complex clothing interference: Traditional methods do not fully consider the impact of complex clothing such as loose clothing on the reconstruction results, resulting in a significant decrease in accuracy under clothing deformation and occlusion scenarios; (2) Inconsistency in intermediate representations: Traditional methods extract intermediate representations through multiple independent networks, and there is a lack of consistency constraints between different representations, resulting in insufficient robustness and accuracy. Summary of the Invention

[0006] In response to the above-mentioned defects or improvement needs of the existing technology, the present invention provides a three-dimensional human body mesh estimation method and system based on clothing shears and human visual models, which can improve the stability of the algorithm under complex clothing conditions and ensure the high fidelity of the reconstructed mesh.

[0007] To achieve the above object, according to one aspect of the present invention, a method for estimating a 3D human body mesh based on clothing shears and a human body visual model is provided, comprising the steps of:

[0008] extracting a first human body image including clothing from the image to be reconstructed;

[0009] Inputting the first human body image into a clothing cutting module, the clothing cutting module is used to identify the boundary between the clothing area and the body area, remove the clothing area from the first human body image, and output a second human body image;

[0010] The second human body image is input into a three-dimensional human body mesh estimation module, which includes an estimation intermediate representation submodule and an iterative optimization parameter submodule. The estimation intermediate representation submodule predicts the first joint heat map information, first depth information and first contour information of the second human body image based on the human visual model. The iterative optimization parameter submodule is used to obtain an initialized three-dimensional human body mesh from the second human body image, and iteratively optimize the initialized three-dimensional human body mesh. During each round of iterative optimization, the second joint heat map information, second depth information and second contour information of the iteratively optimized three-dimensional human body mesh are extracted. The total loss of the iterative optimization parameter submodule is jointly calculated based on the first joint heat map information, first depth information and first contour information of the second human body image and the second joint heat map information, second depth information and second contour information of the iteratively optimized three-dimensional human body mesh. The three-dimensional human body mesh when the iterative optimization stops is used as the three-dimensional human body mesh finally output by the three-dimensional human body mesh estimation module.

[0011] Preferably, the clothing cutting module includes a body segmentation submodule, a human body edge perception submodule and a clothing cutting submodule;

[0012] The body segmentation submodule is used to extract body semantic features and body semantic masks from the first human body image;

[0013] The human body edge perception submodule is used to downsample the first human body image using bilinear interpolation to generate multi-resolution images of multiple scales, fuse the multi-resolution images with the body semantic features, and perform human body edge detection based on the fused features to obtain human body edge features;

[0014] The clothing cropping submodule is used to generate a cropping mask using the body semantic mask and the human body edge features, and output the second human body image according to the cropping mask.

[0015] Preferably, the body semantic mask loss, the human body edge prediction loss and the cropping result loss are calculated respectively, and the body semantic mask loss, the human body edge prediction loss and the cropping result loss are weighted to obtain the total loss of the clothing cropping module, and the training process of the clothing cropping module is supervised by the total loss of the clothing cropping module.

[0016] Preferably, the calculation formula of the body semantic mask loss is:

[0017]

[0018] Among them, L cm represents the body semantic mask loss, C m represents the body semantic mask, G 16x represents the 16-fold downsampling and Gaussian blur operation, α g represents the body semantics ground truth annotation mask, ||.||2 represents the calculation of L2 norm;

[0019] The calculation formula of the human body edge prediction loss is:

[0020]

[0021] in, represents the human body edge prediction loss, Represents the edge feature of the human body, m d represents a binary boundary mask, and ||.||1 represents the L1 norm;

[0022] The calculation formula of the cropping result loss is:

[0023]

[0024] Among them, L C Represents the cropping result loss, C r represents the cropping mask, L sl represents the similarity loss, L kl represents divergent loss;

[0025] The calculation formula of the total loss of the clothing cutting module is:

[0026] L cloth =λ h L c +λ c L C +λ m L cm

[0027] Among them, L cloth represents the total loss of the clothing cutting module, λ h ,λ c ,λ m Respectively represent L H , L C , L cm The weight coefficient of .

[0028] Preferably, the calculation formula for outputting the second human body image according to the cropping mask is:

[0029]

[0030] in, is the second human body image, and I is the first human body image.

[0031] Preferably, the estimated intermediate representation submodule includes an encoder, a first decoder, a second decoder and a third decoder;

[0032] The encoder is used to extract global semantic features of the second human body image, and input the global semantic features of the second human body image to the first decoder, the second decoder and the third decoder respectively;

[0033] The first decoder is used to map the global semantic features of the second human body image to a low-dimensional feature space through a fully connected layer, then upsample the mapped features through two transposed convolution blocks, and then generate first joint heat map information of the second human body image through a 1×1 convolution layer;

[0034] The second decoder is used to generate first depth information of the second human body image;

[0035] The third decoder is used to generate first contour information of the second human body image.

[0036] Preferably, the losses of the first decoder, the second decoder, and the third decoder are calculated respectively, and the losses of the first decoder, the second decoder, and the third decoder are minimized through training:

[0037] The loss of the first decoder is calculated as:

[0038] L pose =MSE(J gt ,G j )

[0039] Among them, L pose represents the loss of the first decoder, MSE(.) represents the mean square error, J gt represents the true joint heat map, G j Represents the predicted first joint heat map information;

[0040] The loss of the second decoder is calculated as:

[0041] Δd i =log(D gti )-log(G di )

[0042]

[0043] Among them, D gti represents the true depth of the i-th pixel, G di Represents the predicted first depth information of the i-th pixel, Δd i represents the intermediate variable, N p Indicates the number of pixels in the human body area in the image, L depth represents the loss of the second decoder;

[0044] The calculation formula of the loss of the third decoder is:

[0045] L mask =||M gt -G m ||1

[0046] Among them, L mask represents the loss of the third decoder, M gt represents the true contour, G m represents the predicted first contour information, and ||.||1 represents the L1 norm.

[0047] Preferably, the calculation formula for the total loss of the iterative optimization parameter submodule is:

[0048] L smpl =λ d L D +λ m L m +λ j L J

[0049] in,

[0050]

[0051] Among them, L smpl represents the total loss of the iterative optimization parameter submodule, λ d ,λ m and λ j L D , L M , L J The coefficient of G j Represents the predicted first joint heat map information, G d Represents the predicted first depth information, G m Represents the predicted first contour information, Represents the second joint heat map information, Represents the second contour information, Indicates the second depth information.

[0052] According to another aspect of the present invention, a 3D human body mesh estimation system based on clothing shears and a human body visual model is provided, comprising:

[0053] A preprocessing module, configured to extract a first human body image including clothing from an image to be reconstructed;

[0054] The clothing cutting module is used to identify the boundary between the clothing area and the body area in the first human body image, remove the clothing area from the first human body image, and output a second human body image;

[0055] A three-dimensional human body mesh estimation module, the three-dimensional human body mesh estimation module includes an estimation intermediate representation submodule and an iterative optimization parameter submodule, the estimation intermediate representation submodule predicts the first joint heat map information, first depth information and first contour information of the second human body image based on the human visual model, the iterative optimization parameter submodule is used to obtain an initialized three-dimensional human body mesh from the second human body image, and iteratively optimize the initialized three-dimensional human body mesh. During each round of iterative optimization, the second joint heat map information, second depth information and second contour information of the iteratively optimized three-dimensional human body mesh are extracted, and the total loss of the iterative optimization parameter submodule is jointly calculated based on the first joint heat map information, first depth information, first contour information of the second human body image and the second joint heat map information, second depth information and second contour information of the iteratively optimized three-dimensional human body mesh, and the three-dimensional human body mesh when the iterative optimization stops is used as the three-dimensional human body mesh finally output by the three-dimensional human body mesh estimation module.

[0056] In general, the above technical solutions conceived by the present invention have beneficial effects compared with the existing technology:

[0057] (1) The present invention achieves significant advantages by collaboratively optimizing clothing cutting and human visual models to reconstruct a three-dimensional human body mesh with high precision from a single image of loose clothing. The clothing cutting module accurately eliminates the deformation interference of loose clothing, significantly improving the stability of the algorithm under complex clothing conditions. The intermediate representations (such as joint points, depth maps, and semantic contours) extracted by the human visual model retain the human body topology and local details (such as joint positions and limb proportions) to the greatest extent, ensuring the high fidelity of the reconstructed mesh.

[0058] (2) The clothing cutting module proposed in this paper significantly improves the algorithm's stability under complex clothing conditions. For loose clothing scenes such as long skirts and coats, the algorithm's stability is significantly improved under complex clothing conditions, effectively solving the mesh distortion problem caused by clothing deformation in traditional methods.

[0059] (3) The proposed 3D human mesh estimation module based on the human visual model maximizes the effectiveness of intermediate representation estimation. It supports stable reconstruction in extreme poses (such as sports and dance movements) and is robust to individual body shape differences (such as height, weight, and body type). It does not require additional calibration and is suitable for a variety of human reconstruction tasks.

[0060] (4) The present invention only requires a single image of a person wearing loose clothing to accurately reconstruct a 3D human body mesh, without relying on multi-view images or sensor data. Compared to traditional methods, the present invention has the following significant advantages: Low cost and convenience: Reconstruction can be completed using only a single image, without the need for complex equipment (such as depth cameras, multi-view sensors) or expensive data acquisition, significantly reducing hardware costs and operational barriers. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flow chart of a single-image three-dimensional human body mesh estimation method according to an embodiment of the present invention;

[0062] Figure 2 Schematic diagram of the network framework used by the clothing cutting module in an embodiment of the present invention;

[0063] Figure 3 3D human body mesh estimation module according to an embodiment of the present invention;

[0064] Figure 4 is a schematic diagram of a network framework of a first decoder according to an embodiment of the present invention;

[0065] Figure 5 This is a rendering of the processing process of a field image according to an embodiment of the present invention;

[0066] Figure 6 This is an example effect diagram of processing field images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0068] In the description of the embodiments of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include at least one of such features. "Multiple" means two or more.

[0069] The terms "including" and "having" and any variations thereof in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product or device comprising a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products or devices.

[0070] The naming or numbering of the steps in the embodiments of the present invention does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0071] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0072] The present invention provides a three-dimensional human body mesh estimation method and system based on clothing shears and human body visual model, which are respectively described below.

[0073] like Figure 1 As shown, the three-dimensional human body mesh estimation method based on clothing shears and human body visual model according to the embodiment of the present invention includes the following steps:

[0074] S1, extracting a first human body image including clothing from an image to be reconstructed.

[0075] Specifically, a portrait image to be reconstructed is obtained, a portrait mask is extracted from the portrait image, and background factors are removed to obtain a first human body image including clothing.

[0076] The background of the input image P to be reconstructed is segmented to obtain a black and white image describing the portrait area, denoted as M. The black and white image is then used to restrict the computer to operate the first human image I of the image to be reconstructed:

[0077]

[0078] Represents dot product.

[0079] In one embodiment, an API interface provided by an existing platform can be used to remove the background outside the human body image from the input image, and then the image mask can be segmented using code.

[0080] S2, inputting the first human body image into a clothing cutting module, the clothing cutting module is used to identify the boundary between the clothing area and the body area, remove the clothing area from the first human body image, and output a second human body image.

[0081] The clothing cropping module takes an input image of a person wearing loose clothing and uses a body segmentation network and edge-aware algorithms to accurately identify the boundary between clothing and the person, removing redundant areas of clothing (such as loose parts and wrinkles), ensuring that the cropped image retains only essential information related to the person's topological structure. This module effectively eliminates the interference of loose clothing deformation on 3D body mesh reconstruction, providing high-quality input for subsequent mesh estimation.

[0082] The network framework of the clothing cutting module is as follows Figure 2 As shown in the figure, it contains three modules: a body segmentation submodule, an edge perception submodule, and a clothing cropping submodule. The body segmentation submodule is used to extract body semantic features and a body semantic mask from the first human image; the human edge perception submodule is used to downsample the first human image using bilinear interpolation to generate multi-resolution images of various scales, fuse the multi-resolution images with the body semantic features, and perform human edge detection based on the fused features to obtain human edge features; the clothing cropping submodule is used to generate a cropping mask from the body semantic mask and the human edge features, and output a second human image based on the cropping mask.

[0083] The body segmentation submodule is the basis of the clothing cutting module. Its core goal is to accurately identify the human body area from the input image and generate the human body mask C m The specific implementation is to first extract multi-scale features from the input image through a multi-layer convolutional neural network. Shallow feature extraction It is mainly used to extract local detail features of the image (such as edges) and pass these features to the edge perception submodule to enhance the accuracy of edge detection. Deep feature extraction It is used to extract the distribution information of the body in the image and provide high-level feature support for body mask generation. During the training process, in order to accurately estimate the body semantic mask C m , the present invention will The input is sent to the convolution layer and then processed by the Sigmoid activation function to compress the channel dimension to 1. Then the real mask α is used g The downsampled version of m Since C m is considered smooth, so the present invention adopts L2 loss. By back propagation, the network parameters are optimized to make the predicted mask C m Gradually approaching the true mask α g .

[0084] Preferably, 16-fold downsampling and Gaussian blur (G 16x ) Process the true annotation mask α g , supervised by L2 loss function C m smoothness and spatial consistency.

[0085]

[0086] Among them, L cm represents the body semantic mask loss, C m represents the body semantic mask, G 16x represents the 16-fold downsampling and Gaussian blur operation, α g represents the body semantic ground truth annotation mask, and ||.||2 represents the calculation of the L2 norm.

[0087] The edge perception submodule fuses multi-resolution input images (bilinear interpolation generates I 1 / s ) and semantic features Generate edge prediction features through Feature Pyramid Fusion (FPF)

[0088] First, the input image is downsampled by bilinear interpolation to generate images of different scales I 1 / s , s=2, 4, 8, 16….

[0089] Its formula is:

[0090]

[0091] Among them, I 1 / s (x,y) represents the interpolated image I1 / s The pixel value at position (x, y). X(i) and Y(j) represent the coordinates corresponding to position (i, j) in the original image I, while X(i+1) and Y(j+1) are the coordinates of the adjacent position in the original image I. When downsampled to 1 / s, the relationship between the new coordinates (x, y) and the original coordinates (i, j) is given by: x = s·X(i) and y = s·Y(j), where I(i, j) represents the coordinates of the pixel in the original image and s represents the scale.

[0092] Then, the features of different resolutions are integrated through the feature pyramid fusion module. Downsampled image and edge features Splicing is performed to obtain the fused features. Finally, the spliced ​​features are input into the convolutional layer The kernel size of the convolution layer is 3x3, and the ReLU activation function is used. The convolution operation can further extract local information and maintain the spatial structure of the image. The edge details are learned through the L1 loss function. And use the binary boundary mask m d Finally, an accurate edge estimation result is output for the subsequent clothing cutting step.

[0093] The formula is as follows:

[0094]

[0095] Where k2 = {0,...,2}, It consists of convolution with ReLU activation function with kernel size of 3. FPF has two advantages, represents the features of the k2th layer of the edge perception submodule, represents the features of the k2-1th layer of the edge perception submodule, Indicates the original image downsampling (2 k2+1 ) times. First, it preserves the spatial structure and features of the image, allowing more local information to be encoded. Second, it achieves fusion by simply concatenating features without the need for additional convolutional or fully connected layers, thereby improving efficiency.

[0096] The calculation formula for human edge prediction loss is:

[0097]

[0098] represents the human edge prediction loss, Represents the edge features of the human body, m d Represents a binary boundary mask, which is obtained from the true value α by dilation and erosion operations gGenerated, if it is inside the boundary transition region, the mask sets the pixel value to 1, otherwise it is set to 0, ||.||1 represents the L1 norm.

[0099] The following describes the specific implementation process of the clothing cropping submodule. The clothing cropping submodule is used to generate a more accurate cropping mask based on the low-resolution body semantic mask obtained by the body segmentation submodule and the high-resolution human edge features obtained by the edge estimation submodule, and outputs a second human image based on the cropping mask.

[0100] First, use bilinear interpolation to Perform double upsampling to obtain the features of the k4th layer of the clothing cutting submodule Only features from layers 6 and 7 are upsampled.

[0101]

[0102] where k3 = {5, 6}, represents the features of the k3th layer of the clothing cutting submodule, express The value of each point in, (x,y) represents the coordinates, (ix) + =max(ix,0),(iy) + =max(jy,0), max(,) means taking the larger value of the two to ensure that the weight is non-negative.

[0103] Next, the decoding feature fusion (FDF) is performed. FDF fuses the edge decoding features Decoding features during the cropping process Get decoding features The specific process can be expressed as follows:

[0104]

[0105] in, It represents the result of upgrading the resolution of the k6-1th layer feature of the clothing cutting submodule. represents the features of the k6-3th layer of the edge perception submodule, C d It consists of a convolutional layer with a kernel size of 1 and a ReLU activation layer. Through the FDF process, the edge splicing results are effectively fused with the clothing cutting features, which improves the final cutting accuracy.

[0106] Through the FDF process, the edge stitching results are effectively integrated with the garment cutting features, which improves the accuracy of the final cutting.

[0107] Finally, the predicted clothing cutting result C r and through L C Optimize.

[0108] L C =||C r -α g ||1+L sl (C r ,α g )+L kl (C r ,α g )

[0109] Among them, L kl is the similarity loss, L sl Indicates divergence loss.

[0110] The overall process of the clothing cutting module is supervised by three loss functions: L cm (Body Mask Loss), L H (marginal prediction loss) and L C (Crop result loss). The final loss function L cloth Is the weighted sum of these three losses. h =4,λ c =4,λ m =2, which are the weight coefficients of each loss function. cm Supervised body mask generation, L H The accuracy of supervised edge prediction, L C Supervise the accuracy of the cropping results.

[0111] L cloth =λ h L H +λ c L C +λ m L cm

[0112] The final output is cropped to obtain the second human body image Provide interference-free input for subsequent 3D reconstruction, the second human body image Clothing is not included.

[0113] In one embodiment, an API interface provided by an existing platform may be used to crop clothing from an input image to obtain a human body after cropping the clothing.

[0114] S3, input the second human body image into the three-dimensional human body mesh estimation module, the three-dimensional human body mesh estimation module includes an estimation intermediate representation submodule and an iterative optimization parameter submodule, the estimation intermediate representation submodule predicts the first joint heat map information, first depth information and first contour information of the second human body image based on the human visual model, the iterative optimization parameter submodule is used to obtain an initialized three-dimensional human body mesh from the second human body image, and iteratively optimize the initialized three-dimensional human body mesh. During each round of iterative optimization, the second joint heat map information, second depth information and second contour information of the iteratively optimized three-dimensional human body mesh are extracted, and the total loss of the iterative optimization parameter submodule is jointly calculated based on the first joint heat map information, first depth information and first contour information of the second human body image and the second joint heat map information, second depth information and second contour information of the iteratively optimized three-dimensional human body mesh. The three-dimensional human body mesh when the iterative optimization stops is used as the three-dimensional human body mesh finally output by the three-dimensional human body mesh estimation module.

[0115] The network framework of the 3D human mesh estimation module is as follows Figure 3 shown.

[0116] The pre-trained estimated intermediate representation submodule based on the human visual model is called to extract the intermediate representation of the human body including joints, depth and contours, providing feature support for 3D human mesh reconstruction.

[0117] Furthermore, the estimated intermediate representation submodule includes an encoder, a first decoder Second decoder and the third decoder

[0118] The encoder is used to extract the global semantic features of the second human body image and input the global semantic features of the second human body image into the first decoder Second decoder and the third decoder

[0119] First decoder It is used to map the global semantic features of the second human body image to a low-dimensional feature space through a fully connected layer, and then upsample the mapped features through two transposed convolution blocks, and then generate the first joint heat map information of the second human body image through a 1×1 convolution layer, with a size of

[0120] Second decoder Used to generate the first depth information of the second human body image; the structure is the same as the first decoder, except that the output dimension is different. The global semantic features are first mapped to the low-dimensional feature space through the fully connected layer, and then the features are gradually upsampled through multiple transposed convolutions, and finally restored to the original image resolution H×W, and a single-channel depth map (size H×W×1) is generated through a 1×1 convolutional layer.

[0121] The third decoder Used to generate first contour information of the second human body image; its structure is consistent with that of the second decoder, and it outputs a contour image of size H×W×1.

[0122] The human vision model uses an encoder-decoder architecture, where the encoder is initialized with pre-trained MAE (MaskedAutoencoder) weights and the decoder It is a lightweight multi-task network (with random parameter initialization) that supports dynamic adaptation of pose, depth and contour estimation tasks. In the pre-training stage, the input image is divided into fixed-size blocks, denoted as (d is the number of blocks, C is the number of channels, H is the image height, and W is the image width). After randomly masking some blocks, the encoder processes the unmasked blocks and generates a latent representation. The decoder then uses the encoder output and position embeddings to reconstruct the full image, including the masked portions. The decoder's tasks include: 2D joint point estimation, body depth estimation, and body contour extraction.

[0123] The fine-tuning stage mainly changes the structure of the encoder to fit specific tasks, including the 2D joint point estimation subtask, the depth estimation subtask, and the contour estimation subtask.

[0124] The first decoder is used in the 2D joint point estimation subtask First, the encoder output F is transformed into out Mapping to a low-dimensional feature space, the decoder upsamples the features through two transposed convolutional blocks, each consisting of a transposed convolutional layer, batch normalization, and ReLU activation. Each block doubles the size of the feature map. In the final step, a 1×1 convolutional layer generates a heatmap of the joint. This process is mathematically represented as:

[0125] G j =Q 1×1 (V(V(F out )))

[0126] where Q 1×1 represents a 1×1 convolutional layer, and V represents a transposed convolutional block. out Output refers to the output of the encoder. is the predicted heat map of each joint, that is, the first joint heat map information, N k Indicates the number of joints to be predicted.

[0127] This process minimizes the predicted heatmap G k and the true value J gt The mean square error (MSE) loss between them.

[0128] L pose =MSE(J gt ,G j )

[0129] Among them, J gt Represents the true joint heatmap.

[0130] The depth estimation subtask uses the second decoder To transform the encoder feature F out The output is regressed into an output with a single channel. d Depend on Given, G d That is, the first depth information, where Based on the maximum and minimum depth, G d The depth values ​​in are normalized to the range [0, 1].

[0131] This process is trained to minimize depth.

[0132] Δd=log(D gt )-log(G d )

[0133] Δd i =log(D gti )-log(G di )

[0134]

[0135] Among them, D gt Indicates the true depth, N p Indicates the number of pixels in the human body area in the image.

[0136] The contour estimation subtask uses the third decoder It is similar to D. The silhouette map is given by Given, where The process is trained to minimize the mask L mask .

[0137] L mask =||M gt -G m ||1

[0138] Among them, Mgt represents the true contour, G m It represents the predicted outline of the human body area after removing the clothing, that is, the first outline information.

[0139] After extracting an intermediate representation from the input image via a human vision model, the iterative parameter optimization module uses a simple iterative fitting strategy to recover the 3D human mesh.

[0140] Preferably, based on the cropped image Initialize the SMPL model parameters using the SPIN, PyMAF, or TokenHMR algorithm to obtain the initial 3D human body mesh

[0141] Generating SMPL models via a differentiable renderer Depth map Mask map and 2D joints That is, second joint heat map information, second depth information, and second contour information;

[0142] Preferably, the multi-task loss is minimized by jointly optimizing the SMPL shape parameter β, pose parameter θ, size parameter s, and translation parameter t:

[0143] L smpl =λ d L D +λ m L m +λ j L J

[0144] in,

[0145]

[0146] Among them, λ d ,λ m and λ j is the weight factor of the three losses, and the weight setting can be set to λ d =5,λ m =5 and λ j =10 to balance the impact of various losses.

[0147] The three-dimensional human body mesh that reaches the iterative optimization stopping condition (for example, 50 optimization iterations) is used as the three-dimensional human body mesh finally output by the three-dimensional human body mesh estimation module.

[0148] In one embodiment, an API interface provided by an existing platform can be used to input a cropped clothing image and output corresponding 2D joint point, depth, and contour information.

[0149] In one embodiment, an API interface provided by an existing platform may be used to input the cropped clothing image and intermediate representation information, and output a three-dimensional human body mesh.

[0150] Thanks to the semantic cropping module's precise elimination of deformation of loose clothing and the extraction of intermediate representations for human visual model estimation, the present invention significantly improves the estimation accuracy and robustness of the three-dimensional human mesh model under complex clothing. Figure 5 Shown are the results of human body mesh estimation of the field dataset reconstructed from a single image by the present invention, including clothing cutting results and final estimation results. Figure 6 The figure shows the human body mesh estimation results of the field dataset reconstructed from a single image by the present invention. It can be seen that the present invention can estimate accurate 3D human body mesh models for different clothing and body shapes.

[0151] A three-dimensional human body mesh estimation system based on clothing shears and a human body visual model according to an embodiment of the present invention includes:

[0152] A preprocessing module, configured to extract a first human body image including clothing from an image to be reconstructed;

[0153] The clothing cutting module is used to identify the boundary between the clothing area and the body area in the first human body image, remove the clothing area from the first human body image, and output a second human body image;

[0154] The three-dimensional human body mesh estimation module includes an estimation intermediate representation submodule and an iterative optimization parameter submodule. The estimation intermediate representation submodule is used to predict the first joint heat map information, first depth information and first contour information of the second human body image. The iterative optimization parameter submodule is used to obtain an initialized three-dimensional human body mesh from the second human body image and iteratively optimize the initialized three-dimensional human body mesh. During each round of iterative optimization, the second joint heat map information, second depth information and second contour information of the iteratively optimized three-dimensional human body mesh are extracted. The total loss of the iterative optimization parameter submodule is jointly calculated based on the first joint heat map information, first depth information and first contour information of the second human body image and the second joint heat map information, second depth information and second contour information of the iteratively optimized three-dimensional human body mesh. The three-dimensional human body mesh when the iterative optimization stops is used as the three-dimensional human body mesh finally output by the three-dimensional human body mesh estimation module.

[0155] The working principle and technical effects of the 3D human body mesh estimation system are the same as those of the above-mentioned 3D human body mesh estimation method, and will not be described in detail here.

[0156] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A 3D human body mesh estimation method based on clothing shears and human body visual model, characterized in that: Including steps: extracting a first human body image including clothing from the image to be reconstructed; Inputting the first human body image into a clothing cutting module, the clothing cutting module is used to identify the boundary between the clothing area and the body area, remove the clothing area from the first human body image, and output a second human body image; The second human body image is input into a three-dimensional human body mesh estimation module, which includes an estimation intermediate representation submodule and an iterative optimization parameter submodule. The estimation intermediate representation submodule predicts the first joint heat map information, first depth information and first contour information of the second human body image based on the human visual model. The iterative optimization parameter submodule is used to obtain an initialized three-dimensional human body mesh from the second human body image, and iteratively optimize the initialized three-dimensional human body mesh. During each round of iterative optimization, the second joint heat map information, second depth information and second contour information of the iteratively optimized three-dimensional human body mesh are extracted. The total loss of the iterative optimization parameter submodule is jointly calculated based on the first joint heat map information, first depth information and first contour information of the second human body image and the second joint heat map information, second depth information and second contour information of the iteratively optimized three-dimensional human body mesh. The three-dimensional human body mesh when the iterative optimization stops is used as the three-dimensional human body mesh finally output by the three-dimensional human body mesh estimation module.

2. The method for estimating a 3D human body mesh based on clothing shears and a human body visual model according to claim 1, wherein: The clothing cutting module includes a body segmentation submodule, a human body edge perception submodule and a clothing cutting submodule; The body segmentation submodule is used to extract body semantic features and body semantic masks from the first human body image; The human body edge perception submodule is used to downsample the first human body image using bilinear interpolation to generate multi-resolution images of multiple scales, fuse the multi-resolution images with the body semantic features, and perform human body edge detection based on the fused features to obtain human body edge features; The clothing cropping submodule is used to generate a cropping mask using the body semantic mask and the human body edge features, and output the second human body image according to the cropping mask.

3. The method for estimating a 3D human body mesh based on clothing shears and a human body visual model according to claim 2, wherein: The body semantic mask loss, human body edge prediction loss and cropping result loss are calculated respectively, and the body semantic mask loss, human body edge prediction loss and cropping result loss are weighted to obtain the total loss of the clothing cropping module. The total loss of the clothing cropping module is used to supervise the training process of the clothing cropping module.

4. The method for estimating a 3D human body mesh based on clothing shears and a human body visual model according to claim 3, wherein: The calculation formula of the body semantic mask loss is: Among them, L cm represents the body semantic mask loss, C m represents the body semantic mask, G 16x represents the 16-fold downsampling and Gaussian blur operation, α g represents the body semantics ground truth annotation mask, ||.||2 represents the calculation of L2 norm; The calculation formula of the human body edge prediction loss is: in, represents the human body edge prediction loss, Represents the edge feature of the human body, m d represents a binary boundary mask, and ||.||1 represents the L1 norm; The calculation formula of the cropping result loss is: L C =||||C r -a g ||||1+L sl (C r ,a g )+L kl (C r ,a g ) Among them, L C Represents the cropping result loss, C r represents the cropping mask, L sl represents the similarity loss, L kl represents divergent loss; The calculation formula of the total loss of the clothing cutting module is: L cloth =λ h L H +λ c L C +λ m L cm Among them, L cloth represents the total loss of the clothing cutting module, λ h ,λ c ,λ m Respectively represent L H , L c , L cm The weight coefficient of .

5. The method for estimating a 3D human body mesh based on clothing shears and a human body visual model according to claim 4, wherein: The calculation formula for outputting the second human body image according to the cropping mask is: in, is the second human body image, and I is the first human body image.

6. The method for estimating a 3D human body mesh based on clothing shears and a human body visual model according to claim 1, wherein: The estimated intermediate representation submodule includes an encoder, a first decoder, a second decoder and a third decoder; The encoder is used to extract global semantic features of the second human body image, and input the global semantic features of the second human body image to the first decoder, the second decoder and the third decoder respectively; The first decoder is used to map the global semantic features of the second human body image to a low-dimensional feature space through a fully connected layer, then upsample the mapped features through two transposed convolution blocks, and then generate first joint heat map information of the second human body image through a 1×1 convolution layer; The second decoder is used to generate first depth information of the second human body image; The third decoder is used to generate first contour information of the second human body image.

7. The method for estimating a 3D human body mesh based on clothing shears and a human body visual model according to claim 6, wherein: Calculate the losses of the first decoder, the second decoder, and the third decoder respectively, and minimize the losses of the first decoder, the second decoder, and the third decoder through training: The loss of the first decoder is calculated as: L pose =MSE(J gt ,G j ) Among them, L pose represents the loss of the first decoder, MSE(.) represents the mean square error, J gt represents the true joint heat map, G j Represents the predicted first joint heat map information; The loss of the second decoder is calculated as: Δd i =log(D gti )-log(G di ) Among them, D gti represents the true depth of the i-th pixel, G di Represents the predicted first depth information of the i-th pixel, Δd i represents the intermediate variable, N p Indicates the number of pixels in the human body area in the image, L deptj represents the loss of the second decoder; The calculation formula of the loss of the third decoder is: L mask =||M gt -G m ||1 Among them, L mask represents the loss of the third decoder, M gt represents the true contour, G m represents the predicted first contour information, and ||.||1 represents the L1 norm.

8. The method for estimating a 3D human body mesh based on clothing shears and a human body visual model according to claim 1, wherein: The calculation formula for the total loss of the iterative optimization parameter submodule is: L smpl =λ d L D +λ m L m +λ j L J in, Among them, L smpl represents the total loss of the iterative optimization parameter submodule, λ d ,λ m and λ j L D , L M , L J The coefficient of G j Represents the predicted first joint heat map information, G d Represents the predicted first depth information, G m Represents the predicted first contour information, Represents the second joint heat map information, Represents the second contour information, Indicates the second depth information.

9. A 3D human body mesh estimation system based on clothing shears and human body visual model, characterized in that: include: A preprocessing module, configured to extract a first human body image including clothing from an image to be reconstructed; The clothing cutting module is used to identify the boundary between the clothing area and the body area in the first human body image, remove the clothing area from the first human body image, and output a second human body image; A three-dimensional human body mesh estimation module, the three-dimensional human body mesh estimation module includes an estimation intermediate representation submodule and an iterative optimization parameter submodule, the estimation intermediate representation submodule predicts the first joint heat map information, first depth information and first contour information of the second human body image based on the human visual model, the iterative optimization parameter submodule is used to obtain an initialized three-dimensional human body mesh from the second human body image, and iteratively optimize the initialized three-dimensional human body mesh. During each round of iterative optimization, the second joint heat map information, second depth information and second contour information of the iteratively optimized three-dimensional human body mesh are extracted, and the total loss of the iterative optimization parameter submodule is jointly calculated based on the first joint heat map information, first depth information, first contour information of the second human body image and the second joint heat map information, second depth information and second contour information of the iteratively optimized three-dimensional human body mesh, and the three-dimensional human body mesh when the iterative optimization stops is used as the three-dimensional human body mesh finally output by the three-dimensional human body mesh estimation module.

Images