3D human modeling method and device based on contrast learning, and medium

Abstract

The application discloses a 3D human body modeling method and device based on contrast learning and a medium. The method comprises the following steps: acquiring 2D image information corresponding to a user and body state information corresponding to the user; extracting original features corresponding to the 2D image information through a shared encoder; performing mask prediction and determining clothing coverage rate according to the original features; performing feature fusion according to the original features, shape prior features corresponding to the body state information and the clothing coverage rate to obtain fused features; performing adaptive feature purification on the fused features according to the clothing coverage rate to obtain pure features; and performing human body parameter prediction according to the pure features.

Description

[0001] This application is based on priority of Chinese Patent Application No. 202510395489.5, filed on March 31, 2025, entitled "3D Human Body Modeling Method, Device and Medium Based on Comparative Learning", the full text of which is incorporated herein by reference. Technical Field

[0002] This application relates to the field of neural networks, specifically to 3D human body modeling methods, devices, and media based on contrastive learning. Background Technology

[0003] In the fields of clothing customization and online shopping, how to accurately obtain a user's body shape parameters from the photos of the clothes they provide and match them with personalized clothing data has always been a technical challenge.

[0004] Traditional methods primarily rely on 2D image analysis techniques to estimate user body dimensions through contour extraction, key point detection (e.g., shoulder width, waist circumference), or template matching algorithms. However, due to the complexity of the human body's 3D form, variations in shooting angles, clothing occlusion, changing lighting conditions, and diverse user postures, the dimensional information extracted from 2D images often contains significant errors, making it difficult to accurately reflect the user's true body shape. For example, loose clothing may lead to misjudgment of contours, while sideways or tilted postures can distort the spatial relationships of key points, further reducing the accuracy of parameter calculations.

[0005] In some scenarios, it may be possible to convert 2D images into 3D human body models and then use these 3D models for clothing data analysis. However, due to the aforementioned issues, the 3D human body modeling results are not accurate enough during the 2D-to-3D conversion process, reducing the precision of parameter calculations. Summary of the Invention

[0006] To address the aforementioned issues, this application proposes a 3D human body modeling method based on contrastive learning, including:

[0007] Acquire 2D image information and body posture information corresponding to the user, and input the 2D image information and body posture information into a pre-trained neural network model;

[0008] Based on the neural network model, the original features corresponding to the 2D image information are extracted through a shared encoder;

[0009] The clothing understanding module performs mask prediction and determines clothing coverage based on the original features.

[0010] Based on the original features, the shape prior features corresponding to the body posture information, and the clothing coverage, feature fusion is performed to obtain fused features;

[0011] Based on the clothing coverage, the fused features are adaptively purified to obtain pure features;

[0012] Based on the purity features, human body parameters are predicted, and 3D human body modeling is performed based on the prediction results.

[0013] In one example, the original features corresponding to the 2D image information are extracted using a shared encoder, specifically including:

[0014] The 2D image information is divided into multiple blocks by using a pre-trained ViT model as the basic feature extractor, and the global dependency between the blocks is determined by self-attention to obtain ViT features.

[0015] Convolutional features corresponding to the 2D image information are extracted using a concatenated convolutional layer structure.

[0016] The ViT features and the convolutional features are weighted and fused to obtain the original features corresponding to the 2D image information.

[0017] In one example, the clothing understanding module performs mask prediction and determines clothing coverage based on the original features, specifically including:

[0018] Based on the original features, feature maps of multiple scales are constructed using the feature pyramid.

[0019] Semantic information is extracted from the feature map using a U-Net-like encoder-decoder structure.

[0020] Based on the extracted semantic information, mask prediction is performed by convolution in the clothing mask branch and the human body mask branch of the dual-branch mask prediction.

[0021] For the predicted mask result, edge features are extracted by an edge detection operator and the edge features are fed back to the shallow layer of the network.

[0022] Output mask results corresponding to multiple scales, and select the mask result corresponding to the highest resolution among the multiple scales. Calculate the percentage of pixels in the clothing area and normalize it to obtain the clothing coverage rate.

[0023] In one example, based on the clothing coverage, the fused features are adaptively purified to obtain pure features, specifically including:

[0024] Based on the clothing coverage and the fusion features, a lightweight MLP model is used to analyze the interference patterns and extract the interference features.

[0025] Using the lightweight MLP model, reset weights are generated based on the interference features and the clothing coverage.

[0026] Based on the reset weights, the fused features are fused with the baseline features corresponding to the body posture information obtained through prior knowledge to obtain pure features.

[0027] In one example, based on the purity features, human parameter prediction is performed, specifically including:

[0028] The shared predictor outputs pose prediction parameters through the first multi-layer MLP model; the second multi-layer MLP model outputs body shape prediction parameters; and the third multi-layer MLP model outputs perspective camera parameters.

[0029] The camera parameters are constrained by scaling factor constraints and translation parameter constraints;

[0030] Based on the posture prediction parameters, the body shape prediction parameters, and the perspective camera parameters, the prediction results corresponding to the obtained human body parameters are output.

[0031] In one example, the training process of the neural network model includes:

[0032] Acquire human body images and perform sample enhancement processing on the human body images;

[0033] For the aforementioned human body image, determine its corresponding data source;

[0034] If the data source is synthetic data, then during the rendering stage, corresponding clothing masks and body masks are generated based on the human body image;

[0035] If the data source is real data, then the corresponding clothing mask and body mask are generated through the pre-trained human body segmentation model;

[0036] Construction of training samples based on human body images;

[0037] For the same user, positive sample pairs are obtained by combining human body images of different dimensions; the different dimensions include: whether there is clothing, the condition of the clothing, and the perspective.

[0038] For different users, if the similarity between their body shapes is higher than the preset similarity, negative sample pairs are obtained by combining them, and the negative sample pairs are then subjected to weight reduction or exclusion.

[0039] In one example, the training process of the neural network model includes:

[0040] By sharing a projection head, training samples with and without clothing are mapped to a common space.

[0041] Within the common space, the pure features are subjected to dimensionality reduction and L2 normalization.

[0042] Based on InfoNCE loss, the similarity between positive and negative sample pairs is calculated, and the sharpness of the feature distribution is adjusted by the temperature parameter τ.

[0043] For each batch of training samples, the neural network model is trained by calculating the cross-entropy loss.

[0044] In one example, the training process of the neural network model includes:

[0045] Based on the clothing coverage, the fused features are adaptively purified to obtain pure features;

[0046] Obtain the corresponding reference features without clothing;

[0047] The neural network model is trained by weighting and summing the pure features and the reference features using direct feature alignment loss, semantic similarity loss, and parameter consistency loss.

[0048] On the other hand, this application also proposes a 3D human body modeling device based on contrastive learning, comprising:

[0049] At least one processor; and,

[0050] A memory communicatively connected to the at least one processor; wherein,

[0051] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to: the contrastive learning-based 3D human body modeling method described in any of the above examples.

[0052] On the other hand, this application also proposes a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as: the 3D human body modeling method based on contrastive learning as described in any of the above examples.

[0053] The 3D human body modeling method based on contrastive learning proposed in this application can bring the following beneficial effects:

[0054] 1. By extracting depth features from 2D images through a shared encoder and combining them with mask prediction and coverage analysis from the clothing understanding module, the system effectively distinguishes between the true outline of the human body and the areas covered by clothing. This significantly reduces the problem of misjudging the outline caused by loose clothing and complex postures, improves the measurement accuracy of key dimensions (such as shoulder width and waist circumference), and achieves high-precision prediction of body shape parameters.

[0055] 2. Introducing shape prior features and adaptive feature purification mechanism, by dynamically weighting the user's body information (e.g., height, weight) and clothing coverage, the impact of lighting changes and shooting angle deviations on the model is reduced, enhancing the algorithm's generalization ability under non-ideal shooting conditions and improving robustness in complex scenes.

[0056] 3. Based on the feature fusion strategy driven by clothing coverage and combined with the contrastive learning framework, it can mine the potential relationship between human body shape and clothing from limited data, accurately model personalized body shape features (such as chest-waist ratio and shoulder slope), break through the adaptation bottleneck of traditional standard size, improve the fit of clothing recommendations, and optimize personalized adaptation capabilities.

[0057] 4. Through an end-to-end neural network model, 3D modeling can be completed with only one or more ordinary 2D images and basic body posture information. It does not rely on high-cost depth sensors or multi-view acquisition equipment, which lowers the hardware threshold and is suitable for rapid deployment in large-scale scenarios such as e-commerce and virtual fitting, ensuring low cost and ease of implementation.

[0058] 5. The adaptive feature purification module can dynamically filter redundant features introduced by noise such as clothing texture and wrinkles, ensuring the stability of human body parameter prediction. It has stronger analytical capabilities, especially for complex clothing (such as multi-layered clothing and irregular cuts), thereby improving anti-interference ability. Attached Figure Description

[0059] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0060] Figure 1 This is a flowchart illustrating the 3D human body modeling method based on contrastive learning in the embodiments of this application;

[0061] Figure 2 This is a detailed flowchart illustrating a 3D human body modeling method based on contrastive learning, as described in one embodiment of this application.

[0062] Figure 3 This is a schematic diagram of a 3D human body modeling device based on contrastive learning in an embodiment of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0064] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0065] like Figure 1 As shown, embodiments of this application provide a 3D human body modeling method based on contrastive learning, including:

[0066] S101: Obtain the 2D image information corresponding to the user and the body posture information corresponding to the user, and input the 2D image information and the body posture information into a pre-trained neural network model.

[0067] Body posture information can include the user's height and weight, and can be further supplemented with more detailed information such as shoulder width, upper body length, and lower body length based on needs.

[0068] like Figure 2 As shown, the neural network model in this application adopts an end-to-end design and is mainly composed of six key modules, forming a complete processing pipeline.

[0069] These include: a shared encoder, a clothing understanding module, a feature fusion layer, a feature purification module, a projection head, and a parameter predictor.

[0070] The shared encoder, as the front end of the entire system, receives the input image and extracts the initial feature representation. For example, the input is a single RGB image (corresponding to the user's 2D image information), and its dimension can be fixed, such as 224×224×3. The output is a high-dimensional visual feature (referred to as the original feature in this application), such as a 1024-dimensional output.

[0071] The clothing understanding module analyzes the clothing state in the input image. Its input is the raw features output by the shared encoder, and its output is the multi-scale predicted masking results (including clothing mask and human body mask) and clothing coverage.

[0072] The feature fusion layer integrates the original features and the shape prior features corresponding to the body posture information. Its inputs are the original features, shape prior features, and clothing coverage, and its output is the fused features, which can be 1024-dimensional.

[0073] The feature purification module dynamically adjusts the feature purification intensity based on clothing coverage. Its inputs are fused features, clothing coverage, and optional clothing-free reference features during the training phase. The output is pure features (which can be 1024-dimensional) that have removed clothing interference.

[0074] The projection head maps high-dimensional features to a low-dimensional space suitable for contrastive learning. Its input is pure features, and its output is a 128-dimensional normalized feature vector.

[0075] The parameter predictor predicts human body parameters from clean features. Its input is clean features, and its output is shape parameters (e.g., set to 10-dimensional), pose parameters (e.g., set to 72-dimensional), and camera parameters (e.g., set to 3-dimensional).

[0076] S102: Based on the neural network model, the original features corresponding to the 2D image information are extracted through a shared encoder.

[0077] Specifically, the shared encoder adopts a hybrid design that combines the advantages of Vision Transformer and Convolutional Neural Networks (CNN).

[0078] By using a pre-trained ViT model as the basic feature extractor, 2D image information is divided into multiple blocks, and the global dependencies between the blocks are determined through self-attention to obtain ViT features.

[0079] In the basic backbone network of the shared encoder, the pre-trained ViT model is used as the basic feature extractor. The input image is divided into multiple patches, and global dependencies are captured through a self-attention mechanism to obtain ViT features.

[0080] By using a concatenated convolutional layer structure, convolutional features corresponding to 2D image information are extracted.

[0081] Enhancement is achieved through auxiliary convolutions, adding a single lightweight convolutional branch to capture local texture details and supplement ViT's global representation capabilities. For example, a 3×3 convolutional layer concatenation structure can be used to preserve local texture information and enhance feature expression.

[0082] By weighted fusion of ViT features and convolutional features, the original features corresponding to the 2D image information are obtained.

[0083] The design features a concise feature fusion module that effectively combines ViT features with convolutional features. It uses a weighted summation method to fuse features from different sources, as shown in Formula 1:

[0084] F orig =w1*F ViT +w2*Fconv Formula 1;

[0085] Among them, F orig For original features, F ViT For ViT features, F conv For convolutional features, w1 and w2 are the weight coefficients corresponding to the ViT feature and the convolutional feature, respectively. These weight coefficients can be learned or preset to ensure that the complementarity of features is fully utilized.

[0086] As for the output dimension, a high-dimensional feature space can fully capture the geometric and appearance information of the human body. However, too high a dimension will increase the complexity of subsequent networks, while too low a dimension may lose detailed information. Therefore, to balance expressiveness and computational efficiency, the output dimension is set to 1024 dimensions.

[0087] S103: Using the clothing understanding module, perform mask prediction and determine clothing coverage based on the original features.

[0088] Specifically, a feature pyramid is used to construct feature maps at multiple scales based on the original features. The feature pyramid structure is employed to build multiple scale feature maps from the encoder features, with each scale corresponding to a different resolution. Each scale has an independent prediction head, enabling multi-scale masked output. For example, low-resolution feature maps (e.g., 7×7) capture the global clothing distribution, medium-resolution feature maps (e.g., 28×28) balance local details with global consistency, and high-resolution feature maps (e.g., 112×112) accurately locate clothing boundaries and subtle regions. This multi-scale architecture ensures the comprehensiveness and accuracy of clothing understanding, thus providing a reliable coverage estimate for subsequent feature purification.

[0089] Semantic information is extracted from feature maps using a U-Net-like encoder-decoder structure. This structure ensures rich detail in the predicted mask. For the encoding path, multi-scale semantic information is extracted through progressive downsampling; for the decoding path, feature map resolution is gradually increased through progressive upsampling to restore spatial details and fuse shallow features with deep semantics.

[0090] Based on the extracted semantic information, the clothing mask branch and the human body mask branch in the dual-branch mask prediction are respectively predicted by convolution. A dual-branch output head is set up, with one branch predicting the clothing mask and the other branch predicting the overall human body mask, thereby realizing dual-branch masking and simultaneously predicting the clothing mask and the human body mask.

[0091] For each prediction head layer, a 1×1 convolution can be used to map to a binary mask space, and auxiliary loss can be added at different scale layers as an intermediate supervision mechanism to improve the mask quality.

[0092] For the predicted mask result, edge features are extracted using edge detection operators and fed back to the shallow layers of the network. For example, Sobel or Canny edge detection operators are used to extract edge features from the mask. These edge features are only used as a preprocessing step to enhance the network's sensitivity to boundary regions. No additional edge supervision data is required; the calculation is based entirely on the mask itself and is not used as a supervision signal in the loss calculation during model training.

[0093] The output includes masking results at multiple scales. For example, the final multi-scale masking prediction result can be: M cloth ={M c1 M c2 ,…,M cn};M human ={M h1 M h2 ,…,M hn}; where M cloth For clothing masking, M cn M is the mask corresponding to the nth scale in the clothing mask. human For human body masking, M hn This is the mask corresponding to the nth scale in the human body mask.

[0094] Among multiple scales, the mask result corresponding to the highest resolution is selected. The clothing coverage is obtained by calculating the percentage of pixels in the clothing area and then normalizing the result. This can be represented by Formula 2:

[0095]

[0096] Where Coverage is the percentage of clothing coverage relative to the human body, M cloth-final M human-final These are the final clothing mask and the final human body mask, respectively, which correspond to the mask result with the highest resolution in the clothing mask and the human body mask.

[0097] After normalizing the clothing coverage, it is mapped to the [0,1] interval, which facilitates subsequent feature purification and control.

[0098] Furthermore, Formula 3 can be used to conduct a more refined assessment of clothing coverage by calculating clothing coverage through regional weighting:

[0099]

[0100] in, For clothing coverage, M cloth [i,j] refers to the clothing mask for the [i,j]th region, with a value range of [0,1]. Mhuman [i,j] refers to the human body mask in the [i,j]th region. W[i,j] refers to the region weight matrix of the [i,j]th region.

[0101] In this process, the masking result is pre-divided into multiple regions using a grid, where the [i,j] region refers to the region corresponding to the grid in the i-th row and j-th column. Compared to simple pixel proportions, the weighted design better meets the needs of human body reconstruction tasks, achieving both coarse-grained segmentation and fine-grained evaluation of the mask.

[0102] S104: Based on the original features, the shape prior features corresponding to the body posture information, and the clothing coverage, feature fusion is performed to obtain fused features.

[0103] Specifically, shape prior features can be obtained by mapping information such as height and weight from body posture information through an encoding network, and then matching them with the original features to facilitate subsequent fusion. For example, users provide body posture information such as height and weight, which is converted into a 256-dimensional feature vector by a shape prior encoder (which can be implemented using an MLP model), and then normalized (e.g., height / 200, weight / 100) to provide an objective body shape reference benchmark, especially providing strong constraints in the case of high coverage.

[0104] During fusion, an attention-guided approach is employed, dynamically adjusting the weights of height and weight information. The influence of shape priors is adaptively adjusted based on clothing coverage; higher clothing coverage results in greater importance for shape prior features. The fused features, encompassing both visual representation and shape prior information, are the primary focus of feature purification.

[0105] The fusion process can be illustrated as shown in Formula 4:

[0106] F fused =F orig +Attn(Coverage)*F shape-prior Formula 4;

[0107] Among them, F fused For fusion features, F orig The original features are represented by Attn, a coverage-based attention function that ensures shape priors are emphasized only when needed, and Coverage represents clothing coverage. shape-prior These are shape prior features.

[0108] In one example, the attention function can be set as shown in Formula 5:

[0109] Attn(Coverage)=sigmoid(5Coverage-2.5) Formula 5;

[0110] Here, sigmoid is the activation function.

[0111] S105: Based on the clothing coverage, perform adaptive feature purification on the fused features to obtain pure features.

[0112] Specifically, for clothing coverage and fusion features, a lightweight MLP model is used to analyze interference patterns and extract interference features. Feature purification is mainly used to remove clothing interference from clothing features. An interference analysis network is set up, whose inputs are fusion features and clothing coverage. Using shape prior features as a reference, possible body shapes under clothing are inferred. An interference pattern is analyzed by a lightweight MLP model to extract interference features, and the interference features are automatically scaled according to the coverage.

[0113] At this point, the interference analysis will combine the fusion features (1024 dimensions) and clothing coverage (1 dimension), as shown in Formula Six:

[0114] F dist =Tanh(MLP([F fused Formula Six;

[0115] Among them, F dist As an interference feature, Tanh is the hyperbolic tangent function, F fused For feature fusion, Coverage represents clothing coverage, and MLP is a lightweight MLP model. The interference feature represents clothing interference information that needs to be removed from the original features; it increases with increasing coverage and automatically approaches zero when the coverage is low.

[0116] When adjusting clothing coverage, interference features can be dynamically adjusted using Formula 7:

[0117] F dist =F dist *Sigmoid(10Coverage-5) Formula 7.

[0118] Using a lightweight MLP model, reset weights are generated based on interference features and clothing coverage. The reset weights are shown in Formula 8.

[0119] R = Sigmoid(MLP([F fused ;F dist Formula 8;

[0120] Where R is the reset weight, F fused For fusion features, F dist This is a feature of interference.

[0121] When adjusting clothing coverage, the reset weight can be dynamically adjusted using Formula Nine:

[0122] R = R * sigmoid(10Coverage - 5) (Formula Nine)

[0123] Thus, when clothing coverage is low (e.g., less than 0.3), sigmoid(10*0.3-5)≈0.007, almost no feature reset is performed; when clothing coverage is medium (e.g., equal to 0.5), sigmoid(10*0.5-5)≈0.5, partially performing feature reset; and when coverage is high (e.g., greater than 0.7), sigmoid(10*0.7-5)≈0.993, almost completely performing feature reset. This achieves a smooth transition based on clothing status, avoiding the discontinuity problem caused by hard threshold classification. In this case, the reset weights are dynamically adjusted according to coverage, and images without clothing are hardly modified.

[0124] Based on the reset weights, the fused features are fused with the baseline features corresponding to the body posture information obtained through prior knowledge to obtain pure features.

[0125] The process of obtaining pure features can be called feature purification. During feature purification, feature reset weights are calculated based on interference analysis, and the feature purification formula is shown in Formula 10:

[0126] F pure =F fused *(1-R)+F ref *R Formula 10;

[0127] Among them, F pure As a characteristic of purity, F fused For feature fusion, R is the reset weight, and F is the weight. ref The baseline features are the basic human body features learned by incorporating prior knowledge of body posture information, used to describe the basic structure of the human body.

[0128] Among them, the benchmark feature F ref As a feature benchmark for network parameter learning, its learning mechanism can be set as follows: initialized with random values, for example, a normal distribution (N(0,0.01)), and learned through end-to-end training to obtain an abstract representation of "ideal unclothed features" without explicit supervision. Unlike directly using specific unclothed sample features, the benchmark features represent a broader priori understanding of human morphology. As a shared reference point for all inputs, the benchmark features provide an implicit representation of "pure human body," ensuring the consistency of feature resetting. Furthermore, in its dynamic operation, it, along with the resetting weights and clothing coverage, determines the intensity and method of feature purification. In the case of no clothing or low coverage, the original features are mainly retained, while in the case of high coverage, a stronger feature purification is performed to remove more clothing interference information.

[0129] S106: Based on the pure features, predict human body parameters and perform 3D human body modeling based on the prediction results.

[0130] Specifically, the shared predictor outputs pose prediction parameters through a first-layer MLP model, body shape prediction parameters through a second-layer MLP model, and perspective camera parameters through a third-layer MLP model. The shared predictor enables unclothed features and purely clothed features to share the same parameter prediction network, ensuring consistent body parameter outputs from different feature sources.

[0131] Parameter predictors, also known as differentiable parameterized human models, primarily refer to shape vectors. and attitude vector A statistical model of a 3D human body mesh V is generated using a linear hybrid skinning function Φ(β,θ). Here, β is a shape vector representing the body's shape features, K represents the dimension of the shape parameters (which can be determined through principal component analysis), θ is a pose vector representing the body's posture, and L depends on the number of joints and the parameter representation of each joint.

[0132] Parametric human models can employ models such as SMPL, SMPL-X, STAR, and GHUM. The parametric prediction architecture includes:

[0133] Pose prediction network: corresponding to the first multilayer MLP model, outputting 72-dimensional pose parameters; Shape prediction network: corresponding to the second multilayer MLP model, outputting 10-dimensional shape parameters; Camera parameter prediction network: corresponding to the third multilayer MLP, outputting 3-dimensional weak perspective camera parameters.

[0134] Camera parameters are constrained by scaling factor constraints and translation parameter constraints. The scaling factor is mapped to the range of 0 to 2 using the sigmoid function, and the translation parameter is limited to the range of [-1,1] using the tanh function, thereby ensuring that the predicted parameters conform to physical meaning and facilitate subsequent rendering and evaluation.

[0135] Based on the posture prediction parameters, body shape prediction parameters, and perspective camera parameters, the system outputs the prediction results corresponding to the obtained human body parameters.

[0136] 1. By extracting depth features from 2D images through a shared encoder and combining them with mask prediction and coverage analysis from the clothing understanding module, the system effectively distinguishes between the true outline of the human body and the areas covered by clothing. This significantly reduces the problem of misjudging the outline caused by loose clothing and complex postures, improves the measurement accuracy of key dimensions (such as shoulder width and waist circumference), and achieves high-precision prediction of body shape parameters.

[0137] 2. Introducing shape prior features and adaptive feature purification mechanism, by dynamically weighting the user's body information (e.g., height, weight) and clothing coverage, the impact of lighting changes and shooting angle deviations on the model is reduced, enhancing the algorithm's generalization ability under non-ideal shooting conditions and improving robustness in complex scenes.

[0138] 3. Based on the feature fusion strategy driven by clothing coverage and combined with the contrastive learning framework, it can mine the potential relationship between human body shape and clothing from limited data, accurately model personalized body shape features (such as chest-waist ratio and shoulder slope), break through the adaptation bottleneck of traditional standard size, improve the fit of clothing recommendations, and optimize personalized adaptation capabilities.

[0139] 4. Through an end-to-end neural network model, 3D modeling can be completed with only one or more ordinary 2D images and basic body posture information. It does not rely on high-cost depth sensors or multi-view acquisition equipment, which lowers the hardware threshold and is suitable for rapid deployment in large-scale scenarios such as e-commerce and virtual fitting, ensuring low cost and ease of implementation.

[0140] 5. The adaptive feature purification module can dynamically filter redundant features introduced by noise such as clothing texture and wrinkles, ensuring the stability of human body parameter prediction. It has stronger analytical capabilities, especially for complex clothing (such as multi-layered clothing and irregular cuts), thereby improving anti-interference ability.

[0141] In one embodiment, the neural network model employs a unified forward propagation design, a standardized feature processing pipeline, and all inputs undergo the same processing. It also incorporates an adaptive feature processing mechanism and multi-view feature alignment, and supports diverse clothing states.

[0142] For the adaptive feature processing mechanism, continuous values ​​are used to represent clothing coverage, and the processing intensity is dynamically adjusted through the adaptive mechanism; clothing coverage is directly calculated from the mask without the need for complex estimation; the processing flow is completely consistent, only handling the dynamic changes in intensity, ensuring the simplicity and stability of the inference process.

[0143] For multi-view feature alignment, it supports comparative learning of images of the same human body from different viewpoints.

[0144] For diverse clothing states, including different perspectives with and without clothing, and between with and without clothing, all samples undergo the same feature extraction and processing procedures to ensure consistency in the feature space.

[0145] In one embodiment, during the training of the neural network model, human images are acquired and subjected to sample augmentation processing. For example, data augmentation operations such as scaling, random cropping, horizontal flipping, and color dithering are performed on the images to ensure that the model is robust to changes in viewpoint, lighting, and occlusion.

[0146] For human body images, determine their corresponding data sources.

[0147] If the data source is synthetic data, then during the rendering stage, corresponding clothing masks and body masks are generated based on the human body image; if the data source is real data, then corresponding clothing masks and body masks are generated through a pre-trained human body segmentation model or through manual annotation.

[0148] Furthermore, it is necessary to ensure the quality of the mask, with special attention paid to the edge areas of the clothing, in order to accurately calculate the coverage of the clothing.

[0149] Construction of training samples based on human images.

[0150] For the same user, positive sample pairs are obtained by combining human images from different dimensions; these dimensions include: presence of clothing, clothing condition, and perspective. This enhances the model's ability to capture inherent human features.

[0151] For different users, if the similarity between their body shapes is higher than a preset similarity, negative sample pairs are generated and then either downweighted or excluded. This reduces confusion caused by body shape similarity.

[0152] During training, positive and negative sample pairs are sampled in batches, and the feature purification module and parameter prediction module are jointly optimized through comparative learning and multiple supervision constraints.

[0153] During the training phase, multiple loss functions can be set, such as: Parameter prediction loss: using MSE to measure the deviation between predicted and true parameters; Mask prediction loss: supervising the prediction quality of clothing and human body masks, using binary cross-entropy loss; Contrast loss (InfoNCE): improving the discriminative power of human body features through positive and negative sample pairs; Feature alignment loss: aligning purified features with features without clothing when using reference features without clothing; Camera parameter loss: limiting weak perspective camera parameters within a reasonable range to enhance projection accuracy; Vertex loss: L vertex =MSE(V pred V gt This aligns the predicted mesh vertices with the actual mesh vertices; Joint 2D loss: L joint2d =MSE(J2D) pred J2D gt To ensure accurate joint positioning projected onto the image plane; Joint 3D loss: L joint3d =MSE(J3D)pred J3D gt Align key joints in three-dimensional space. Where L... vertex For fixed-point loss, V pred V represents the predicted grid vertex coordinates. gt L represents the vertex coordinates of the real mesh. joint2d For joint 2D loss, J2D pred For the predicted coordinates of the key points on the 2D image plane, J2D gt L represents the coordinates of the actual joints on the two-dimensional image plane. joint3d For joint 3D loss, J3D pred For the predicted coordinates of the joints on the 3D image plane, J3D gt These are the coordinates of the actual key points on the 3D image plane. Ultimately, only the core losses are retained: contrast loss, direct feature alignment loss, parameter prediction loss, and mask prediction loss.

[0154] In one embodiment, during the training of the neural network model, training samples with and without clothing are mapped to a common space using a shared projection head. The training samples without clothing are used only during training.

[0155] Within the shared feature space, clean features are dimensionality-reduced and L2 normalized. The shared projection head uses the same projection network for both features without clothing and features with clothing. It is configured as a two-layer MLP structure, consisting of a Linear layer, a BN layer, a ReLU layer, another Linear layer, and an L2 normalization layer in sequence. This ensures that features from different sources are mapped to the same feature space, which is beneficial for feature alignment.

[0156] Features of unclothed and clothed samples are mapped to a common space and contrast loss is calculated. Based on InfoNCE loss, the similarity between positive and negative sample pairs is calculated, and the sharpness of the feature distribution is adjusted by the temperature parameter τ.

[0157] For each batch of training samples, the neural network model is trained by calculating the cross-entropy loss.

[0158] The InfoNCE loss is shown in Formula 11:

[0159]

[0160] Among them, f q To query sample features, For positive sample features, For negative sample features, τ is the temperature parameter, with a default value of 0.07.

[0161] In one embodiment, during the training of the feature purification module, unclothed reference features can also be introduced as additional guidance to improve purification quality through direct feature alignment loss.

[0162] During the training of the neural network model, the fused features are adaptively purified based on the clothing coverage to obtain pure features.

[0163] During the training phase, when paired unclothed images are available, their features are used as direct references for clean features. Corresponding unclothed reference features are then obtained. These reference features can be obtained from unclothed images of the same person processed by the same encoder, and are only available from a subset of samples during the training phase.

[0164] The neural network model is trained by weighting and summing the direct feature alignment loss, semantic similarity loss, and parameter consistency loss for the pure features and the reference features.

[0165] Among them, the direct feature alignment loss can be the L2 distance loss, the semantic similarity loss can be the cosine similarity loss, and the parameter consistency loss can be included at the structural feature level through the structural similarity index between feature maps.

[0166] At the level of original features, calculate the L2 distance loss: L direct =||F purified –F naked || 2 At the semantic feature level, the cosine similarity loss is calculated: L direct-cos =1–cos(F purified ,F naked At the structural feature level, multi-level feature alignment is achieved through structural similarity metrics between feature maps (e.g., SSIM applied to features).

[0167] Among them, L direct For L2 distance loss, L direct-cos For cosine similarity loss, F purified For calculating the purity characteristics, F aaked Reference features.

[0168] Based on this, we set the feature alignment comprehensive loss L. align =α*L direct +β*L cos+γ *L consist The default weights are: α = 1.0, β = 0.5, and γ = 0.2.

[0169] In the early stages of training, forced feature alignment is implemented using a high-weight L2 distance loss. In the mid-stage, feature alignment is balanced with other objectives (e.g., the accuracy of parameter prediction). In the late stage, slight feature guidance is used, relying primarily on the purification capabilities learned by the model. The inference stage, however, does not require unclothed images and relies entirely on the trained model for feature purification.

[0170] In addition, attention can be set for body parts, and attention weights for each body part can be predicted based on pure features and height and weight priors. Anatomical guidance can also be provided to assist in subsequent parameter prediction.

[0171] like Figure 3 As shown in the illustration, this application also provides a 3D human body modeling device based on contrastive learning, comprising:

[0172] At least one processor; and,

[0173] A memory communicatively connected to the at least one processor; wherein,

[0174] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to: perform the contrastive learning-based 3D human body modeling method as described in any of the above embodiments.

[0175] This application also provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as: the 3D human body modeling method based on contrastive learning as described in any of the above embodiments.

[0176] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0177] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0178] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0179] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0180] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0181] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0182] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0183] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0184] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0185] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0186] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A 3D human body modeling method based on contrastive learning, characterized in that, include: Acquire 2D image information and body posture information corresponding to the user, and input the 2D image information and body posture information into a pre-trained neural network model; Based on the neural network model, the original features corresponding to the 2D image information are extracted through a shared encoder; The clothing understanding module performs mask prediction and determines clothing coverage based on the original features, specifically including: Based on the original features, feature maps of multiple scales are constructed using the feature pyramid. Semantic information is extracted from the feature map using a U-Net-like encoder-decoder structure. Based on the extracted semantic information, mask prediction is performed by convolution in the clothing mask branch and the human body mask branch of the dual-branch mask prediction. For the predicted mask result, edge features are extracted by an edge detection operator and the edge features are fed back to the shallow layer of the network. Output mask results corresponding to multiple scales, and select the mask result corresponding to the highest resolution among the multiple scales. Calculate the percentage of pixels in the clothing area and normalize it to obtain the clothing coverage rate. Based on the original features, the shape prior features corresponding to the body posture information, and the clothing coverage, feature fusion is performed to obtain fused features; Based on the clothing coverage, the fused features are adaptively purified to obtain pure features, specifically including: Based on the clothing coverage and the fusion features, a lightweight MLP model is used to analyze the interference patterns and extract the interference features. Using the lightweight MLP model, reset weights are generated based on the interference features and the clothing coverage. Based on the reset weights, the fused features are fused with the baseline features corresponding to the body posture information obtained through prior knowledge to obtain pure features; Based on the purity features, human body parameters are predicted, and 3D human body modeling is performed based on the prediction results.

2. The method according to claim 1, characterized in that, By using a shared encoder, the original features corresponding to the 2D image information are extracted, specifically including: The 2D image information is divided into multiple blocks by using a pre-trained ViT model as the basic feature extractor, and the global dependency between the blocks is determined by self-attention to obtain ViT features. Convolutional features corresponding to the 2D image information are extracted using a concatenated convolutional layer structure. The ViT features and the convolutional features are weighted and fused to obtain the original features corresponding to the 2D image information.

3. The method according to claim 1, characterized in that, Based on the purity characteristics, human body parameter prediction is performed, specifically including: The shared predictor outputs pose prediction parameters through the first multi-layer MLP model; the second multi-layer MLP model outputs body shape prediction parameters; and the third multi-layer MLP model outputs perspective camera parameters. The camera parameters are constrained by scaling factor constraints and translation parameter constraints; Based on the posture prediction parameters, the body shape prediction parameters, and the perspective camera parameters, the prediction results corresponding to the obtained human body parameters are output.

4. The method according to claim 1, characterized in that, The training process of the neural network model includes: Acquire human body images and perform sample enhancement processing on the human body images; For the aforementioned human body image, determine its corresponding data source; If the data source is synthetic data, then during the rendering stage, corresponding clothing masks and body masks are generated based on the human body image; If the data source is real data, then the corresponding clothing mask and body mask are generated through the pre-trained human body segmentation model; Construction of training samples based on human body images; For the same user, positive sample pairs are obtained by combining human body images of different dimensions; the different dimensions include: whether there is clothing, the condition of the clothing, and the perspective. For different users, if the similarity between their body shapes is higher than the preset similarity, negative sample pairs are obtained by combining them, and the negative sample pairs are then subjected to weight reduction or exclusion.

5. The method according to claim 1, characterized in that, The training process of the neural network model includes: By sharing a projection head, training samples with and without clothing are mapped to a common space. Within the common space, the pure features are subjected to dimensionality reduction and L2 normalization. Based on InfoNCE loss, the similarity between positive and negative sample pairs is calculated, and the sharpness of the feature distribution is adjusted by the temperature parameter τ. For each batch of training samples, the neural network model is trained by calculating the cross-entropy loss.

6. The method according to claim 1, characterized in that, The training process of the neural network model includes: Based on the clothing coverage, the fused features are adaptively purified to obtain pure features; Obtain the corresponding reference features without clothing; The neural network model is trained by weighting and summing the pure features and the reference features using direct feature alignment loss, semantic similarity loss, and parameter consistency loss.

7. A 3D human body modeling device based on contrastive learning, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to: the 3D human body modeling method based on contrastive learning as described in any one of claims 1 to 6.

8. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured as follows: the 3D human body modeling method based on contrastive learning as described in any one of claims 1 to 6.

Images