3D human body modeling method and device based on comparative learning and medium

Abstract

The invention discloses a 3D human body modeling method and device based on comparative learning and a medium. The method comprises the steps that 2D image information corresponding to a user and posture information corresponding to the user are acquired; extracting an original feature corresponding to the 2D image information through a shared encoder; according to the original features, mask prediction is carried out, and the clothing coverage rate is determined; performing feature fusion according to the original feature, a shape prior feature corresponding to the posture information and the clothing coverage rate to obtain a fused feature; according to the clothing coverage rate, performing adaptive feature purification on the fused feature to obtain a pure feature; and performing human body parameter prediction according to the pure features.

Description

[0001] This application is based on the priority claim of an invention patent application submitted to the China Patent Office on March 31, 2025, with application number 202510395489.5 and invention name "3D human body modeling method, equipment and medium based on contrastive learning". This application cites the full text of the above-mentioned Chinese patent application. Technical Field

[0002] The present application relates to the field of neural networks, and specifically to a 3D human body modeling method, device, and medium based on contrastive learning. Background Art

[0003] In the field of clothing customization and online shopping, how to accurately obtain the body parameters of users through the clothing pictures provided by them and match them with personalized clothing data has always been a technical difficulty.

[0004] Traditional methods rely primarily on two-dimensional image analysis techniques to estimate user body dimensions through body contour extraction, key point detection (e.g., shoulder width, waist circumference), or template matching algorithms. However, due to factors such as the complexity of the human body's three-dimensional form, variations in shooting angles, clothing occlusion, variable lighting conditions, and the diversity of user postures, the dimensional information extracted from two-dimensional images often contains significant errors, making it difficult to accurately reflect the user's true body characteristics. For example, loose clothing can lead to misjudgment of the outline, while sideways or tilted postures can distort the spatial relationships between key points, further reducing the accuracy of parameter calculations.

[0005] In some scenarios, it's possible to convert 2D images into 3D human models and use them to analyze clothing data. However, the aforementioned issues can still lead to inaccurate 3D human modeling during the 2D-to-3D conversion process, reducing parameter calculation accuracy. Summary of the Invention

[0006] To solve the above problems, this application proposes a 3D human body modeling method based on contrastive learning, including:

[0007] Obtaining 2D image information corresponding to the user and body posture information corresponding to the user, and inputting the 2D image information and the body posture information into a pre-trained neural network model;

[0008] Extracting original features corresponding to the 2D image information through a shared encoder based on the neural network model;

[0009] Through the clothing understanding module, mask prediction and clothing coverage are performed based on the original features;

[0010] Perform feature fusion based on the original features, the shape prior features corresponding to the body information, and the clothing coverage to obtain a fused feature;

[0011] According to the clothing coverage, the fusion features are adaptively purified to obtain pure features;

[0012] Based on the pure features, human body parameters are predicted, and 3D human body modeling is performed based on the prediction results.

[0013] In one example, extracting original features corresponding to the 2D image information through a shared encoder specifically includes:

[0014] Using the pre-trained ViT model as the basic feature extractor, the 2D image information is divided into multiple blocks, and the global dependency between the blocks is determined through self-attention to obtain ViT features;

[0015] Extracting convolution features corresponding to the 2D image information through a convolutional layer series structure;

[0016] The ViT feature and the convolution feature are weightedly fused to obtain the original feature corresponding to the 2D image information.

[0017] In one example, the clothing understanding module performs mask prediction and clothing coverage determination based on the original features, specifically including:

[0018] Constructing feature maps of multiple scales based on the original features through a feature pyramid;

[0019] Extracting semantic information based on the feature map through a U-Net-like codec structure;

[0020] Based on the extracted semantic information, the clothing mask branch and the human mask branch in the dual-branch mask prediction are used to perform mask prediction through convolution respectively;

[0021] For the predicted mask results, edge features are extracted through edge detection operators and fed back to the shallow layer of the network;

[0022] Output the mask results corresponding to multiple scales, and select the mask result corresponding to the highest resolution among the multiple scales. Calculate the percentage of pixels in the clothing area and normalize it to obtain the clothing coverage.

[0023] In one example, adaptive feature purification is performed on the fused features according to the clothing coverage to obtain pure features, specifically including:

[0024] Analyze interference patterns and extract interference features using a lightweight MLP model based on the clothing coverage and the fusion features;

[0025] Generating a reset weight according to the interference feature and the clothing coverage by the lightweight MLP model;

[0026] According to the reset weight, the fused feature is fused with the reference feature corresponding to the posture information obtained through prior knowledge to obtain a pure feature.

[0027] In one example, human body parameter prediction is performed based on the pure features, specifically including:

[0028] Through the shared predictor, the first multi-layer MLP model outputs the posture prediction parameters; through the second multi-layer MLP model, the body shape prediction parameters are output; and through the third multi-layer MLP model, the perspective camera parameters are output;

[0029] Constraining the camera parameters by means of zoom factor constraints and translation parameter constraints;

[0030] According to the posture prediction parameters, the body shape prediction parameters, and the perspective camera parameters, a prediction result corresponding to the obtained human body parameter prediction is output.

[0031] In one example, the training process of the neural network model includes:

[0032] Acquiring a human body image, and performing sample enhancement processing on the human body image;

[0033] For the human body image, determining a corresponding data source;

[0034] If the data source is synthetic data, then in the rendering stage, corresponding clothing masks and human body masks are generated according to the human body image;

[0035] If the data source is real data, the corresponding clothing mask and body mask are generated through the pre-trained human segmentation model;

[0036] Constructing training samples based on human body images;

[0037] Among them, for the same user, the corresponding human body images of different dimensions are combined to obtain positive sample pairs; the different dimensions include: whether there is clothing, clothing status, and viewing angle;

[0038] For different users, if the similarity between their body shapes is higher than a preset similarity, negative sample pairs are obtained by combination, and the negative sample pairs are downgraded or excluded.

[0039] In one example, the training process of the neural network model includes:

[0040] By sharing the projection head, the training samples with clothing and without clothing are mapped to a common space;

[0041] In the common space, the pure features are subjected to dimensionality reduction processing and L2 normalization;

[0042] Based on InfoNCE loss, the similarity between positive sample pairs and negative sample pairs is calculated, and the sharpness of the feature distribution is adjusted by the temperature parameter τ;

[0043] For each batch of training samples, the neural network model is trained by calculating the cross entropy loss.

[0044] In one example, the training process of the neural network model includes:

[0045] According to the clothing coverage, the fusion features are adaptively purified to obtain pure features;

[0046] Obtain the corresponding reference features without clothing;

[0047] The neural network model is trained by performing weighted summation of the pure features and the reference features through direct feature alignment loss, semantic similarity loss, and parameter consistency loss.

[0048] On the other hand, this application also proposes a 3D human body modeling device based on contrastive learning, comprising:

[0049] at least one processor; and,

[0050] a memory communicatively connected to the at least one processor; wherein,

[0051] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: the 3D human body modeling method based on contrast learning as described in any of the above examples.

[0052] On the other hand, the present application also proposes a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as: a 3D human body modeling method based on contrastive learning as described in any of the above examples.

[0053] The 3D human body modeling method based on contrastive learning proposed in this application can bring the following beneficial effects:

[0054] 1. By extracting deep features from 2D images through a shared encoder and combining it with mask prediction and coverage analysis from the clothing understanding module, it effectively distinguishes the true human body contour from areas obscured by clothing, significantly reducing contour misjudgments caused by loose clothing and complex postures, improving the measurement accuracy of key dimensions (e.g., shoulder width and waist circumference), and achieving high-precision body parameter prediction.

[0055] 2. Introducing shape prior features and an adaptive feature purification mechanism, by integrating user body information (e.g., height and weight) with dynamic weight distribution of clothing coverage, we can reduce the impact of lighting changes and shooting angle deviations on the model, enhance the algorithm's generalization ability under non-ideal shooting conditions, and improve its robustness in complex scenes.

[0056] 3. A feature fusion strategy driven by clothing coverage, combined with a contrastive learning framework, can mine the potential correlation between human body shape and clothing from limited data, accurately model personalized body features (e.g., chest-to-waist ratio, shoulder slope), break through the adaptation bottleneck of traditional standard sizes, improve the fit of clothing recommendations, and optimize personalized adaptation capabilities.

[0057] 4. Through the end-to-end neural network model, 3D modeling can be completed with only one or multiple ordinary 2D images and basic body posture information, without relying on high-cost depth sensors or multi-view acquisition equipment, lowering the hardware threshold and being suitable for rapid deployment in large-scale scenarios such as e-commerce and virtual fitting, ensuring low cost and easy implementation.

[0058] 5. The adaptive feature purification module can dynamically filter out redundant features introduced by noise such as clothing texture and wrinkles to ensure the stability of human body parameter prediction. In particular, it has stronger parsing capabilities for complex clothing scenes (for example, multi-layered clothing and irregular cuts), thereby improving anti-interference capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0060] Figure 1 Schematic diagram of the process of a 3D human body modeling method based on contrastive learning in an embodiment of the present application;

[0061] Figure 2 A detailed flowchart of a 3D human body modeling method based on contrastive learning in one embodiment of the present application is shown;

[0062] Figure 3 Schematic diagram of a 3D human body modeling device based on contrastive learning in an embodiment of the present application. DETAILED DESCRIPTION

[0063] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0064] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0065] like Figure 1 As shown, the embodiment of the present application provides a 3D human body modeling method based on contrastive learning, including:

[0066] S101: Acquire 2D image information corresponding to a user and body posture information corresponding to the user, and input the 2D image information and the body posture information into a pre-trained neural network model.

[0067] Body shape information may include the user's height information, weight information, etc., and may further include more detailed information such as shoulder width, upper body length, lower body length, etc. based on needs.

[0068] like Figure 2 As shown, the model architecture of the neural network model in this application adopts an end-to-end design and is mainly composed of six key modules to form a complete processing pipeline.

[0069] These include: shared encoder, clothing understanding module, feature fusion layer, feature purification module, projection head, and parameter predictor.

[0070] Among them, the shared encoder serves as the front end of the entire system, receiving the input image and extracting the initial feature representation. For example, the input is a single RGB image (corresponding to the user's 2D image information), whose dimension can be fixed, for example, set to 224×224×3, and the output is a high-dimensional visual feature (referred to as the original feature in this application), for example, set to a 1024-dimensional output.

[0071] The clothing understanding module analyzes the clothing state in the input image. Its input is the original features output by the shared encoder, and its output is the multi-scale predicted mask results (including clothing mask, human body mask) and clothing coverage.

[0072] The feature fusion layer integrates the original features and the shape prior features corresponding to the body information. Its input is the original features, shape prior features, and clothing coverage, and its output is the fused features, for example, it can be 1024 dimensions.

[0073] The feature purification module dynamically adjusts the feature purification strength based on clothing coverage. Its input is the fusion feature, clothing coverage, and the optional no-clothing reference feature during the training phase. The output is the pure feature (which can be 1024-dimensional) with clothing interference removed.

[0074] The projection head maps high-dimensional features to a low-dimensional space suitable for contrastive learning. Its input is pure features and its output is a 128-dimensional normalized feature vector.

[0075] The parameter predictor predicts human body parameters from clean features, where the input is the clean features and the output is shape parameters (e.g., set to 10 dimensions), pose parameters (e.g., set to 72 dimensions), and camera parameters (e.g., set to 3 dimensions).

[0076] S102: Based on the neural network model, extract original features corresponding to the 2D image information through a shared encoder.

[0077] Specifically, the shared encoder adopts a hybrid design that combines the advantages of Vision Transformer and Convolutional Neural Networks (CNN).

[0078] The pre-trained ViT model is used as the basic feature extractor to divide the 2D image information into multiple blocks, and the global dependencies between the blocks are determined through self-attention to obtain the ViT features.

[0079] In the basic backbone network of the shared encoder, the pre-trained ViT model is used as the basic feature extractor, the input image is divided into multiple blocks (patches), and the global dependencies are captured through the self-attention mechanism to obtain ViT features.

[0080] Through the convolutional layer series structure, the convolution features corresponding to the 2D image information are extracted.

[0081] Enhanced by auxiliary convolution, a single lightweight convolution branch is added to capture local texture details and supplement the global representation capability of ViT. For example, a 3×3 convolution layer series structure is used to retain local texture information and enhance feature expression.

[0082] ViT features and convolution features are weightedly fused to obtain the original features corresponding to the 2D image information.

[0083] A simple feature fusion module is designed to effectively combine ViT features with convolutional features, and adopt a weighted summation method to fuse features from different sources, as shown in Formula 1:

[0084] F orig =w1*F ViT +w2*Fconv Formula 1;

[0085] Among them, F orig is the original feature, F ViT is the ViT feature, F conv is the convolution feature, w1 and w2 are the weight coefficients corresponding to the ViT feature and the convolution feature, respectively. The weight coefficients can be learned or preset to ensure that the complementarity of features is fully utilized.

[0086] As for the output dimension, a high-dimensional feature space can fully capture the geometric and appearance information of the human body. However, too high a dimension will increase the complexity of the subsequent network, and too low a dimension may lose detailed information. Therefore, considering the balance between expressiveness and computational efficiency, the output dimension is set to 1024 dimensions.

[0087] S103: Using a clothing understanding module, mask prediction is performed based on the original features and clothing coverage is determined.

[0088] Specifically, through the feature pyramid, feature maps of multiple scales are constructed based on the original features. The feature pyramid structure is used to construct multiple scale feature maps from the encoder features, and the feature maps of each scale correspond to different resolutions. Each scale has an independent prediction head to achieve multi-scale mask output. For example, low-resolution feature maps (e.g., 7×7) capture the global clothing distribution, medium-resolution feature maps (e.g., 28×28) balance local details with global consistency, and high-resolution feature maps (e.g., 112×112) accurately locate clothing boundaries and subtle areas. This multi-scale architecture ensures the comprehensiveness and accuracy of clothing understanding, thereby providing reliable coverage estimation for subsequent feature purification.

[0089] Using a U-Net-like codec structure, semantic information is extracted from feature maps. This architecture ensures rich predicted mask details. In the encoding path, multi-scale semantic information is extracted through progressive downsampling. In the decoding path, progressive upsampling increases the feature map resolution, restores spatial details, and fuses shallow features with deep semantics.

[0090] Based on the extracted semantic information, the clothing mask branch and the human mask branch in the dual-branch mask prediction are used for mask prediction through convolution. A dual-branch output head is set up, one branch predicts the clothing mask, and the other branch predicts the entire human mask, thus achieving dual-branch masking and simultaneous prediction of clothing and human masks.

[0091] For each layer of prediction head, 1×1 convolution can be used to map to the binary mask space, and auxiliary losses can be added at different scale layers as an intermediate supervision mechanism to improve the mask quality.

[0092] Edge features are extracted from the predicted mask using edge detection operators and fed back to the shallow layers of the network. For example, Sobel or Canny edge detection operators are used to extract edge features from the mask. These edge features serve only as a preprocessing step to enhance the network's sensitivity to boundary regions. They do not require additional edge supervision data and are calculated entirely based on the mask itself. They do not serve as supervisory signals in loss calculations during model training.

[0093] Output the mask results corresponding to multiple scales. For example, the final output multi-scale mask prediction result can be: M cloth ={M c1 ,M c2 ,…,M cn};M human ={M h1 ,M h2 ,…,M hn}; Among them, M cloth For clothing mask, M cn is the mask corresponding to the nth scale in the clothing mask, M human For human body mask, M hn is the mask corresponding to the nth scale in the human body mask.

[0094] Among multiple scales, the mask result corresponding to the highest resolution is selected, and the clothing coverage is obtained by calculating the percentage of pixels in the clothing area and normalizing it. It can be shown as Formula 2:

[0095]

[0096] Among them, Coverage is the coverage of clothing, which means the coverage of clothing relative to the human body, M cloth-final 、M human-final They are the final clothing mask and the final human body mask, which correspond to the mask results corresponding to the highest resolution among the clothing mask and the human body mask.

[0097] After normalizing the clothing coverage, it is mapped to the [0,1] interval to facilitate subsequent feature purification and control.

[0098] Furthermore, we can use Formula 3 to make a more refined assessment of clothing coverage and calculate clothing coverage by regional weighting:

[0099]

[0100] in, is the clothing coverage, M cloth [i,j] refers to the clothing mask of the [i,j]th region, and its value range is [0,1]. Mhuman [i,j] refers to the human body mask of the [i,j]th region, W[i,j] refers to the region weight matrix of region [i,j].

[0101] The mask result is pre-divided into multiple regions using a grid. Region [i, j] refers to the region corresponding to the grid in row i and column j. Compared to simple pixel ratios, a weighted design better meets the requirements of human body reconstruction tasks, enabling both coarse-grained segmentation and refined evaluation of the mask.

[0102] S104: performing feature fusion according to the original features, the shape prior features corresponding to the body information, and the clothing coverage to obtain fused features.

[0103] Specifically, shape prior features can be obtained by mapping body information such as height and weight through an encoding network, matching them with the original features to facilitate subsequent fusion. For example, a user provides body information such as height and weight, which is converted into a 256-dimensional feature vector through a shape prior encoder (which can be implemented through an MLP model). This is then normalized (for example, height / 200, weight / 100) to provide an objective body shape reference, providing strong constraints, especially in high coverage situations.

[0104] During fusion, an attention-guided approach is used to dynamically adjust the weight of height and weight information, and adaptively adjust the influence of shape priors based on clothing coverage. The higher the clothing coverage, the more important the shape priors are. The fused features, which combine visual representation and shape prior information, are the primary focus of feature purification.

[0105] The fusion process can be expressed as formula 4:

[0106] F fused =F orig +Attn(Coverage)*F shape-prior Formula 4;

[0107] Among them, F fused is the fusion feature, F orig is the original feature, Attn is the coverage-based attention function, ensuring that the shape prior is emphasized only when necessary, Coverage is the clothing coverage, and F shape-prior is the shape prior feature.

[0108] In one example, the attention function can be set as shown in Formula 5:

[0109] Attn(Coverage)=sigmoid(5Coverage-2.5) Formula 5;

[0110] Among them, sigmoid is the activation function.

[0111] S105: According to the clothing coverage, adaptive feature purification is performed on the fusion feature to obtain pure features.

[0112] Specifically, a lightweight MLP model analyzes interference patterns and extracts interference features based on clothing coverage and fused features. Feature purification is primarily used to remove clothing interference from existing clothing features. An interference analysis network is set up, taking the fused features and clothing coverage as input. Using shape priors as a reference, it infers the likely body shape beneath the clothing. A lightweight MLP model analyzes interference patterns and extracts interference features, which are automatically scaled based on coverage.

[0113] At this point, interference analysis combines the fusion features (1024 dimensions) and clothing coverage (1 dimension), which can be shown as Formula 6:

[0114] F dist =Tanh(MLP([F fused ;Coverage])) Formula 6;

[0115] Among them, F dist is the interference feature, Tanh is the hyperbolic tangent function, F fused is the fusion feature, Coverage is the clothing coverage, and MLP is the lightweight MLP model. The interference feature represents the clothing interference information that needs to be removed from the original feature. It increases with the coverage and automatically approaches zero when the coverage is low.

[0116] When adjusting the clothing coverage, the interference characteristics can be dynamically adjusted using Formula 7:

[0117] F dist =F dist *Sigmoid(10Coverage-5) Formula 7.

[0118] Through the lightweight MLP model, the reset weight is generated according to the interference characteristics and clothing coverage. The reset weight is shown in Formula 8:

[0119] R = Sigmoid(MLP([F fused ; F dist ])) Formula 8;

[0120] Among them, R is the reset weight, F fused is the fusion feature, F dist Interference characteristics.

[0121] When adjusting the clothing coverage, the reset weight can be dynamically adjusted using Formula 9:

[0122] R = R*sigmoid(10Coverage-5) Formula 9;

[0123] Thus, when the clothing coverage is low (e.g., less than 0.3), sigmoid(10*0.3-5)≈0.007, almost no feature reset is performed, while when the clothing coverage is medium (e.g., equal to 0.5), sigmoid(10*0.5-5)≈0.5, partial feature reset is performed, and when the coverage is high (e.g., greater than 0.7), sigmoid(10*0.7-5)≈0.993, almost complete feature reset is performed. This achieves a smooth transition according to the clothing state and avoids the discontinuity problem caused by hard threshold classification. At this time, the reset weight is dynamically adjusted according to the coverage, and the image without clothing is hardly modified.

[0124] According to the reset weights, the fused features are fused with the baseline features corresponding to the body posture information obtained through prior knowledge to obtain pure features.

[0125] The process of obtaining pure features can be called feature purification. During the feature purification process, the feature reset weight is calculated based on interference analysis, and the feature purification formula can be shown as Formula 10:

[0126] F pure =F fused *(1-R)+F ref *R Formula 10;

[0127] Among them, F pure is a pure feature, F fused is the fusion feature, R is the reset weight, F ref The baseline features are basic features of the human body learned by combining prior knowledge of body posture information, and are used to describe the basic structure of the human body.

[0128] Among them, the reference feature F ref As a feature benchmark for network parameter learning, its learning mechanism can be set as follows: initialized to random values, for example, with a normal distribution of (N(0, 0.01)), and learned through end-to-end training to an abstract representation of "ideal clothing-free features" without the need for explicit supervision. Unlike directly using features of specific clothing-free samples, the benchmark features represent a broader prior of human morphology. Serving as a shared reference point for all inputs, the benchmark features provide an implicit representation of the "pure body" and ensure consistency in feature resetting. Furthermore, in its dynamic function, the benchmark features, along with the reset weights and clothing coverage, jointly determine the strength and method of feature purification. In the absence of clothing or low coverage, the original features are primarily retained, while in the case of high coverage, more intensive feature purification is performed to remove more clothing interference information.

[0129] S106: Predicting human body parameters based on the pure features, and performing 3D human body modeling based on the prediction results.

[0130] Specifically, a shared predictor is used to output pose prediction parameters through a first multi-layer MLP model, body shape prediction parameters through a second multi-layer MLP model, and perspective camera parameters through a third multi-layer MLP model. This shared predictor enables both clothing-free and clothing-only models to share the same parameter prediction network, ensuring that different feature sources produce consistent body parameter outputs.

[0131] The parameter predictor can also be called a differentiable parameterized human body model, which mainly refers to the shape vector and posture vector The statistical model of the 3D human body mesh V is generated by the linear mixed skinning function Φ(β,θ). Among them, β is the shape vector used to represent the body shape characteristics, K is the dimension of the shape parameters, which can be determined by principal component analysis, θ is the pose vector used to represent the pose of the human body, and L depends on the number of joints and the parameter representation of each joint.

[0132] Among them, the parametric human body model can adopt models such as SMPL, SMPL-X, STAR, GHUM, etc. The parameter prediction architecture includes:

[0133] Pose prediction network: corresponds to the first multi-layer MLP model, outputs 72-dimensional pose parameters; shape prediction network: corresponds to the second multi-layer MLP model, outputs 10-dimensional shape parameters; camera parameter prediction network: corresponds to the third multi-layer MLP, outputs 3-dimensional weak perspective camera parameters.

[0134] The camera parameters are constrained by scaling factor and translation parameter constraints. The scaling factor is mapped to the range of 0 to 2 using a sigmoid function, and the translation parameter is constrained to the range of [-1, 1] using a tanh function. This ensures that the predicted parameters are physically meaningful and facilitates subsequent rendering and evaluation.

[0135] According to the posture prediction parameters, body shape prediction parameters, and perspective camera parameters, the prediction results corresponding to the human body parameter prediction are output.

[0136] 1. By extracting deep features from 2D images through a shared encoder and combining it with mask prediction and coverage analysis from the clothing understanding module, it effectively distinguishes the true human body contour from areas obscured by clothing, significantly reducing contour misjudgments caused by loose clothing and complex postures, improving the measurement accuracy of key dimensions (e.g., shoulder width and waist circumference), and achieving high-precision body parameter prediction.

[0137] 2. Introducing shape prior features and an adaptive feature purification mechanism, by integrating user body information (e.g., height and weight) with dynamic weight distribution of clothing coverage, we can reduce the impact of lighting changes and shooting angle deviations on the model, enhance the algorithm's generalization ability under non-ideal shooting conditions, and improve its robustness in complex scenes.

[0138] 3. A feature fusion strategy driven by clothing coverage, combined with a contrastive learning framework, can mine the potential correlation between human body shape and clothing from limited data, accurately model personalized body features (e.g., chest-to-waist ratio, shoulder slope), break through the adaptation bottleneck of traditional standard sizes, improve the fit of clothing recommendations, and optimize personalized adaptation capabilities.

[0139] 4. Through the end-to-end neural network model, 3D modeling can be completed with only one or multiple ordinary 2D images and basic body posture information, without relying on high-cost depth sensors or multi-view acquisition equipment, lowering the hardware threshold and being suitable for rapid deployment in large-scale scenarios such as e-commerce and virtual fitting, ensuring low cost and easy implementation.

[0140] 5. The adaptive feature purification module can dynamically filter out redundant features introduced by noise such as clothing texture and wrinkles to ensure the stability of human body parameter prediction. In particular, it has stronger parsing capabilities for complex clothing scenes (for example, multi-layered clothing and irregular cuts), thereby improving anti-interference capabilities.

[0141] In one embodiment, a unified forward propagation design is implemented in the neural network model. The network uses a unified feature processing pipeline, and all inputs undergo the same processing. An adaptive feature processing mechanism and multi-view feature alignment are implemented, supporting diverse clothing states.

[0142] For the adaptive feature processing mechanism, a continuous value is used to represent clothing coverage, and the processing intensity is dynamically adjusted through an adaptive mechanism; clothing coverage is directly calculated from the mask without the need for complex estimation; the processing flow is completely consistent, and only the dynamic changes in processing intensity are processed to ensure the simplicity and stability of the inference process.

[0143] For multi-view feature alignment, it supports comparative learning of images of the same person from different viewpoints.

[0144] For the diversity of clothing states, including different perspectives without clothing, different perspectives with clothing, and between clothing and no clothing; all samples go through the same feature extraction and processing process to ensure the consistency of the feature space.

[0145] In one embodiment, during the training of the neural network model, human images are acquired and sample augmentation is performed on the human images. For example, data augmentation operations such as scaling, random cropping, horizontal flipping, and color dithering are performed on the images to ensure that the model is robust to changes in perspective, lighting, and occlusion.

[0146] For human body images, determine the corresponding data source.

[0147] If the data source is synthetic data, the corresponding clothing mask and human body mask are generated according to the human body image during the rendering stage; if the data source is real data, the corresponding clothing mask and human body mask are generated through a pre-trained human body segmentation model or through manual annotation.

[0148] Moreover, the mask quality needs to be ensured, with special attention paid to the clothing edge area to accurately calculate the clothing coverage.

[0149] Construction of training samples based on human body images.

[0150] For the same user, a positive sample pair is obtained by combining human body images of different dimensions, including whether clothing is present, clothing status, and viewing angle. This enhances the model's ability to capture the inherent characteristics of the human body.

[0151] For different users, if the body shape similarity between them is higher than the preset similarity, negative sample pairs are generated and downgraded or excluded to reduce confusion caused by body shape similarity.

[0152] During the training process, positive and negative sample pairs are sampled in batches, and the feature purification module and parameter prediction module are jointly optimized through comparative learning and multiple supervision constraints.

[0153] Multiple loss functions can be set during the training phase, such as: parameter prediction loss: using MSE to measure the deviation between the predicted parameters and the true parameters; mask prediction loss: supervising the quality of clothing and human mask prediction, using binary cross entropy loss; contrast loss (InfoNCE): improving the discriminability of human features through positive and negative sample pairs; feature alignment loss: when using no-clothing reference features, aligning purified features with no-clothing features; camera parameter loss: limiting weak perspective camera parameters within a reasonable range to enhance projection accuracy; vertex loss: L vertex =MSE(V pred ,V gt ), aligning the predicted mesh with the real mesh vertices; Joint 2D loss: L joint2d =MSE(J2D pred ,J2D gt ), ensuring the accuracy of the joint position projected onto the image plane; Joint 3D loss: L joint3d =MSE(J3Dpred ,J3D gt ), align key joints in three-dimensional space. vertex is the fixed-point loss, V pred is the predicted mesh vertex coordinate, V gt is the vertex coordinate of the real mesh, L joint2d Joint 2D loss, J2D pred J2D is the coordinate of the predicted joint point on the two-dimensional image plane. gt is the coordinate of the real joint point on the two-dimensional image plane, L joint3d Joint 3D loss, J3D pred J3D is the coordinate of the predicted joint point on the 3D image plane. gt is the coordinate of the real joint point on the 3D image plane. In the end, only the core losses are retained: contrast loss, direct feature alignment loss, parameter prediction loss, mask prediction loss, etc.

[0154] In one embodiment, during the training of the neural network model, the training samples with clothing and the training samples without clothing are mapped to a common space through a shared projection head. The training samples without clothing are only used during training.

[0155] In the common space, pure features are subjected to dimensionality reduction and L2 normalization. The shared projection head uses the same projection network for both clothing-free and pure clothing features. It is set up as a two-layer MLP structure, which includes a linear layer, a batch normalization layer, a ReLU layer, a linear layer, and an L2 normalization layer in order. This ensures that features from different sources are mapped to the same feature space, which facilitates feature alignment.

[0156] The features without clothing and with clothing are mapped to a common space and the contrast loss is calculated. Based on the InfoNCE loss, the similarity between positive and negative sample pairs is calculated, and the sharpness of the feature distribution is adjusted by the temperature parameter τ.

[0157] For each batch of training samples, the neural network model is trained by calculating the cross entropy loss.

[0158] Among them, the InfoNCE loss is shown in Formula 11:

[0159]

[0160] Among them, f q To query sample features, is the positive sample feature, is the negative sample feature, τ is the temperature parameter, and the default value is 0.07.

[0161] In one embodiment, during the training process of the feature purification module, clothing-free reference features can be introduced as additional guidance to improve the purification quality through direct feature alignment loss.

[0162] During the training process of the neural network model, the fusion features are adaptively purified according to the clothing coverage to obtain pure features.

[0163] During the training phase, when paired unclothed images are available, their features are used as direct references for the clean features. The corresponding unclothed reference features are obtained. Reference features can be obtained from unclothed images of the same person processed by the same encoder, which are only available for a subset of samples during the training phase.

[0164] The neural network model is trained by weighted summing the pure features and the reference features through direct feature alignment loss, semantic similarity loss, and parameter consistency loss.

[0165] Among them, the direct feature alignment loss can be L2 distance loss, the semantic similarity loss can be cosine similarity loss, and the parameter consistency loss can include a structural similarity indicator between feature maps at the structural feature level.

[0166] At the original feature level, calculate the L2 distance loss: L direct =||F purified –F naked || 2 , at the semantic feature level, calculate the cosine similarity loss: L direct-cos =1–cos(F purified ,F naked ), at the structural feature level, the structural similarity index between feature maps (for example, SSIM is applied to features) is used to perform multi-level feature alignment.

[0167] Among them, L direct is the L2 distance loss, L direct-cos is the cosine similarity loss, F purified To calculate the pure characteristics, F aaked Reference features.

[0168] Based on this, set the feature alignment comprehensive loss L align =α*L direct +β*L cos+γ *L consist , where the default weights are: α = 1.0, β = 0.5, γ = 0.2.

[0169] In the early stages of training, mandatory feature alignment and a high-weight L2 distance loss are used. In the mid-stage, a balance is established between feature alignment and other objectives (e.g., parameter prediction accuracy). In the late stages, slight feature guidance is used, relying primarily on the model's learned purification capabilities. The inference stage does not require clothing-free images and relies entirely on the trained model for feature purification.

[0170] In addition, you can also set body part attention, jointly predict the attention weights of each body part based on pure features and height and weight priors, and provide anatomical guidance to assist subsequent parameter prediction.

[0171] like Figure 3 As shown, the embodiment of the present application further provides a 3D human body modeling device based on contrastive learning, comprising:

[0172] at least one processor; and,

[0173] a memory communicatively connected to the at least one processor; wherein,

[0174] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: the 3D human body modeling method based on contrast learning as described in any of the above embodiments.

[0175] An embodiment of the present application further provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to be: the 3D human body modeling method based on contrastive learning described in any of the above embodiments.

[0176] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device and medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0177] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0178] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0179] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0180] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0181] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0182] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0183] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0184] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0185] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0186] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A 3D human body modeling method based on contrastive learning, characterized in that: include: Obtaining 2D image information corresponding to the user and body posture information corresponding to the user, and inputting the 2D image information and the body posture information into a pre-trained neural network model; Extracting original features corresponding to the 2D image information through a shared encoder based on the neural network model; Through the clothing understanding module, mask prediction and clothing coverage are performed based on the original features; Perform feature fusion based on the original features, the shape prior features corresponding to the body information, and the clothing coverage to obtain a fused feature; According to the clothing coverage, the fusion features are adaptively purified to obtain pure features; Based on the pure features, human body parameters are predicted, and 3D human body modeling is performed based on the prediction results.

2. The method according to claim 1, characterized in that Extracting original features corresponding to the 2D image information through a shared encoder, specifically including: Using the pre-trained ViT model as the basic feature extractor, the 2D image information is divided into multiple blocks, and the global dependency between the blocks is determined through self-attention to obtain ViT features; Extracting convolution features corresponding to the 2D image information through a convolutional layer series structure; The ViT feature and the convolution feature are weightedly fused to obtain the original feature corresponding to the 2D image information.

3. The method according to claim 1, characterized in that Through the clothing understanding module, mask prediction and clothing coverage are performed based on the original features, specifically including: Constructing feature maps of multiple scales based on the original features through a feature pyramid; Extracting semantic information based on the feature map through a U-Net-like codec structure; Based on the extracted semantic information, the clothing mask branch and the human mask branch in the dual-branch mask prediction are used to perform mask prediction through convolution respectively; For the predicted mask results, edge features are extracted through edge detection operators and fed back to the shallow layer of the network; Output the mask results corresponding to multiple scales, and select the mask result corresponding to the highest resolution among the multiple scales. Calculate the percentage of pixels in the clothing area and normalize it to obtain the clothing coverage.

4. The method according to claim 1, wherein According to the clothing coverage, the fusion features are adaptively purified to obtain pure features, specifically including: Analyze interference patterns and extract interference features using a lightweight MLP model based on the clothing coverage and the fusion features; Generating a reset weight according to the interference feature and the clothing coverage by the lightweight MLP model; According to the reset weight, the fused feature is fused with the reference feature corresponding to the posture information obtained through prior knowledge to obtain a pure feature.

5. The method according to claim 1, characterized in that Based on the pure features, human body parameter prediction is performed, specifically including: Through the shared predictor, the first multi-layer MLP model outputs the posture prediction parameters; through the second multi-layer MLP model, the body shape prediction parameters are output; and through the third multi-layer MLP model, the perspective camera parameters are output; Constraining the camera parameters by means of zoom factor constraints and translation parameter constraints; According to the posture prediction parameters, the body shape prediction parameters, and the perspective camera parameters, a prediction result corresponding to the obtained human body parameter prediction is output.

6. The method according to claim 1, characterized in that The training process of the neural network model includes: Acquiring a human body image, and performing sample enhancement processing on the human body image; For the human body image, determining a corresponding data source; If the data source is synthetic data, then in the rendering stage, corresponding clothing masks and human body masks are generated according to the human body image; If the data source is real data, the corresponding clothing mask and body mask are generated through the pre-trained human segmentation model; Constructing training samples based on human body images; Among them, for the same user, the corresponding human body images of different dimensions are combined to obtain positive sample pairs; the different dimensions include: whether there is clothing, clothing status, and viewing angle; For different users, if the similarity between their body shapes is higher than a preset similarity, negative sample pairs are obtained by combination, and the negative sample pairs are downgraded or excluded.

7. The method according to claim 1, characterized in that The training process of the neural network model includes: By sharing the projection head, the training samples with clothing and without clothing are mapped to a common space; In the common space, the pure features are subjected to dimensionality reduction processing and L2 normalization; Based on InfoNCE loss, the similarity between positive sample pairs and negative sample pairs is calculated, and the sharpness of the feature distribution is adjusted by the temperature parameter τ; For each batch of training samples, the neural network model is trained by calculating the cross entropy loss.

8. The method according to claim 1, characterized in that The training process of the neural network model includes: According to the clothing coverage, the fusion features are adaptively purified to obtain pure features; Obtain the corresponding reference features without clothing; The neural network model is trained by performing weighted summation of the pure features and the reference features through direct feature alignment loss, semantic similarity loss, and parameter consistency loss.

9. A 3D human body modeling device based on contrastive learning, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: the 3D human body modeling method based on contrast learning as described in any one of claims 1 to 8.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured as: the 3D human body modeling method based on contrastive learning as described in any one of claims 1 to 8.

Images