Method and device for generating digital human based on commodity upper body and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- DONSON TIMES INFORMATION TECH CO LTD
- Filing Date
- 2026-02-11
- Publication Date
- 2026-05-29
Smart Images
Figure CN121707684B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of virtual try-on technology, and in particular to a method, device and storage medium for generating digital humans based on wearing products. Background Technology
[0002] With the rapid popularization of digital human technology in the marketing field, its application scope has gradually expanded from basic image display to product promotion and interactive experiences. Digital humans can highly simulate human appearance, voice, and behavior, providing 24 / 7 service for industries such as e-commerce and advertising, significantly reducing labor and operational costs. However, current technology has not yet overcome a key bottleneck: how to efficiently and realistically achieve natural integration and personalized display between massive amounts of products and digital humans. Especially in a marketing environment where products are frequently updated, traditional methods struggle to achieve seamless adaptation of various products to digital human images without relying on manual re-creation, limiting further improvements in marketing efficiency and user experience.
[0003] Current mainstream digital human marketing solutions suffer from several technical limitations. Static generation solutions rely on manual modeling and rendering, which can produce high-quality static images, but each product requires individual modeling and binding, making the process lengthy and difficult to scale. Motion capture solutions, while capable of presenting relatively natural movements, are highly dependent on equipment and environment, have low flexibility in product changes, and cannot meet the marketing needs of rapid deployment across multiple product categories. While AIGC solutions based on generative adversarial networks or diffusion models have improved content generation efficiency, they still face problems such as unrealistic body effects and insufficient reproduction of material and lighting details when handling interactions between products and digital humans. Furthermore, existing methods generally lack a unified understanding of product types, interaction methods, and background coordination, resulting in deficiencies in semantic consistency and visual plausibility in the generated results.
[0004] In conclusion, in marketing scenarios where products frequently change, there is a lack of a systematic method that can automatically understand product attributes, adapt appropriate interaction methods, and generate product images that harmonize with the background. This systemic deficiency severely restricts the large-scale promotion and actual business effectiveness of digital human marketing.
[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention
[0006] The main purpose of this application is to provide a method, device and storage medium for generating digital humans based on product images, which aims to solve the technical problem of poor image fusion effect of product images in existing digital human marketing solutions.
[0007] To achieve the above objectives, this application proposes a method for generating digital humans based on product-related body parts, the method comprising:
[0008] Obtain text instructions and product images, input the text instructions and product images into a multimodal model, and obtain the semantic association features and visual detail features output by the multimodal model;
[0009] The semantic association features and the visual detail features are fused using a cross-attention mechanism, and the product wearing instructions are obtained based on the fusion result.
[0010] In response to the product wearing instruction, the image ratio and spatial perspective of the product image and the preset model image are adjusted, and the adjustment results are used to generate an initial product wearing image using a diffusion model;
[0011] Locate the modification area of the initial product image and verify the consistency between the image information in the modification area and the corresponding area in the product image and the preset model image. If the verification result is consistent, call the text rendering strategy to render the initial product image to obtain the product image. The text rendering strategy is a text editing operation based on the initial product image.
[0012] In one embodiment, the step of inputting the text instruction and the product image into a multimodal model and outputting semantic association features and visual detail features includes:
[0013] The text instructions are parsed using the semantic encoding path of the multimodal model, and semantic association features are generated based on the parsing results;
[0014] Using the appearance encoding pathway of the multimodal model, visual detail features corresponding to the semantic association features are extracted from the product image.
[0015] In one embodiment, the step of fusing the semantic association features and the visual detail features through a cross-attention mechanism, and obtaining the product wearing instruction based on the fusion result, includes:
[0016] The semantic association features are used as query sequences, and the visual detail features are used as key and value sequences, which are then input into the cross-attention layer.
[0017] Based on the cross-attention layer, the relevance weights between the query sequence and the key sequence are calculated, and the value sequence is weighted and summed according to the relevance weights to generate a fusion feature representation;
[0018] After performing linear transformation and normalization on the fused feature representation, the product wearing instruction is generated.
[0019] In one embodiment, the step of generating the product wearing instruction after performing linear transformation and normalization on the fused feature representation includes:
[0020] The fused feature representation is input into an adaptive normalization layer, where a dynamic scaling factor and a bias factor are generated based on the semantic association features.
[0021] The fused feature representation is subjected to channel affine transformation using the scaling factor and the bias factor, and the fused feature representation after channel affine transformation is normalized.
[0022] The normalized fusion feature representation is then subjected to gated linear filtering, and the product wearing instructions are obtained based on the filtering results.
[0023] In one embodiment, the step of adjusting the image ratio and spatial perspective of the product image and the preset model image includes:
[0024] Extract the coordinates of key body points related to product interaction from the preset model image, and the coordinates of key contact areas from the preset product image;
[0025] Based on the body key point coordinates and the contact area key point coordinates, a spatial transformation matrix is calculated to adapt the product image to the preset model image. The transformation matrix includes at least a scaling factor, a rotation angle, and a perspective transformation parameter.
[0026] Based on the spatial transformation matrix, bilinear interpolation and pixel-level resampling are performed on the product image, and the product image and the preset model image are aligned using the operation results and sampling results.
[0027] Based on the alignment results, a dense correspondence field is created between the pixel coordinates of the resampled product image and the pixel coordinates of the preset model image.
[0028] In one embodiment, the step of generating an initial product image based on the product wearing instructions using a diffusion model after adjustments includes:
[0029] The latent space vector of the product image and the product wearing instructions are input into the conditional denoising network of the diffusion model for denoising iterative operation, and a clean latent representation is obtained through the iterative operation results;
[0030] The VAE decoder based on the diffusion model reconstructs the clean latent representation, and the initial product image is obtained based on the reconstruction result.
[0031] In one embodiment, the step of inputting the latent space vector of the product image and the product wearing instructions into the conditional denoising network of the diffusion model for denoising iterative computation, and obtaining a clean latent representation through the iterative computation results, includes:
[0032] The conditional gradient of the product wearing instruction is calculated using a cross-attention mechanism;
[0033] Based on the geometric correspondence between the product image and the preset model image, a spatial constraint mask is generated, and the spatial constraint mask is applied to the latent space vector of the product image to obtain the noise latent representation of the current time step;
[0034] The conditional gradient calculation results and the noise latent representation at the current time step are input into the U-Net noise predictor of the diffusion model to predict the noise residual.
[0035] Based on the noise residual, a stochastic differential equation solver is used to calculate and update the noise latent representation for the next time step, and the clean latent representation is obtained based on the update result.
[0036] In one embodiment, the VAE decoder based on the diffusion model reconstructs the clean latent representation, and the initial product image is obtained based on the reconstruction result, including:
[0037] The clean latent representation is upsampled hierarchically through a series of transposed convolutional networks in the VAE decoder, and a spatial feature map is output based on the sampling results.
[0038] The spatial feature map is input into an activation function for nonlinear transformation, and then the transformed spatial feature map is fused with the multi-scale features of the model image in the feature fusion layer of the VAE decoder.
[0039] The fusion result is subjected to a convolution operation, and the number of channels of the convolution operation result is mapped to the RGB color space to generate the initial product image.
[0040] Furthermore, to achieve the above objectives, this application also proposes a digital human generation device based on a product, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the digital human generation method based on a product as described above.
[0041] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the digital human generation method based on product body as described above.
[0042] One or more technical solutions proposed in this application have at least the following technical effects:
[0043] The technical solution of this application involves acquiring text instructions and product images, inputting the text instructions and product images into a multimodal model, acquiring semantic association features and visual detail features output by the multimodal model, fusing the semantic association features and visual detail features through a cross-attention mechanism, and obtaining product wearing instructions based on the fusion result, adjusting the image ratio and spatial perspective of the product image and a preset model image in response to the product wearing instructions, and generating an initial product wearing image using a diffusion model, locating the modification area of the initial product wearing image, and verifying the consistency between the image information in the modification area and the corresponding area in the product image and the preset model image, and when the verification result is consistent, calling a text rendering strategy to render the initial product wearing image to obtain a product wearing image, wherein the text rendering strategy is a text editing operation based on the initial product wearing image.
[0044] This application extracts semantic and visual features of products through a multimodal model, integrates them to generate body commands, and uses a diffusion model to adjust image proportions and spatial perspective. After consistency verification and text rendering, it achieves a natural and efficient integration of products and digital humans, improving marketing efficiency and user experience. Attached Figure Description
[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0047] Figure 1 This is a flowchart illustrating the first embodiment of the digital human generation method based on product body in this application;
[0048] Figure 2 This is a detailed step diagram based on step S10 in the first embodiment;
[0049] Figure 3 This is a detailed step diagram based on step S20 in the first embodiment;
[0050] Figure 4 This is a detailed step diagram based on step S30 in the first embodiment;
[0051] Figure 5 This is a schematic diagram of another detailed step based on step S30 in the first embodiment;
[0052] Figure 6 This is an illustration based on product images and preset model images;
[0053] Figure 7 This is a schematic diagram illustrating the synthesis of a product image based on a product image and a preset model image;
[0054] Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the digital human generation method based on product body in the embodiments of this application.
[0055] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0056] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.
[0057] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.
[0058] The main solution of this application embodiment is as follows: Obtain text instructions and product images; input the text instructions and product images into a multimodal model to obtain semantic association features and visual detail features output by the multimodal model; fuse the semantic association features and visual detail features through a cross-attention mechanism, and obtain a product wearing instruction based on the fusion result; respond to the product wearing instruction, adjust the image ratio and spatial perspective of the product image and a preset model image, and use a diffusion model to generate an initial product wearing image from the adjustment result; locate the modification area of the initial product wearing image, and verify the consistency between the image information in the modification area and the corresponding area in the product image and the preset model image; when the verification result is consistent, call a text rendering strategy to render the initial product wearing image to obtain a product wearing image, wherein the text rendering strategy is a text editing operation based on the initial product wearing image.
[0059] In existing technologies, the reliance on manual modeling leads to low efficiency in integrating digital humans with products. Furthermore, the lack of a unified understanding of product attributes, interaction methods, and background results in a stiff integration of digital humans with products and unrealistic visual effects, making it difficult to adapt to the marketing needs of rapidly changing products.
[0060] This application provides a solution that extracts semantic and visual features of products through a multimodal model, integrates them to generate body commands, and uses a diffusion model to adjust image proportions and spatial perspective. After consistency verification and text rendering, it achieves a natural and efficient integration of products and digital humans, thereby improving marketing efficiency and user experience.
[0061] Based on this, the embodiments of this application provide a method for generating digital humans based on product images, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the digital human generation method based on product body image according to this application. In this embodiment, the digital human generation method based on product body image image includes steps S10 to S40:
[0062] Step S10: Obtain text instructions and product images, input the text instructions and product images into a multimodal model, and obtain the semantic association features and visual detail features output by the multimodal model;
[0063] In this embodiment, a product image is acquired through an image acquisition interface, and text instructions are received through a text input interface. The product image is a digital product image to be marketed on a mannequin, and its format includes RGB or grayscale pixel data. The text instructions are natural language descriptions provided by the user, used to define the semantic requirements for displaying the product, such as specific display instructions like "place the product in the mannequin's hand" or "adjust the product size to match the mannequin's proportions." The product image and text instructions serve as input to a multimodal model and are transmitted to a multimodal processing model based on the Qwen2.5-VL-7B architecture for data processing.
[0064] Specifically, the multimodal model consists of a visual encoder and a language model decoder. The visual encoder employs a pre-trained Vision Transformer (ViT) structure to segment the input product image, converting pixel data into a visual token sequence. Specifically, the visual encoder extracts spatial features through a multi-layer self-attention mechanism, outputting a high-dimensional visual feature vector, where each vector corresponds to a local region of the image. Simultaneously, the text instructions are converted into a text token sequence using a Byte Pair Encoding (BPE) tokenizer. The visual token sequence and the text token sequence are concatenated into a multimodal input sequence, which is then fed into the language model decoder.
[0065] The language model decoder is based on the Qwen2.5 LLM architecture and processes multimodal input sequences in an autoregressive manner. The decoder fuses visual and textual information through a cross-attention layer, where visual tokens serve as key-value pairs and text tokens as query sequences. This query process generates two types of output features: semantic association features and visual detail features. The semantic association features capture high-level semantic information, including product category, functional attributes, and interaction relationships with the model; for example, parsing the "holding" action from text instructions and associating it with the hand region in the product image. The visual detail features retain low-level visual attributes, such as product material, color gradient, and texture pattern, and are extracted through the convolutional layers of the visual encoder.
[0066] In its implementation, the multimodal model preprocesses the input data. The product images are normalized and adjusted to a fixed resolution (e.g., 224×224 pixels) to eliminate scale differences. The text instructions are converted into vector representations through an embedding layer. Multimodal training uses a large-scale image-text pair dataset and optimizes the cross-entropy loss function to improve the accuracy of feature extraction. The semantic association features and visual detail features are output as floating-point tensors and stored in a feature buffer for subsequent steps.
[0067] Step S20: The semantic association features and the visual detail features are fused through a cross-attention mechanism, and the product wearing instructions are obtained based on the fusion result.
[0068] In this embodiment, the output semantic association features and visual detail features are used as input and fused through a cross-attention mechanism to generate a product wearing instruction. The semantic association features are high-order semantic tensors that encode the logical relationship between the product and the model; the visual detail features are low-order visual tensors that contain the product's appearance attributes.
[0069] Furthermore, the described cross-attention mechanism is implemented based on a transformer architecture, where semantic association features serve as the query sequence and visual detail features serve as the key and value sequences. Specifically, the query, key, and value vectors are mapped to the same dimensional space through a linear projection layer. The attention weights are calculated using a scaled dot product attention formula: first, the dot product of the query and the key is calculated, then a softmax function is applied for normalization to obtain the attention distribution; finally, the attention distribution and the value vector are weighted and summed to output a fused feature tensor. This process uses multi-head attention expansion to capture feature interactions between different subspaces.
[0070] The fusion feature tensor combines semantic constraints with visual references, for example, fusing the semantics of "holding a cup" with the cylindrical shape and reflective material of the cup. The product wearing instructions are generated from the fusion feature tensor through decoding, taking the form of a structured vector or token sequence, defining the modification area, fusion parameters, and style constraints. The decoding process uses fully connected layers or lightweight transformers to map the fusion features to the instruction space. The specific content of the product wearing instructions covers spatial relationships (such as product placement coordinates), appearance attributes (such as color adjustment), and interaction methods (such as grip angle).
[0071] In practice, the cross-attention layer is configured with 8 heads and a hidden dimension of 512. An adversarial training strategy is employed during training, using a discriminator network to evaluate the authenticity of the fused features and optimize the generator loss function. After the product wearing instruction is output, it is serialized and stored in an instruction cache, serving as the conditional input to the diffusion model. This step balances the contributions of semantic and visual information through dynamic weight allocation, ensuring the accuracy and executability of the instruction.
[0072] Step S30: In response to the product wearing instruction, adjust the image ratio and spatial perspective of the product image and the preset model image, and use the diffusion model to generate an initial product wearing image from the adjustment results;
[0073] In this embodiment, the product image and the preset model image are used as input. First, the image ratio and spatial perspective are adjusted. Then, a diffusion model is used to generate an initial product image based on the product wearing instruction.
[0074] Specifically, image proportion adjustments are achieved through affine transformations, including scaling, cropping, and translation. Specifically, the relative size ratio between the product image and the preset model image is calculated. For example, bounding box detection determines the target size of the product on the model's body, and the scaling factor is dynamically calculated based on the model's anatomical structure (such as hand or torso proportions) to ensure the product size conforms to ergonomics. Spatial perspective adjustment uses a perspective transformation model to project the product image onto the model's reference coordinate system. Projection parameters are estimated through feature point matching, for example, using SIFT (Scale-Invariant Feature Transform) or ORB (Oriented Fast and Rotated BRIEF) algorithms to detect key points on the product and model, and solving for the homography matrix.
[0075] The adjusted product image and the preset model image are stitched together as a multi-image input and fed into the diffusion model. The diffusion model is based on the Qwen-Image-Edit-2509 architecture and includes a U-Net denoising network and a conditional coding module. The diffusion process adopts the DDPM (Denoising Diffusion Probabilistic Model) framework. In the reverse denoising stage, U-Net receives the noisy image and time step, and predicts the noise residual based on the product wearing instructions. The product wearing instructions are injected into U-Net through a cross-attention layer, where the instruction token serves as a conditional input, interacting with image features. The denoising process is executed iteratively, gradually generating a high-quality image from random noise.
[0076] The initial product image generation is based on pixel-level prediction, with the diffusion model outputting RGB (Red, Green, Blue) image data. During training, the diffusion model uses a mean squared error loss function to optimize noise prediction accuracy; simultaneously, a CLIP (Contrastive Language-Image Pre-training) guided loss is introduced to enhance semantic consistency. The diffusion model integrates a VAE (Variational Autoencoder) encoder to compress and reconstruct the input image, improving generation efficiency. Finally, the initial product image output is in a standard image format and may contain local unnatural elements or boundary artifacts (human intervention), requiring further optimization in subsequent steps.
[0077] Step S40: Locate the modification area of the initial product image and verify the consistency between the image information in the modification area and the corresponding area in the product image and the preset model image. If the verification result is consistent, call the text rendering strategy to render the initial product image to obtain the product image. The text rendering strategy is a text editing operation based on the initial product image.
[0078] In this embodiment, an initial product image is used as input to locate the modified areas. The consistency of the image information is then verified, and a text rendering strategy is invoked to generate the final product image. The location of the modified areas employs a semantic segmentation network, such as a model based on the DeepLabV3+ architecture, to perform pixel-level classification of the initial product image. The segmentation network outputs a binary mask to identify the areas modified by the diffusion model, such as added or replaced parts of the product. The localization process incorporates attention map analysis, using Grad-weighted Class Activation Mapping (Grad-CAM) to visualize high-contribution areas, ensuring localization accuracy.
[0079] Furthermore, the consistency verification includes visual consistency and semantic consistency evaluation. Visual consistency is quantified using the Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR), comparing the matching degree of the modified region with the product image and the preset model image in terms of color, texture, and lighting. Semantic consistency is calculated using the CLIP (Contrastive Language-Image Pre-training) model, aligning the modified region image with the text instruction embedding vector and evaluating the semantic matching score. If the verification score is below a threshold, the system activates the post-processing module, applying image inpainting algorithms (such as partial convolution-based inpainting) or color correction techniques to optimize the inconsistent regions.
[0080] The text rendering strategy addresses text-related editing needs, such as adding brand logos or marketing slogans. The rendering process is based on a high-precision text generator. First, a text detection module determines the insertion location, then a stylized font rendering engine generates the text image. Text perspective adjustment is performed according to scene geometry; for example, homography transformation is used to make the text conform to background perspective rules. Lighting blending employs a Gaussian Mixture Model to match the lighting conditions of the text with the surrounding environment.
[0081] The final product image output is a high-resolution image in PNG (Portable Network Graphics) or JPEG (Joint Photographic Experts Group) format, stored in the results database. This step uses automated verification and rendering to ensure that the generated image is visually and semantically consistent with the input data, meeting the requirements of marketing applications. The above implementation process can be viewed... Figure 6 ,in, Figure 6 This is an illustration based on product images and preset model images. Figure 7 This is a schematic diagram illustrating the synthesis of a product image based on a product image and a preset model image.
[0082] In summary, by extracting semantic and visual features of products through a multimodal model, adjusting image proportions and spatial perspective using a diffusion model, and performing consistency checks and text rendering, a natural and efficient integration of products and digital humans was achieved.
[0083] Furthermore, you can also view Figure 2 , Figure 2This is a detailed step diagram based on step S10 in the first embodiment. Figure 2 The step of inputting the text instruction and the product image into the multimodal model and outputting semantic association features and visual detail features includes steps S11-12:
[0084] Step S11: Parse the text instruction using the semantic encoding path of the multimodal model, and generate semantic association features based on the parsing results;
[0085] Step S12: Using the appearance encoding path of the multimodal model, extract visual detail features corresponding to the semantic association features from the product image.
[0086] In this embodiment, the semantic encoding path and appearance encoding path of the multimodal model are used to process text instructions and product images respectively, and output semantic association features and visual detail features to ensure the accuracy and correspondence of feature extraction.
[0087] The semantic encoding pathway is implemented based on the Qwen2.5-VL architecture and is specifically designed to parse the text instructions and generate semantic association features. The semantic encoding pathway receives the text instructions as input and first converts the natural language description into a sequence of text tokens using a byte-pair encoding (BPE) tokenizer. The token sequence is mapped to a high-dimensional vector representation through an embedding layer, forming the initial text features. Subsequently, the text features are fed into a transformer decoder layer, which consists of a multi-head self-attention mechanism and a feedforward neural network. The self-attention mechanism calculates the dependencies between tokens, capturing key semantic elements in the instructions, such as actions (e.g., "holding"), object relationships (e.g., "interaction between the product and the model"), and attribute constraints (e.g., "size adjustment"). The transformer output undergoes layer normalization and linear projection to generate semantic association features. These features are floating-point tensors that encode high-level semantic information, including the product's functional role, spatial constraints, and interaction logic. The dimensionality of the semantic association features is optimized through hidden layer configuration, for example, set to a 512-dimensional vector sequence to balance expressive power and computational efficiency. During training, the semantic encoding pathway is pre-trained on a large-scale text-image dataset and fine-tuned using the cross-entropy loss function to improve the accuracy of semantic parsing.
[0088] The appearance encoding path is implemented based on a variational autoencoder (VAE) architecture, used to extract visual detail features corresponding to the semantically related features from the product image. The appearance encoding path receives the product image as input. The image is first preprocessed, including normalization to a fixed resolution (e.g., 256×256 pixels) and color space conversion (RGB to Lab format) to enhance feature stability. The VAE encoder consists of a convolutional neural network (CNN), including multiple convolutional layers, batch normalization layers, and a ReLU (Rectified Linear Unit) activation function. The convolutional layers extract local visual patterns, such as texture gradients, color distribution, and shape contours, through a sliding window operation, outputting a latent space representation. To ensure that the visual detail features correspond to the semantically related features, the VAE encoder employs a conditional mechanism, where the semantically related features are used as conditional inputs, mapped to the latent space through a conditional projection layer, and concatenated with the image features along the channel dimension. The conditional mechanism is implemented through gated attention. The semantically associated features modulate the feature activation of the convolutional layer, focusing on semantically relevant regions. For example, when the semantically associated features indicate a "handheld" action, the encoder enhances the extraction of details in the hand region. The VAE encoder outputs visual detail features, which are multi-channel feature maps that preserve low-level visual attributes such as material reflectivity, edge sharpness, and illumination consistency. The dimensions of the visual detail features are aligned with the semantically associated features, for example, by adjusting the spatial size through adaptive pooling to ensure compatibility with subsequent fusion. During training, the appearance encoding pathway is jointly optimized using reconstruction loss and KL divergence (KL divergence) loss to maintain visual fidelity and conditional consistency.
[0089] As shown above, efficient extraction of multimodal features is achieved through parallel execution of semantic and appearance encoding. The semantic association features and visual detail features, after being output, are stored in a shared feature buffer, providing input for subsequent cross-attention fusion. This implementation improves the targeting and reliability of feature extraction through path specialization design, meeting the real-time requirements of marketing digital human generation.
[0090] Furthermore, you can also view Figure 3 , Figure 3 This is a detailed step diagram based on step S20 in the first embodiment. Figure 3 The step of fusing the semantic association features and the visual detail features through a cross-attention mechanism to obtain the product wearing instructions based on the fusion result includes steps S21-23:
[0091] Step S21: The semantic association features are used as a query sequence, and the visual detail features are used as a key sequence and a value sequence, and then input into the cross-attention layer.
[0092] Step S22: Based on the cross-attention layer, calculate the relevance weights between the query sequence and the key sequence, and perform a weighted summation on the value sequence according to the relevance weights to generate a fusion feature representation;
[0093] Step S23: After performing linear transformation and normalization on the fused feature representation, the product wearing instruction is generated.
[0094] In this embodiment, the semantic association features and the visual detail features are processed by a cross-attention layer to generate a fused feature representation, and the product wearing instruction is output after post-processing to ensure the accuracy of feature fusion and the executability of the instruction.
[0095] The semantic association features are used as the query sequence, and the visual detail features are used as the key and value sequences, respectively, and input into the cross-attention layer. The semantic association features are high-dimensional tensors output from a multimodal model, encoding the semantic relationship between the product and the model; the visual detail features are multi-channel feature maps extracted from the appearance encoding pathway, containing the product's appearance attributes. Before input, the query sequence, key sequence, and value sequence are mapped to the same dimensional space through a linear projection layer to eliminate feature dimensionality differences. The linear projection layer is implemented by a fully connected network, projecting each sequence to a fixed hidden dimension (e.g., 512 dimensions). The projection weights are optimized through training to enhance feature alignment. After sequence mapping, the query sequence, key sequence, and value sequence are reorganized into a multi-head format, for example, split into 8 attention heads, each with a dimension of 64, to support parallel computation. The cross-attention layer is constructed based on a transformer architecture, receiving the projected sequences as input to prepare for subsequent attention computation.
[0096] In the cross-attention layer, the relevance weights of the query sequence and the key sequence are calculated, and the value sequences are weighted and summed based on these relevance weights to generate a fused feature representation. The calculation of the relevance weights employs a scaled dot product attention mechanism: first, the dot product of the query sequence and the key sequence is calculated to obtain the original attention score; then, a scaling factor (i.e., the reciprocal of the square root of the key sequence dimension) is applied to the attention score to prevent gradient vanishing; finally, the scaled score is normalized using the softmax function to generate relevance weights in the form of a probability distribution. The weighted summation process multiplies the relevance weights by the value sequences and sums them to output a weighted vector. This calculation process is formally expressed as:
[0097] ;
[0098] Among them, the For the query sequence, the For key sequences, the For a value sequence, the for A square root operation is performed. The output of the multi-head attention is then combined through a concatenation operation to form a preliminary fused feature representation. This fused feature representation combines semantic constraints and visual references, such as fusing the semantics of "holding goods" with the shape and texture details of the goods, ensuring the diversity and robustness of the generated features.
[0099] The product application instructions are output after linear transformation and normalization of the fused feature representation. The linear transformation is implemented through a fully connected layer, mapping the fused feature representation to the instruction space. The output dimension of the fully connected layer is set according to the instruction complexity (e.g., 1024 dimensions). A non-linear activation function (such as GELU, Gaussian Error Linear Unit) is introduced during the transformation process to enhance expressive power. Layer normalization is used to standardize the mean and variance of the linearly transformed features, improving training stability and output consistency. The product application instructions are generated from the normalized features. The instructions are in the form of a structured token sequence or vector, defining modification parameters (such as spatial coordinates, color adjustment factors) and action constraints (such as interaction type). The decoding process uses a lightweight transformer or linear projection head to convert the features into an executable instruction format. After the instructions are output, they are serialized and encoded and stored in an instruction cache for use by the diffusion model. During training, this step optimizes end-to-end to minimize the mean squared error (MSE) between instructions and ground truth annotations, and incorporates adversarial training strategies to improve instruction quality. In summary, through refined attention design and post-processing, the semantic consistency and visual plausibility of product placement instructions are ensured.
[0100] Based on the above Figure 3 The content described in step S23 is further refined. Specifically, the step of outputting the product wearing instruction after performing linear transformation and normalization on the fused feature representation includes steps S23-1 to S23-3:
[0101] Step S23-1: Input the fused feature representation into the adaptive normalization layer, and generate a dynamic scaling factor and a bias factor based on the semantic association features in the adaptive normalization layer;
[0102] Step S23-2: Perform channel affine transformation on the fused feature representation using the scaling factor and the bias factor, and normalize the fused feature representation after the channel affine transformation.
[0103] Step S23-3: The normalized fusion feature representation is subjected to gated linear filtering, and the product wearing instruction is obtained based on the filtering result.
[0104] In this embodiment, the fused feature representation is refined through a value adaptive normalization layer, channel affine transformation, and gated linear filtering, and finally the product wearing instruction is output, ensuring the stability and semantic consistency of the instruction generation.
[0105] The fused feature representation is input to a value-adaptive normalization layer, which generates dynamic scaling and bias factors based on the semantic association features. The value-adaptive normalization layer employs a conditional normalization architecture, where the semantic association features serve as conditional inputs. The scaling and bias factors are calculated through two independent linear projection networks. Each linear projection network consists of fully connected layers, with its input dimension aligned to the semantic association features and its output dimension matching the number of channels in the fused feature representation. The generation of the scaling and bias factors utilizes non-linear activation functions (such as SiLU, Sigmoid Linear Unit, or Sigmoid Linear Unit) to enhance the flexibility of the representation. Specifically, the semantic association features are first mapped to an intermediate hidden space via a multilayer perceptron, and then the output layer generates scaling and bias vectors, each corresponding to a channel in the fused feature representation. The calculation of the dynamic factors introduces residual connections to mitigate the vanishing gradient problem and ensure training stability. The parameters of the value-adaptive normalization layer are optimized through end-to-end training to improve the accuracy of the response to semantic conditions.
[0106] The fused feature representation is subjected to a channel affine transformation using the scaling factor and bias factor, and then normalized. The channel affine transformation is applied along the channel dimension, where the feature value of each channel is multiplied by the corresponding scaling factor and then added to the bias factor, mathematically expressed as: Y_i = X_i γ_i + β_i, where X_i is the i-th channel of the fused feature representation, γ_i and β_i are the i-th components of the scaling factor and bias factor, respectively, and Y_i is the transformed feature channel. This process is implemented through channel-by-channel operations to ensure fine-grained alignment between visual details and semantic constraints. Normalization is performed after the channel affine transformation, using layer normalization technology to calculate the mean and variance of the channel dimension for each feature sample and then standardize it. The standardization formula is:
[0107] ;
[0108] in and For the mean and variance, To prevent numerical instability of small constants, for Perform a square root operation. The normalized features are scaled and shifted using learnable parameters to restore their expressive power, outputting a standardized fused feature representation.
[0109] The normalized fused feature representation is then subjected to gated linear filtering, and the product application instruction is obtained based on the filtering result. The gated linear filtering is implemented using a gated linear unit (GLU). First, the normalized fused feature representation is input to two parallel linear projection layers, generating a principal feature vector and a gate vector, respectively. The principal feature vector is mapped to the instruction space dimension through a fully connected layer, and the gate vector is used to generate gate weights between 0 and 1 using a sigmoid activation function. The filtering process calculates the element-wise product of the principal feature vector and the gate vector to achieve feature selection. Where Y_linear is the principal feature vector, Y_gate is the gate vector, and σ is the sigmoid function. As described in the element-wise multiplication method, a gating mechanism suppresses irrelevant features, strengthens semantically relevant components, and improves the accuracy of instructions. The filtering result is decoded into product application instructions through a final linear transformation layer. These product application instructions are in the form of a structured vector sequence, encoding space modification parameters, appearance adjustment factors, and interaction constraints. Furthermore, the output product application instructions are serialized and stored in an instruction cache for use by the diffusion model. During training, the mean squared error loss function is used to optimize the filtering parameters, and gradient pruning techniques are combined to prevent overfitting. In the above process, adaptive normalization and gating filtering ensure that product application instructions retain multimodal information while eliminating noise and redundancy, meeting generation efficiency requirements.
[0110] Furthermore, you can also view Figure 4 , Figure 4 This is a detailed step diagram based on step S30 in the first embodiment. Figure 4 Adjusting the image ratio and spatial perspective of the product image and the preset model image includes steps S31-34:
[0111] Step S31: Extract the coordinates of key body points related to product interaction from the preset model image, and the coordinates of key contact areas from the preset product image.
[0112] Step S32: Based on the coordinates of the body key points and the coordinates of the contact area key points, calculate the spatial transformation matrix to adapt the product image to the preset model image. The transformation matrix includes at least a scaling factor, a rotation angle, and a perspective transformation parameter.
[0113] Step S33: Perform bilinear interpolation and pixel-level resampling on the product image according to the spatial transformation matrix, and align the product image and the preset model image using the calculation results and sampling results;
[0114] Step S34: Based on the alignment results, create a dense correspondence field between the pixel coordinates of the resampled product image and the pixel coordinates of the preset model image.
[0115] In this embodiment, precise spatial alignment between the product image and the preset model image is achieved through key point detection, spatial transformation calculation, image resampling, and corresponding field establishment.
[0116] Specifically, a pre-trained keypoint detection model extracts the coordinates of body keypoints related to product interaction from the preset model image, as well as the coordinates of preset contact area keypoints from the product image. The keypoint detection model is implemented based on the HRNet (High-Resolution Network) architecture, employing multi-resolution parallel sub-networks to extract feature pyramids. For the preset model image, the keypoint detection model detects anatomical keypoints, including hand joints, torso contours, and facial feature points, focusing on locating areas interacting with the product; for example, when the text instruction includes "holding," it focuses on finger joints and the center point of the palm. For the product image, the keypoint detection model detects preset contact area keypoints, which are predefined according to the product type, such as the edge of a cup handle or the end of a headband on headphones. The keypoint coordinate output is a two-dimensional floating-point coordinate sequence with a confidence score, filtering low-confidence detection results to ensure reliability.
[0117] Based on the coordinates of the body keypoints and the coordinates of the contact area keypoints, a spatial transformation matrix is calculated to adapt the product image to the model image. The spatial transformation matrix employs a combination of affine and perspective transformations, and includes at least a scaling factor, rotation angle, and perspective transformation parameters. The calculation process uses least squares estimation to solve for the optimal transformation parameters between keypoint pairs: first, outlier matching pairs are eliminated using the RANSAC (Random Sample Consensus) algorithm; then, a system of linear equations is constructed, using the body keypoint coordinates as the target points and the contact area keypoint coordinates as the source points, to solve for the projection coefficients of the transformation matrix. The scaling factor is calculated based on the relative size of the keypoint bounding box; the rotation angle is estimated through principal component analysis to determine the orientation of the product in the model coordinate system; and the perspective transformation parameters are obtained through homography matrix decomposition. The transformation matrix is stored as a 3×3 floating-point matrix to ensure mathematical invertibility and numerical stability.
[0118] Based on the spatial transformation matrix, bilinear interpolation and pixel-level resampling are performed on the product image. The resampling process projects the product image from the source coordinate system to the target model coordinate system: First, for each pixel in the target image, its corresponding position in the source image is calculated through an inverse transformation; then, a bilinear interpolation algorithm is used to calculate the target pixel value based on the weighted average of the four neighboring pixels in the source image, with the weights determined by the relative distance between pixels. The interpolation operation preserves the texture details of the product image while eliminating jagged edges, outputting a smooth resampled image. Pixel-level resampling introduces a boundary processing mechanism, filling areas exceeding the boundaries of the source image with transparent pixels or environment pixels to ensure seamless integration.
[0119] A dense correspondence field is created based on the alignment results, between the pixel coordinates of the resampled product image and the pixel coordinates of the model image. The dense correspondence field is a two-dimensional vector field that encodes the displacement vector of each pixel between the resampled product image and the model image. The creation process uses an optical flow estimation network based on the PWC-Net (Pyramidal Warping and Cost Volume Network) architecture. Taking the resampled product image and model image as input, it extracts multi-scale features through a feature pyramid and calculates the correlation volume at each scale, ultimately regressing the dense flow field. The output of the correspondence field is a floating-point tensor with the same spatial dimensions as the input image and a channel dimension of 2, representing the displacement in the x and y directions, respectively. This correspondence field provides spatial guidance for the subsequent diffusion model, ensuring the naturalness and geometric consistency of the interaction between the product and the model.
[0120] Furthermore, you can also view Figure 5 , Figure 5This is a schematic diagram of another detailed step based on step S30 in the first embodiment. Figure 5 The step of using a diffusion model to generate an initial product image based on the product wearing instructions includes steps S35-36:
[0121] Step S35: Input the latent space vector of the product image and the product wearing instructions into the conditional denoising network of the diffusion model for denoising iterative operation, and obtain a clean latent representation through the iterative operation results;
[0122] Step S36: The VAE decoder based on the diffusion model reconstructs the clean latent representation, and the initial product image is obtained based on the reconstruction result.
[0123] In this embodiment, a refined implementation of the initial product image is generated using a diffusion model. Through a conditional denoising network and a VAE decoder, the latent space vector of the product image and the product image command are converted into a high-quality initial product image.
[0124] The latent space vector of the product image and the product wearing instructions are input into the conditional denoising network of the diffusion model for iterative denoising computation. The conditional denoising network is built on the U-Net architecture and includes an encoder-decoder structure and skip connections. The latent space vector of the product image is pre-extracted using a VAE encoder and represented as low-dimensional latent features; the product wearing instructions are converted into a sequence of conditional tokens through an embedding layer. The denoising iteration adopts the DDPM framework. During the backdiffusion process, the conditional denoising network receives the noisy latent vector and the time step embedding, and fuses the conditional instructions through a cross-attention layer. Specifically, the conditional tokens, as key-value pairs, interact with the query sequence of the noisy latent features to modulate the denoising direction. Each iteration calculates the noise residual prediction and updates the latent representation through a scheduler, as expressed by the formula:
[0125] ;
[0126] Among them, the As the current noise potential, the For the noise predicted by the network, the and stated As scheduling parameters, the As a noise scale, the For random noise, the For variables The result of the square root mathematical operation. The denoising iterative process is executed for a preset number of steps (e.g., 50 steps) to gradually remove noise and output a clean latent representation.
[0127] Furthermore, the VAE (Variational Autoencoder) decoder based on the diffusion model reconstructs the clean latent representation to generate the initial product image. The VAE decoder consists of a symmetric convolutional network containing multiple upsampling and convolutional layers, upsampling the low-dimensional clean latent representation to the original image space. The reconstruction process first expands the feature map space size through transposed convolutional layers, then refines local details through residual blocks, and finally generates an RGB image through an output convolutional layer. The VAE decoder introduces a style modulation mechanism, where the product image instruction is injected through an Adaptive Instance Normalization (AdaIN) layer to adjust the mean and variance of the feature map and enhance semantic consistency. The reconstruction result is constrained by a sigmoid activation function to output the initial product image in the format of a standard RGB array. During training, mean squared error reconstruction loss and perceptual loss are jointly optimized to ensure image visual quality and detail fidelity. Through the conditional denoising and refined reconstruction processes described above, controllable generation of high-quality images from multimodal instructions is achieved.
[0128] Based on the above Figure 5 The content described in step S35 is further refined here. Specifically, the step of inputting the latent space vector of the product image and the product wearing instructions into the conditional denoising network of the diffusion model for denoising iterative calculation, and obtaining a clean latent representation through the iterative calculation results includes steps S35-1 to S35-4:
[0129] Step S35-1: Calculate the conditional gradient of the product wearing instruction using a cross-attention mechanism;
[0130] Step S35-2: Based on the geometric correspondence between the product image and the preset model image, generate a spatial constraint mask, and apply the spatial constraint mask to the latent space vector of the product image to obtain the noise latent representation of the current time step;
[0131] Step S35-3: Input the conditional gradient calculation results and the noise latent representation of the current time step into the U-Net noise predictor of the diffusion model to predict the noise residual;
[0132] Step S35-4: Based on the noise residual, use a stochastic differential equation solver to calculate and update the noise latent representation for the next time step, and obtain the clean latent representation based on the update result.
[0133] In this embodiment, the conditional gradient is calculated through a cross-attention mechanism, spatial constraint masks are generated, noise residuals are predicted, and noisy latent representations are updated. This process gradually denoises and generates clean latent representations, ensuring precise control and semantic consistency in the diffusion generation process.
[0134] The conditional gradient of the product application instructions is calculated using a cross-attention mechanism. This conditional gradient calculation is based on a transformer architecture, where the product application instructions serve as a key-value sequence, and the latent space vector of the product image serves as a query sequence. Specifically, the latent space vector is mapped to a query vector through a linear projection layer, and the product application instructions are mapped to key and value vectors through an embedding layer. The cross-attention layer calculates the dot product of the query and the key, applies a softmax function to generate attention weights, and outputs the conditional features through a weighted sum of the weights and value vectors. The conditional gradient is derived from the conditional features by calculating the partial derivative of the latent space vector with respect to the conditional features through backpropagation, expressed by the following formula: Where L is the loss function between the conditional features and the target, and x is the latent space vector. Gradient calculation uses the chain rule to optimize the guiding role of conditional features in the denoising process, ensuring that the generated image conforms to the instruction constraints. During training, the conditional gradient is optimized using the mean squared error loss function, and gradient clipping techniques are used to prevent numerical overflow.
[0135] Based on the geometric correspondence between the product image and the preset model image, a spatial constraint mask is generated. This mask is then applied to the latent space vector of the product image to obtain a noise latent representation for the current time step. The spatial constraint mask generation process uses dense correspondence field data, which is output from step S34 above and encodes pixel-level displacement vectors. The mask is created through binarization, where high-confidence corresponding regions are assigned a value of 1, and low-confidence or abnormal regions are assigned a value of 0. The mask size is aligned with the latent space vector. During application, the mask is multiplied element-wise with the latent space vector to suppress noise interference from irrelevant regions and enhance the latent representation of the interaction area between the product and the model.
[0136] The noise latent representation at the current time step is generated using the forward formula of the diffusion process:
[0137] ;
[0138] Among them, the Let the initial latent space vector be the vector. The noise is Gaussian. For accumulating scheduling parameters, the For variables The square root calculation result, the for The square root result. Masking ensures that noise is added within the spatially constrained region, improving the local accuracy of denoising.
[0139] The conditional gradient calculation result and the noise latent representation at the current time step are input into the U-Net noise predictor of the diffusion model to predict the noise residual. The U-Net noise predictor is implemented using an encoder-decoder structure. Specifically, the encoder extracts multi-scale features through convolutional layers and downsampling layers, and the decoder recovers spatial details through upsampling layers and skip connections. The conditional gradient, as an additional input, is mapped to the intermediate layer of U-Net through a conditional projection layer and concatenated with the noise latent features along the channel dimension. The noise prediction process uses residual blocks to process the concatenated features. Each residual block ensemble is normalized and processed with the SiLU activation function, and finally, the noise residual is predicted through the output convolutional layer. The noise residual represents the difference between the current noise latent and the clean latent, and the prediction formula is: Where x_t is the noise latent representation of the current time step, t is the time step embedding, and the... This is the conditional gradient. During training, the noise predictor uses the mean squared error loss function to optimize the match between the noise residuals and the true noise.
[0140] Based on the noise residual, a stochastic differential equation solver is used to calculate and update the noise latent representation for the next time step, and the clean latent representation is obtained based on the update result. The stochastic differential equation solver is implemented based on the DDIM or PLMS algorithm, and updates the latent representation through a discretized back-diffusion process. The update formula is: , wherein and stated As scheduling parameters, the As a noise scale, the For standard Gaussian noise, the For network prediction noise, the For variables The square root result is obtained. The solver iterates, gradually reducing the noise component in the noisy latent representation. After each update, the convergence of the latent representation is checked, for example, by judging it through a latent norm change threshold. A clean latent representation is obtained at the end of the iteration, that is, when time step t=0, the latent representation no longer contains perceptible noise and the semantics are consistent with the product wearing instructions. The entire iterative process balances efficiency and quality through adaptive step size control, and finally outputs a clean latent representation for VAE decoder reconstruction.
[0141] Based on the above Figure 5The content described in step S36 is further refined to include steps S36-1 to S36-3: The VAE decoder based on the diffusion model reconstructs the clean latent representation, and the initial product image is obtained based on the reconstruction result.
[0142] Step S36-1: The clean latent representation is upsampled hierarchically through a series of transposed convolutional networks in the VAE decoder, and a spatial feature map is output based on the sampling results.
[0143] Step S36-2: Input the spatial feature map into the activation function for nonlinear transformation, and in the feature fusion layer of the VAE decoder, fuse the transformed spatial feature map with the multi-scale features of the model image;
[0144] Step S36-3: Perform a convolution operation on the fusion result, map the number of channels of the convolution operation result to the RGB color space, and generate the initial product image.
[0145] In this embodiment, the clean latent representation is reconstructed into the initial product image through the multi-layer transposed convolutional network, nonlinear transformation and feature fusion, and RGB channel mapping of the VAE decoder, ensuring the visual quality and detail fidelity of the generated image.
[0146] The clean latent representation is hierarchically upsampled using a series of multi-layer transposed convolutional networks in the VAE decoder. The multi-layer transposed convolutional network consists of five cascaded transposed convolutional blocks, each containing a transposed convolutional layer, a group normalization layer, and a GELU activation function. The transposed convolutional layers use 4×4 convolutional kernels with a stride of 2 and padding of 1, achieving a 2x upsampling. The input size of the clean latent representation is 64×64×512. After the first transposed convolutional block, it is upsampled to 128×128×256; the second block to 256×256×128; the third block to 512×512×64; and the fourth block to 1024×1024×32, finally outputting a 1024×1024×16 spatial feature map. Residual connections are introduced during the hierarchical upsampling process to concatenate low-level and high-level features along the channel dimension, preserving multi-scale details. The spatial feature map is output as a floating-point tensor, which encodes the geometric structure and texture basis of the reconstructed image.
[0147] The spatial feature map is input into an activation function for nonlinear transformation. In the feature fusion layer of the VAE decoder, the transformed spatial feature map is fused with the multi-scale features of the model image. The nonlinear transformation uses the GELU activation function, applied independently to each channel of the spatial feature map, enhancing the expressive power of the features and introducing nonlinear relationships. The feature fusion layer is implemented based on a cross-attention mechanism, where the transformed spatial feature map serves as the query sequence, and the multi-scale features of the model image serve as the key-value sequence. The multi-scale features of the model image are extracted by a pre-trained VAE encoder, including feature pyramids at four scales (256×256×64, 128×128×128, 64×64×256, 32×32×512). The fusion process calculates the similarity matrix between the query and the key, generates attention weights through softmax normalization, and then performs a weighted summation to output the fused features. The fused features undergo channel compression through convolutional layers to eliminate redundant information, ensuring semantic consistency and visual coherence.
[0148] The fusion result is subjected to convolution, and the number of channels of the convolution result is mapped to the RGB color space to generate the initial product image. The convolution operation is implemented by three consecutive convolutional layers: the first layer uses a 3×3 convolution kernel with 16 output channels; the second layer uses a 1×1 convolution kernel with 8 output channels; and the third layer uses a 1×1 convolution kernel with 3 output channels, corresponding to the three RGB channels. After each convolution, group normalization and the GELU activation function are applied, and finally, the pixel values are constrained to the range [-1, 1] by the tanh function. The spatial size of the convolution operation remains unchanged (1024×1024), and the final output initial product image is a standard RGB image with a resolution of 1024×1024. A pixel-level weighting mechanism is introduced during the generation process. According to the spatial constraints in the product image command, different weights are assigned to the product area and the background area to ensure the highlighting of product details. During training, this step uses mean squared error loss and perceptual loss for joint optimization, and adversarial training is used to enhance the realism of the image. The entire S36-1 to S36-3 process achieves the controllable generation of high-quality marketing digital human images through a refined reconstruction process.
[0149] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the digital human generation method based on the product. Any simple modifications based on this technical concept are within the protection scope of this application.
[0150] This application provides a digital human generation device based on a product, the digital human generation device based on a product includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the digital human generation method based on a product in the first embodiment described above.
[0151] The following is for reference. Figure 8 The diagram illustrates a structural schematic suitable for implementing the product-based digital human generation device in the embodiments of this application. The product-based digital human generation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The digital human generation device based on the product shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0152] like Figure 8As shown, the digital human generation device based on a product-like body may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the digital human generation device based on the product-like body. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the product-based digital human generation device to exchange data wirelessly or via wired communication with other devices. Although the figure shows product-based digital human generation devices with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.
[0153] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.
[0154] The digital human generation device based on product images provided in this application, employing the product-based digital human generation method described in the above embodiments, addresses the technical problem of poor product image fusion in existing digital human marketing solutions. Compared to the prior art, the beneficial effects of the product-based digital human generation device provided in this application are the same as those of the product-based digital human generation method described in the above embodiments, and other technical features of this product-based digital human generation device are the same as those disclosed in the previous embodiment method, and will not be repeated here.
[0155] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0156] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0157] This application provides a storage medium, which is a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the digital human generation method based on product body in the above embodiments.
[0158] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0159] The aforementioned computer-readable storage medium may be included in a digital human generation device based on a product; or it may exist independently and not incorporated into a digital human generation device based on a product.
[0160] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the product-based digital human generation device, enable the product-based digital human generation device to implement the technical content of the product-based digital human generation method embodiment shown above.
[0161] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0162] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0163] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.
[0164] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described digital human generation method based on product body image, which can solve the technical problem of poor product body image fusion effect in existing digital human marketing solutions. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the digital human generation method based on product body image provided in the above embodiments, and will not be repeated here.
Claims
1. A method for generating digital humans based on product body images, characterized in that, The method for generating digital humans based on product bodies includes the following steps: Obtain text instructions and product images, input the text instructions and product images into a multimodal model, and obtain the semantic association features and visual detail features output by the multimodal model; The semantic association features and the visual detail features are fused using a cross-attention mechanism, and the product wearing instructions are obtained based on the fusion result. In response to the product wearing instruction, the image ratio and spatial perspective of the product image and the preset model image are adjusted, and the adjustment results are used to generate an initial product wearing image using a diffusion model; Locate the modification area of the initial product image and verify the consistency between the image information in the modification area and the corresponding area in the product image and the preset model image. When the verification result is consistent, call the text rendering strategy to render the initial product image to obtain the product image. The text rendering strategy is a text editing operation based on the initial product image. The steps for adjusting the image ratio and spatial perspective of the product image and the preset model image include: Extract the coordinates of key body points related to product interaction from the preset model image, and the coordinates of key contact areas from the preset product image; Based on the body key point coordinates and the contact area key point coordinates, a spatial transformation matrix is calculated to adapt the product image to the preset model image. The transformation matrix includes at least a scaling factor, a rotation angle, and a perspective transformation parameter. Based on the spatial transformation matrix, bilinear interpolation and pixel-level resampling are performed on the product image, and the product image and the preset model image are aligned using the operation results and sampling results. Based on the alignment results, a dense correspondence field is created between the pixel coordinates of the resampled product image and the pixel coordinates of the preset model image.
2. The digital human generation method based on product body as described in claim 1, characterized in that, The step of inputting the text instruction and the product image into a multimodal model to obtain the output semantic association features and visual detail features includes: The text instructions are parsed using the semantic encoding path of the multimodal model, and semantic association features are generated based on the parsing results; Using the appearance encoding pathway of the multimodal model, visual detail features corresponding to the semantic association features are extracted from the product image.
3. The digital human generation method based on product-based body model as described in claim 1, characterized in that, The step of fusing the semantic association features and the visual detail features through a cross-attention mechanism, and obtaining the product wearing instructions based on the fusion result, includes: The semantic association features are used as query sequences, and the visual detail features are used as key and value sequences, which are then input into the cross-attention layer. Based on the cross-attention layer, the relevance weights between the query sequence and the key sequence are calculated, and the value sequence is weighted and summed according to the relevance weights to generate a fusion feature representation; After performing linear transformation and normalization on the fused feature representation, the product wearing instruction is generated.
4. The digital human generation method based on product body as described in claim 3, characterized in that, The step of generating the product wearing instruction after performing linear transformation and normalization on the fused feature representation includes: The fused feature representation is input into an adaptive normalization layer, where a dynamic scaling factor and a bias factor are generated based on the semantic association features. The fused feature representation is subjected to channel affine transformation using the scaling factor and the bias factor, and the fused feature representation after channel affine transformation is normalized. The normalized fusion feature representation is then subjected to gated linear filtering, and the product wearing instructions are obtained based on the filtering results.
5. The digital human generation method based on product body as described in claim 1, characterized in that, The step of using the diffusion model to generate an initial product image based on the product wearing instructions includes: The latent space vector of the product image and the product wearing instructions are input into the conditional denoising network of the diffusion model for denoising iterative operation, and a clean latent representation is obtained through the iterative operation results; The VAE decoder based on the diffusion model reconstructs the clean latent representation, and the initial product image is obtained based on the reconstruction result.
6. The digital human generation method based on product body as described in claim 5, characterized in that, The step of inputting the latent space vector of the product image and the product wearing instructions into the conditional denoising network of the diffusion model for denoising iterative computation, and obtaining a clean latent representation through the iterative computation results, includes: The conditional gradient of the product wearing instruction is calculated using a cross-attention mechanism; Based on the geometric correspondence between the product image and the preset model image, a spatial constraint mask is generated, and the spatial constraint mask is applied to the latent space vector of the product image to obtain the noise latent representation of the current time step; The conditional gradient calculation results and the noise latent representation at the current time step are input into the U-Net noise predictor of the diffusion model to predict the noise residual. Based on the noise residual, a stochastic differential equation solver is used to calculate and update the noise latent representation for the next time step, and the clean latent representation is obtained based on the update result.
7. The digital human generation method based on product body as described in claim 5, characterized in that, The VAE decoder based on the diffusion model reconstructs the clean latent representation, and the initial product image obtained from the reconstruction result includes: The clean latent representation is upsampled hierarchically through a series of transposed convolutional networks in the VAE decoder, and a spatial feature map is output based on the sampling results. The spatial feature map is input into the activation function for nonlinear transformation, and in the feature fusion layer of the VAE decoder, the transformed spatial feature map is fused with the multi-scale features of the preset model image. The fusion result is subjected to a convolution operation, and the number of channels of the convolution operation result is mapped to the RGB color space to generate the initial product image.
8. A digital human generation device based on a product's body, characterized in that, The digital human generation device based on the product body stores a computer program, which, when executed by a processor, implements the digital human generation method based on the product body as described in any one of claims 1-7.
9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the digital human generation method based on a product body as described in any one of claims 1-7.