Commodity handheld display method and device of generative digital human

Abstract

The invention provides a commodity handheld display method and device for a generative digital human, and relates to the technical field of digital human handheld image generation, and the method comprises the steps: extracting key point information of a hand through a pre-trained key point detection model, and enabling the key point information to serve as a condition input to be embedded into a DiT model; the key point information is subjected to position coding and heat map conversion and is deeply fused with image features through a cross attention mechanism, so that the model can dynamically pay attention to a key area where a hand is in contact with a commodity, and a more natural and reasonable handheld posture is generated. Besides, key point alignment loss is introduced in the model training process, the distance between key points of the generated image and the target image is calculated, the generation effect is further optimized, and the accuracy and naturalness of the hand posture are ensured. The method aims at solving the problems that in a traditional display method, postures are not natural, and the method cannot adapt to different commodity sizes.

Description

Technical Field

[0001] The present invention relates to the technical field of digital human handheld image generation, and in particular to a method and device for generating a digital human handheld commodity display. Background Art

[0002] With the continuous development of digital technology, the application of digital human technology in product display is becoming increasingly widespread. Traditional product display methods typically rely on physical models or static images, but these methods have many limitations, such as high cost, lack of flexibility, and difficulty in achieving personalized displays. Digital human technology can effectively address these issues by generating virtual digital models to display products, while also providing richer and more diverse display effects.

[0003] However, existing digital human product display technology still has some shortcomings. Currently, a common technical solution is static template overlay technology. In this technology, after the user uploads a product image, the system uses simple image segmentation (such as background cutout) to overlay the product image onto a pre-stored digital human template (such as the position of a model's hands in a fixed pose) to generate a static display image or simple animation. For example, the SAM model is used to extract the main body of the product, and alpha blending is used to achieve the overlay of the product and hand images. Limited background replacement (such as solid color or simple scenes) is also supported.

[0004] Although this technology can realize the digital display of goods to a certain extent, it has obvious shortcomings. First, the posture of contact between the goods and the hands is often unreasonable, the hand is held stiffly, and the contact point is distorted. For example, when displaying flat goods (such as mobile phones), the mobile phone is placed flat on the palm, but the palm concave deformation is not simulated according to the effect of gravity, resulting in an unnatural display effect. Secondly, this technology cannot dynamically adjust the hand posture according to the size of the goods. For example, for large-sized bags, a single-handed grasping template is still used, which is obviously not in line with the actual usage scenario, further reducing the authenticity and credibility of the display. The effects of existing technical solutions are as follows: Figure 1 shown.

[0005] Furthermore, existing technical solutions also lack the diversity and flexibility of generated images. Relying on fixed templates, the generated images often lack naturalness and diversity, making it difficult to meet the display needs of different users and scenarios. Simply put, existing technologies for displaying digital humans holding products suffer from problems such as unreasonable posture, an inability to dynamically adjust hand posture, and a lack of naturalness and diversity in the generated images.

[0006] In view of this, this application is filed. Summary of the Invention

[0007] The present invention provides a method and device for handheld display of commodities using a generative digital human, which can at least partially improve the above-mentioned problems.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A method for handheld display of goods using a generative digital human, comprising:

[0010] Obtain the product image to be processed, call the pre-trained key point detection model, and generate multiple key point coordinates of the hand in the existing hand-held product action image;

[0011] Preprocessing the key point coordinates and the product image to obtain key point features and image features;

[0012] Based on the DiT model and the cross attention mechanism, the key point features and the image features are fused to generate a fused image;

[0013] The L1 distance between the key points extracted from the fused image and the preset target key points is calculated according to the total loss function to generate a loss result, and the fused image is adjusted according to the loss result to obtain the final generated image; wherein the total loss function includes noise prediction loss and key point alignment loss.

[0014] The present invention also provides a handheld commodity display device of a generative digital human, which comprises:

[0015] A key point extraction unit is used to obtain the product image to be processed, call the pre-trained key point detection model, and generate the coordinates of multiple key points of the hand in the existing hand-held product action image;

[0016] A preprocessing unit, configured to preprocess the key point coordinates and the product image to obtain key point features and image features;

[0017] A fusion unit, configured to fuse the key point features and image features based on a DiT model and a cross attention mechanism to generate a fused image;

[0018] A loss adjustment unit is used to calculate the L1 distance between the key points extracted from the fused image and the preset target key points according to the total loss function, generate a loss result, and adjust the fused image according to the loss result to obtain a final generated image; wherein the total loss function includes noise prediction loss and key point alignment loss.

[0019] In summary, traditional techniques typically rely on static template overlays. While simple, this approach has numerous limitations in practical applications, such as stiff hand gestures and an inability to dynamically adjust hand movements based on product size and shape, resulting in unnatural presentations. To address these issues, a generative digital human product hand-held presentation method was proposed, aiming to enhance the realism and naturalness of product presentations. This method leverages the Diffusion Transformer (DiT) model, combined with geometric prior information, to dynamically adjust hand gestures through keypoint detection technology, achieving a more natural and plausible product hand-held presentation. Specifically, hand keypoint information is extracted using a pre-trained keypoint detection model and embedded as conditional input into the DiT model. This keypoint information undergoes specialized processing, such as position encoding and heatmap conversion, to enable deep fusion with image features. Through a cross-attention mechanism, the model dynamically focuses on key areas of contact between the hand and the product, generating a more natural hand-held presentation.

[0020] In addition, the method introduces an optimized loss function that further optimizes the accuracy and naturalness of the hand pose by calculating the distance between the key points of the generated image and the target image. This loss function design not only improves the quality of the generated image but also ensures the rationality of the hand pose, avoiding abnormal postures such as reverse joints.

[0021] Simply put, the innovation of this generative digital human handheld product display method lies in combining deep learning technology with geometric prior information. By dynamically adjusting hand posture, this method significantly improves the realism and naturalness of the digital human handheld product display. Compared with traditional technologies, this method not only solves existing problems but also provides a more efficient, flexible, and realistic solution for digital human product display. This technology has significant application value and broad market prospects in a variety of fields, including e-commerce, virtual try-on, and advertising display. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a rendering of the prior art solution provided by the present invention;

[0023] Figure 2 This is a flow chart of the method for handheld display of goods using a generative digital human provided by the first embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the framework of the method for handheld display of goods using a generative digital human provided by an embodiment of the present invention;

[0025] Figure 4 21 key points of the hand provided in an embodiment of the present invention;

[0026] Figure 5 This is a diagram of the DiT model design framework provided by an embodiment of the present invention;

[0027] Figure 6 This is an effect diagram of the method for hand-held display of goods using a generative digital human provided by an embodiment of the present invention;

[0028] Figure 7 This is a module diagram of a handheld commodity display device of a generative digital human provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0030] refer to Figure 2 、 Figure 3 As shown, the first embodiment of the present invention discloses a method for handheld display of goods using a generative digital human. The method can be executed by a handheld display device for goods using a generative digital human (hereinafter referred to as a display device), and in particular, by one or more processors within the display device to implement the following method:

[0031] See also Figure 4 ,S1, obtains the product image to be processed, calls the pre-trained key point detection model, and generates multiple key point coordinates of the hand in the existing hand-held product action picture;

[0032] Specifically, step S1 includes: the key point detection model may adopt an OpenPose model.

[0033] In this embodiment, first, the product image to be processed is obtained. The product image can be a picture uploaded by the user or an image extracted from the product database; the resolution and format of the image need to meet the preprocessing requirements so that it can smoothly enter the subsequent key point detection link.

[0034] Next, the pre-trained key point detection model is called to generate the coordinates of multiple key points of the hands in the existing hand-held product action pictures. In this embodiment, the key point detection model uses the OpenPose model. OpenPose is a deep learning model widely used in human posture estimation, which can accurately detect the coordinates of multiple key points of the hands, such as wrists, finger joints, etc. These key point coordinates provide important reference information for subsequent gesture adjustment and image fusion. Through the OpenPose model, the coordinates of 21 key points of the hand can be automatically generated, and these coordinates will be embedded in the subsequent DiT (Diffusion Transformer) model as geometric prior information.

[0035] S2, preprocessing the key point coordinates and the product image to obtain key point features and image features;

[0036] Specifically, step S2 includes: normalizing the coordinates of the key points to within a preset range, and the formula is: Among them, x min 、x max 、y min 、y max These are the image boundary coordinates;

[0037] Get the normalized coordinate pair (x norm ,y norm );

[0038] The coordinate pairs are converted into a probability distribution map, and local context information is captured to generate a heat map of the same size as the image or after downsampling. The formula of the heat map is: (x, y) is the normalized keypoint coordinate scaled to the heatmap size, σ is the standard deviation, i is the row where the current pixel is located (vertical position, corresponding to the y-axis direction of the original image), and j is the column where the current pixel is located (horizontal position, corresponding to the x-axis direction of the original image). Each keypoint generates a Gaussian distribution, and the center of each Gaussian distribution is integrated to obtain the keypoint feature.

[0039] Divide the product image into multiple rectangular blocks of the same size, and flatten each rectangular block into a vector;

[0040] The vector is mapped to a preset fixed-dimensional embedding space through linear transformation and encoded to obtain image features.

[0041] In this embodiment, when pre-processing the key point coordinates, they are first normalized to a preset range. Normalization is achieved through a specific formula that uses the image boundary coordinates to normalize the key point coordinates; so as to normalize the original pixel coordinates to the range of [0,1] or [-1,1] to eliminate the influence of image size. Specifically, the parameters in the formula represent the boundary coordinates of the image respectively. By performing such normalization on the key point coordinates, they can be unaffected by the difference in image size, thereby improving the accuracy and consistency of subsequent processing. The normalized coordinate pair will serve as the basic data for subsequent processing, where the horizontal and vertical coordinates are represented by specific symbols respectively.

[0042] Next, the normalized coordinate pairs are converted into probability distribution maps to capture local context information. This process is achieved by generating heat maps, the formula of which is based on Gaussian distribution, where the key point coordinates are normalized and scaled to the heat map size. The standard deviation is an important parameter that determines the range of the Gaussian distribution, which in turn affects the accuracy and details of the heat map. By generating a Gaussian distribution for each key point and integrating the centers of these Gaussian distributions, the key point features are finally obtained. The key point features are encoded structured information. This representation of key point features not only retains the precise location information of the key points, but also reflects the local context information around the key points, providing rich details for subsequent fusion.

[0043] Specifically, the coordinates are converted into a probability distribution map to capture the local context (such as the bending direction of the finger joints and the relationship between adjacent keypoints). The resolution is further adjusted to generate a heatmap of the same size as the input image or a downsampled one (e.g., 64x64). Finally, the Gaussian kernel parameters are obtained, and a Gaussian distribution is generated for each keypoint, centered at the coordinate location and with a standard deviation σ controlling the distribution range (typically 2 to 5 pixels).

[0044] At the same time, a series of preprocessing operations are also performed on the product images. First, the product image is divided into multiple rectangular blocks of the same size. This is a common image processing method that can decompose the image into smaller blocks that are easier to process. Then, each rectangular block is flattened into a vector for subsequent processing. Through linear transformation, these vectors are mapped to a preset fixed-dimensional embedding space and encoded to obtain image features. In the DiT model, image features refer to patch embeddings. This process enables the product image to be represented in a form suitable for subsequent processing, and also lays the foundation for the fusion of image features and key point features.

[0045] See also Figure 5,S3, based on the DiT model and cross attention mechanism, the key point features and image features are fused to generate a fused image;

[0046] Specifically, step S3 includes: under the condition of the DiT model, aligning the dimension of the key point feature with the dimension of the image feature through linear projection;

[0047] After the alignment is completed, based on the cross attention mechanism, the cross attention weight is calculated to dynamically focus the key point features on the relevant areas in the image features to generate a fused image. The calculation formula is: Q=W q Keypoints, K=W k Image, V = W v Image, Q is the query matrix, K is the key matrix, V is the value matrix, T is the transpose, d is, Softmax is the normalization function, d is the dimension, W q 、W k 、W v Both are learnable weight matrices, Keypoints equals key point features, and Image equals image features.

[0048] In this embodiment, based on the DiT model and the cross-attention mechanism, the key point features and image features are fused to generate a high-quality fused image. Under the conditions of the DiT model, the dimensions of the key point features (as the query matrix) are aligned with the dimensions of the image features (as the key matrix and the value matrix) through linear projection. This alignment process is achieved through a set of learnable weight matrices that can map features from different modalities (key point features and image features) into a common feature space. This dimensional alignment is the basis for subsequent fusion, ensuring that features from different sources can effectively interact and fuse in the same dimension. In this way, the model can better understand the relationship between key point features and image features, laying the foundation for generating high-quality fused images.

[0049] After dimensional alignment, the cross-attention mechanism calculates cross-attention weights, allowing keypoint features to dynamically focus on relevant areas within the image features (e.g., the position of a hand holding a product). The core of this process is to dynamically determine the correlation between keypoint features and image features through the interaction between the query matrix (Query), key matrix (Key), and value matrix (Value). Based on this, the model can dynamically focus on the areas within the image features that are most relevant to the keypoint features, thereby achieving deep feature fusion.

[0050] The introduction of the cross-attention mechanism is a key innovation of this method. It not only allows the model to dynamically focus on the areas of the image most relevant to hand keypoints, but also adaptively adjusts its focus based on different inputs. This dynamic focusing capability significantly improves the realism and naturalness of the generated images, avoiding the problems common in traditional methods such as stiff hand gestures and distorted contact points. Through the cross-attention mechanism, the model can better understand the spatial relationship between the hand and the product, generating more natural and reasonable product holding postures.

[0051] See also Figure 6 , S4, calculates the L1 distance between the key point extracted from the fused image and the preset target key point according to the total loss function, generates a loss result, and adjusts the fused image according to the loss result to obtain the final generated image; wherein the total loss function includes noise prediction loss and key point alignment loss.

[0052] Specifically, step S4 includes: diffusion On the basis of, increase the key point alignment loss L krypoints , get the total loss function;

[0053] A pre-trained key point detector is used to extract key points from the fused image, and the L1 distance between the key points and the preset target key points is calculated to generate a loss result.

[0054] The formula of the total loss function: L total =L diffusion +λ·L krypoints , where λ is the weight coefficient, is the noise prediction loss, x t is the noisy image of the tth step, ∈ is the true noise, ∈ θ is the predicted noise, is the key point alignment loss, G θ (x t ) is the generated denoising function, and φ(·) is the pre-trained keypoint detector.

[0055] In this embodiment, a key point alignment loss is added to the noise prediction loss to construct a total loss function. Noise prediction loss is a loss function commonly used in diffusion models, which is used to measure the difference between the noise predicted by the model and the actual noise. By minimizing this difference, the model can more accurately recover clear image content from noisy images. However, relying solely on the noise prediction loss can generate relatively clear images, but there may still be deficiencies in the accuracy and naturalness of the hand posture. Therefore, a key point alignment loss is further introduced to ensure that the hand posture in the generated image is highly consistent with the preset target key points.

[0056] The keypoint alignment loss is calculated as follows: a pre-trained keypoint detector is used to extract keypoints from the fused image and calculate the L1 distance between them and the preset target keypoints. L1 distance is a commonly used metric that directly reflects the difference between the coordinates of two keypoints. By minimizing this difference, the model can more accurately adjust the hand pose, making it more natural and reasonable. This keypoint alignment loss design allows the model to focus more on the spatial relationship between the hand and the product when generating images, avoiding the problems of stiff hand gestures and distorted contact points common in traditional methods.

[0057] Specifically, the total loss function consists of a noise prediction loss and a keypoint alignment loss. The noise prediction loss measures the difference between the noisy image at step t and the true noise, while the keypoint alignment loss measures the L1 distance between the keypoints in the generated image and the target keypoints. The weight coefficient balances the two losses, ensuring that the model balances image clarity and hand pose accuracy during optimization.

[0058] Through this comprehensive total loss function, the model can simultaneously optimize the image denoising effect and the accuracy of the hand posture when generating images. During the training process, the model will continuously adjust the pixel values and key point positions in the generated image based on the feedback of the total loss function, thereby gradually improving the quality of the generated image. Ultimately, through this optimization process, the model is able to generate high-quality fused images in which the hand posture is natural and reasonable, and highly integrated with the display effect of the product. Simply put, based on the powerful base film capability of the DiT model, further constraints on the finger parts help to achieve the naturalness of the fingers generated in the image of holding the product.

[0059] In summary, this generative digital human handheld product display method based on a total loss function not only significantly improves the realism and naturalness of the generated images, but also provides a more efficient, flexible, and realistic solution for digital human handheld product display. Compared with traditional methods, this method can better adapt to the needs of different products and scenarios, generating more diverse and personalized display effects. This method has significant application value and broad market prospects in a variety of fields, such as e-commerce, virtual try-on, and advertising display.

[0060] See also Figure 7 The second embodiment of the present invention provides a handheld product display device of a generative digital human, which includes:

[0061] The key point extraction unit 101 is used to obtain a product image to be processed, call a pre-trained key point detection model, and generate multiple key point coordinates of the hand in an existing picture of the hand-held product action;

[0062] A preprocessing unit 102 is used to preprocess the key point coordinates and the product image to obtain key point features and image features;

[0063] A fusion unit 103 is configured to fuse the key point features and image features based on a DiT model and a cross attention mechanism to generate a fused image;

[0064] The loss adjustment unit 104 is used to calculate the L1 distance between the key point extracted from the fused image and the preset target key point according to the total loss function, generate a loss result, and adjust the fused image according to the loss result to obtain the final generated image; wherein the total loss function includes noise prediction loss and key point alignment loss.

[0065] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for handheld display of goods using a generative digital human, characterized in that: include: Obtain the product image to be processed, call the pre-trained key point detection model, and generate multiple key point coordinates of the hand in the existing hand-held product action image; Preprocessing the key point coordinates and the product image to obtain key point features and image features; Based on the DiT model and the cross attention mechanism, the key point features and the image features are fused to generate a fused image; The L1 distance between the key points extracted from the fused image and the preset target key points is calculated according to the total loss function to generate a loss result, and the fused image is adjusted according to the loss result to obtain the final generated image; wherein the total loss function includes noise prediction loss and key point alignment loss.

2. The method for handheld display of goods using a generative digital human according to claim 1, characterized in that: The key point detection model adopts the OpenPose model.

3. The method for handheld display of goods using a generative digital human according to claim 1, characterized in that: Preprocess the key point coordinates and product images to obtain key point features and image features, specifically: The key point coordinates are normalized to within a preset range, and the formula is: Among them, x min 、x max 、y min 、y max These are the image boundary coordinates; Get the normalized coordinate pair (x norm ,y norm ); The coordinate pairs are converted into a probability distribution map, and local context information is captured to generate a heat map of the same size as the image or after downsampling. The formula of the heat map is: (x, y) is the normalized keypoint coordinate scaled to the heatmap size, σ is the standard deviation, i is the row where the current pixel is located, and j is the column where the current pixel is located. Each keypoint generates a Gaussian distribution, and the center of each Gaussian distribution is integrated to obtain the keypoint feature.

4. The method for handheld display of goods using a generative digital human according to claim 3, characterized in that: Also includes: Divide the product image into multiple rectangular blocks of the same size, and flatten each rectangular block into a vector; The vector is mapped to a preset fixed-dimensional embedding space through linear transformation and encoded to obtain image features.

5. The method for handheld display of goods using a generative digital human according to claim 1, characterized in that: Based on the DiT model and the cross attention mechanism, the key point features and image features are fused to generate a fused image, specifically: Under the conditions of the DiT model, the dimensions of the key point features are aligned with the dimensions of the image features through linear projection; After the alignment is completed, based on the cross attention mechanism, the cross attention weight is calculated to dynamically focus the key point features on the relevant areas in the image features to generate a fused image. The calculation formula is: Q=W q Keypoints, K=W k Image, V = W v Image, Q is the query matrix, K is the key matrix, V is the value matrix, T is the transpose, d is, Softmax is the normalization function, d is the dimension, W q 、W k 、W v Both are learnable weight matrices, Keypoints equals key point features, and Image equals image features.

6. The method for handheld display of goods using a generative digital human according to claim 5, characterized in that: The L1 distance between the key points extracted from the fused image and the preset target key points is calculated according to the total loss function to generate the loss result, which is: In the noise prediction loss L diffusion On the basis of, increase the key point alignment loss L krypoints , get the total loss function; A pre-trained key point detector is used to extract key points from the fused image, and the L1 distance between the key points and the preset target key points is calculated to generate a loss result.

7. The method for handheld display of goods using a generative digital human according to claim 6, characterized in that: The formula of the total loss function: L total =L diffysion +λ·L krypoints , where λ is the weight coefficient, is the noise prediction loss, x t is the noisy image of the tth step, ∈ is the true noise, ∈ θ is the predicted noise, is the key point alignment loss, G θ (x t ) is the generated denoising function, and φ(·) is the pre-trained keypoint detector.

8. A handheld display device for commodities using a generative digital human, characterized in that: include: A key point extraction unit is used to obtain the product image to be processed, call the pre-trained key point detection model, and generate the coordinates of multiple key points of the hand in the existing hand-held product action image; A preprocessing unit, configured to preprocess the key point coordinates and the product image to obtain key point features and image features; A fusion unit, configured to fuse the key point features and image features based on a DiT model and a cross attention mechanism to generate a fused image; A loss adjustment unit is used to calculate the L1 distance between the key points extracted from the fused image and the preset target key points according to the total loss function, generate a loss result, and adjust the fused image according to the loss result to obtain a final generated image; wherein the total loss function includes noise prediction loss and key point alignment loss.

Images