Conditional diffusion model-based holding digital model generation method and system

Abstract

The invention belongs to the technical field of computer vision and generative artificial intelligence, and particularly relates to an object-holding digital model generation method and system based on a conditional diffusion model, and the method comprises the steps: mapping the relative position relation between model joints and a contrast object to corresponding joint points in a model chart, and obtaining the target position data of a corresponding commodity; scaling a to-be-displayed commodity in the commodity graph according to the target scale factor to obtain a commodity scaling feature; performing conditional diffusion by using a conditional diffusion model, and generating an image of the object-holding / wearing commodity in the digital model according to the posture reference image as an object-holding model image; a contrast object in the posture reference graph is used as a position and proportion reference, and accurate proportion control is achieved; it is ensured that the proportion of the commodity graph in the model graph is natural, and the proportion distortion problem in a traditional method is solved; the mandatory posture reference image comprises a contrast object, the contour and key points of the contrast object are used as position reference, manual adjustment is not needed, and the automation degree and the generation precision are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and generative artificial intelligence, and particularly relates to a holding digital model generation method and system based on a conditional diffusion model. BACKGROUND

[0002] In the field of e-commerce, advertisement design, etc., images of models holding or wearing goods need to be frequently generated. The traditional method has the following defects:

[0003] 1. Inconsistent scale: when fusing a product image with a model image, the size of the product often leads to scale distortion due to differences in viewing angle and model posture (e.g., the product is too large / small or not held naturally).

[0004] 2. Ambiguous posture reference: existing technologies lack clear references for the position and size of the held object, and manual parameter adjustment is required, which is inefficient and lacks accuracy.

[0005] Therefore, there is an urgent need for an automatic generation method based on clear posture references and accurate scale calculation. SUMMARY

[0006] The application provides a holding digital model generation method and system based on a conditional diffusion model to solve the problem of inconsistent scale or ambiguous posture reference in existing product images.

[0007] In one aspect, the application provides a holding digital model generation method based on a conditional diffusion model, comprising:

[0008] receiving a model image, a product image, and a posture reference image input by a user; the posture reference image has a model and a reference object;

[0009] extracting model joint position data in the model image and the relative size of the reference object in the posture reference image, determining a target scale factor of the target held / worn object; extracting the relative position relationship between the model joints and the reference object in the posture reference image;

[0010] mapping the relative position relationship between the model joints and the reference object to the corresponding joint nodes in the model image to obtain target position data of the corresponding product; scaling the product image to be displayed according to the target scale factor to obtain product scaling features;

[0011] inputting the model joint position data, the product scaling features, and the target position data into the conditional diffusion model for conditional diffusion to generate an image of the digital model holding / wearing the product according to the posture reference image as a holding model image.

[0012] Preferably, the method for obtaining the relative position relationship is:

[0013] In the pose reference map, the outline of the comparison object, the holding object key point and the key point of the comparison object are detected, the outline of the comparison object is represented by a mask, the holding object key point is the contact point of the comparison object and the model, and the key point of the comparison object includes a geometric center and a size;

[0014] The relative position relationship between the model joint and the comparison object is established based on the mask of the comparison object, the holding object key point and the key point of the comparison object, including distance and angle.

[0015] Preferably, the determination method of the target scale factor is:

[0016] The overall size of the pose reference map and the size of the comparison object are obtained, and the relative size of the comparison object in the pose reference map is calculated;

[0017] The joint position in contact with the comparison object in the model joint position data is obtained, the expected size of the model holding / wearing area is calculated, and the target scale factor is generated.

[0018] Preferably, the design method of the conditional diffusion model is:

[0019] Set the input conditions: extract the model joint position data, the scaled product scaling feature and the mask of the comparison object;

[0020] A geometric condition embedding layer is added on the basis of the basic U-Net structure, the position, scale and angle are encoded into a 128-dimensional vector in the geometric condition embedding layer, which is input in parallel with the text embedding layer to obtain a modified U-Net structure; The geometric conditions in the geometric condition embedding layer include target position, rotation angle, scale factor and human body joint point;

[0021] The input conditions are input into the modified U-Net structure to generate an image of the digital model holding / wearing the product in the pose in the pose reference map as the holding model image.

[0022] Preferably, after obtaining the holding model image, Poisson blending is used to optimize the edge blending of the holding model image to obtain a clearer holding model image.

[0023] Preferably, the model image is input into the human pose model to detect the human body joint point to obtain the model joint position data.

[0024] Preferably, the human pose model uses OpenPose.

[0025] Preferably, after receiving the model image, the product image and the pose reference map input by the user, the model image, the product image and the pose reference map are normalized in size.

[0026] Preferably, the target position data is obtained by:

[0027] computing the relative size of the reference item in the pose reference image , wherein: ;

[0028] According to the joint position data of the model in contact with the goods, the expected size of the model holding / wearing the goods is calculated, the target scaling factor is generated, and the target scaling factor The calculation formula is:

[0029] ;

[0030] In the formula, is the width in the pose reference image, is the height in the pose reference image; is the width in the model image, is the height in the model image;

[0031] Map the relative position relationship between the holding key points in the pose reference image and the key points of the reference item to the corresponding joint of the model in the model image, to obtain the target position data of the corresponding goods.

[0032] Another aspect of the present application is a holding digital model generation system based on a conditional diffusion model, comprising:

[0033] An input module for receiving a model image, a goods image and a pose reference image input by a user; the pose reference image has a model and a reference item;

[0034] A detection module for extracting model joint position data in the model image and relative size of the reference item in the pose reference image, determining a target scaling factor of the target holding / wearing; extracting the relative position relationship between the model joints in the pose reference image and the reference item;

[0035] A calculation module for mapping the relative position relationship between the model joints and the reference item to the corresponding joint in the model image, to obtain the target position data of the corresponding goods; scaling the goods to be displayed in the goods image according to the target scaling factor, to obtain the goods scaling feature;

[0036] A diffusion generation module for inputting the model joint position data, the goods scaling feature and the target position data into the conditional diffusion model, performing conditional diffusion, and generating an image of the digital model holding / wearing the goods in the pose reference image as a holding model image.

[0037] The holding digital model generation method and system based on the conditional diffusion model of the present application use the reference item in the pose reference image as a position and scaling reference to achieve precise scaling control; ensure the natural scaling of the goods image in the model image, and solve the scaling distortion problem in the traditional method;

[0038] The reference posture includes a contrast article, and the contour and key points thereof are used as position references, without manual adjustment, thereby improving the automation degree and generation accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions provided by the present application, the following will briefly introduce the drawings. Obviously, the drawings described below are only some embodiments of the present application.

[0040] Figure 1 The figure is a schematic diagram of the overall process in the first embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0042] A holding digital model generation method based on a conditional diffusion model, the overall design is:

[0043] Receiving a model image, a product image and a posture reference image input by a user; the posture reference image has a model and a contrast article;

[0044] Extracting model joint position data in the model image and relative sizes of the contrast article in the posture reference image, determining a target scale factor of a target holding / wearing product; extracting relative position relationships between the model joints and the contrast article in the posture reference image;

[0045] Mapping the relative position relationships between the model joints and the contrast article to corresponding joint nodes in the model image to obtain target position data of the corresponding product; scaling a product to be displayed in the product image according to the target scale factor to obtain product scaling features;

[0046] Inputting the model joint position data, the product scaling features and the target position data into a conditional diffusion model to perform conditional diffusion, generating an image of the digital model holding / wearing the product in the posture reference image as a holding model image.

[0047] The contrast article in the posture reference image is used as a position and scale reference to map and synchronously fuse the product in the product image into the model in the model image, solving the scale distortion problem of product and model fusion, and realizing automatic and high-precision holding / wearing model image generation.

[0048] Embodiment one, a holding digital model generation method based on a conditional diffusion model, as shown in Figure 1As shown, specifically comprising the following steps:

[0049] Step S100, input preprocessing: receiving user input model picture M, product picture G and posture reference picture P, wherein the posture reference picture must contain an article consistent with the holding / wearing method of the product to be displayed, denoted as the control article;

[0050] The model picture has various joints of the human body, including hands, wrists, shoulders, legs, etc.; the model in the model picture is a digital model, which can adjust the posture through data;

[0051] The product picture is the product to be displayed.

[0052] Preferably, after receiving the user input model picture, product picture and posture reference picture, the model picture, product picture and posture reference picture are size normalized, such as unified to 512x512 pixels, to preliminarily eliminate image errors.

[0053] Step S200, input the model picture into the human body posture estimation model, detect the human body joints, and arrange to obtain the model joint position data;

[0054] Preferably, the human body posture estimation model uses OpenPose to detect 18 human body joints.

[0055] Step S300, in the posture reference picture, the relative position relationship is obtained, including: the outline of the control article, the holding key point and the key point of the control article; the relative position relationship of the control article in the posture reference picture is established. The holding key point includes the contact point between the model's hand and the control article, etc.

[0056] Preferably, the key points of the control article include the geometric center and the size of the control article; the relative position relationship includes the distance and angle between the control article and the hand joint.

[0057] Preferably, the method for obtaining the relative position relationship is:

[0058] In the posture reference picture, the outline of the control article, the holding key point and the key point of the control article are detected by the deep learning model MaskR-CNN, and the outline of the control article is represented by a mask;

[0059] Based on the mask of the control article, the holding key point and the key point of the control article, the relative position relationship between the model joint and the control article is established.

[0060] Step S400, mapping the relative position relationship: mapping the relative position relationship between the model joints and the reference object to the corresponding joint nodes in the model graph to obtain the target position data of the corresponding commodity; calculating the relative size of the reference object in the posture reference graph according to the target position data; then determining the target area size of the model in the model graph according to the holding key point to generate a target scaling factor.

[0061] Mapping the relative position relationship based on the target scaling factor to determine the target position of the corresponding commodity in the model graph. The target position includes coordinates and rotation angles. Scaling the commodity to be displayed in the commodity graph according to the target scaling factor to obtain commodity scaling features.

[0062] Preferably, the method for determining the target scaling factor is:

[0063] Obtaining the overall size of the posture reference graph and the size of the reference object, and calculating the relative size of the reference object in the posture reference graph;

[0064] Obtaining the joint position of the model joint position data in contact with the reference object, calculating the expected size of the model holding / wearing area, and generating a target scaling factor.

[0065] Preferably, the method for obtaining the target position data is:

[0066] Calculating the relative size of the reference object in the posture reference graph , ;

[0067] According to the joint position of the model joint position data in contact with the commodity, calculating the expected size of the model holding / wearing commodity, generating a target scaling factor, and the calculation formula of the target scaling factor is:

[0068] ;

[0069] In the formula, is the width in the posture reference graph, is the height in the posture reference graph; is the width in the model graph, is the height in the model graph;

[0070] Mapping the relative position relationship between the holding key point in the posture reference graph and the key point of the reference object to the holding / wearing corresponding joint node of the model in the model graph to obtain the target position data of the corresponding commodity.

[0071] Step S500, according to the relative position relationship of the reference object in the posture reference map and the target position of the reference object in the model map, using the conditional diffusion model to generate an image of the digital model holding / dressing the commodity in the posture in the posture reference map as a holding model image.

[0072] Preferably, after obtaining the holding model image, edge fusion optimization is performed on the holding model image using Poisson fusion to eliminate the boundary artifacts of the commodity and the digital model, and a clearer holding model image is obtained. Of course, other conventional means can also be used for edge fusion optimization, which are within the design scope of the present application.

[0073] Preferably, the design method of the conditional diffusion model is:

[0074] Set input conditions: extract human joint features in the model map; scale the commodity map according to the scale factor to obtain the scaled commodity position feature; extract the mask of the reference object; and obtain the target position, rotation angle, scale factor and human joint node of the reference object in the model map;

[0075] Generate a conditional diffusion model based on the basic graph, add a geometric condition embedding layer based on the basic U-Net structure, encode the position, scale and angle into a 128-dimensional vector in the geometric condition embedding layer, and input it in parallel with the text embedding layer to obtain a modified U-Net structure;

[0076] Input the input conditions into the geometric condition embedding layer of the modified U-Net structure to generate an image of the digital model holding the reference object in the posture in the posture reference map as a holding model image.

[0077] In one specific example, the model map is a model front view, the commodity map is a sports shoe commodity map, the model holds a virtual shoe of the same size in the posture reference map, and the reference object is a virtual shoe; the system detects the size of the virtual shoe in the posture reference map and the relative position of the hand, scales the commodity in the commodity map to the same scale, and maps it to the hand position in the model map, and generates a realistic image of the model map M holding the commodity map G through the conditional diffusion model.

[0078] In summary, the present application uses the reference object in the posture reference map as a position and scale reference to achieve precise scale control; ensures that the scale of the commodity map in the model map is natural, and solves the scale distortion problem in traditional methods;

[0079] The posture reference map forcibly includes the reference object, which uses the outline and key points as position references, eliminating the need for manual adjustment and improving the automation level and generation accuracy.

[0080] It is suitable for e-commerce commodity display, advertisement design and other scenes, greatly reduces the cost of manual modeling, and supports rapid customization of holding model image generation.

[0081] Embodiment two, a holding digital model generation system based on a conditional diffusion model, comprising:

[0082] An input module is configured to receive a model image, a product image and a pose reference image input by a user; the pose reference image has a model and a reference product;

[0083] A detection module is configured to extract model joint position data in the model image and relative sizes of the reference product in the pose reference image, determine a target scale factor of a target holding / wearing product; and extract relative position relationships between model joints and the reference product in the pose reference image;

[0084] A calculation module is configured to map the relative position relationships between the model joints and the reference product to corresponding joint nodes in the model image, to obtain target position data of the corresponding product; and scale a product to be displayed in the product image according to the target scale factor, to obtain product scaling features;

[0085] A diffusion generation module is configured to input the model joint position data, the product scaling features and the target position data into a conditional diffusion model, perform conditional diffusion, and generate an image of a digital model holding / wearing a product in the pose reference image as a holding model image.

[0086] Through the coordinated cooperation of the four modules, the reference product is effectively mapped as a position and a scale, and natural product generation in the model is achieved.

[0087] Preferably, the system further comprises a post-processing module configured to perform edge fusion optimization on the holding model image using Poisson fusion, to eliminate boundary artifacts of the product and the digital model, and obtain a clearer holding model image.

[0088] Finally, it should be noted that: in the drawings of the disclosed embodiments, only structures related to the disclosed embodiments are involved, other structures can be referred to the usual design, and in the case of no conflict, the same embodiment and different embodiments of the present application can be combined with each other;

[0089] Finally: the above only describes the preferred embodiments of the present application and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for generating a digital mannequin based on a conditional diffusion model, characterized in that, The method comprises the following steps: receiving a model picture, a product picture and a posture reference picture input by a user; the posture reference picture has a model and a reference product; extracting model joint position data in the model picture and relative size of the reference product in the posture reference picture, and determining a target scale factor of a target holding / dressing product; extracting relative position relationship between the model joint and the reference product in the posture reference picture; mapping the relative position relationship between the model joint and the reference product to corresponding joint nodes in the model picture to obtain target position data of the corresponding product; scaling the product to be displayed in the product picture according to the target scale factor to obtain product scaling features; inputting the model joint position data, the product scaling features and the target position data into a conditional diffusion model to perform conditional diffusion, and generating an image of the digital model holding / dressing the product in the posture reference picture as a holding model image. 2.The method of claim 1, wherein, The method for obtaining the relative position relationship comprises the following steps: in the posture reference picture, detecting an outline of the reference product, holding key points of the reference product and key points of the reference product, the outline of the reference product being represented by a mask, the holding key points being contact points between the reference product and the model, and the key points of the reference product including a geometric center and a size; establishing the relative position relationship between the model joint and the reference product based on the mask of the reference product, the holding key points and the key points of the reference product, including distance and angle. 3.The method of claim 2, wherein, The method for determining the target scale factor comprises the following steps: obtaining overall size of the posture reference picture and size of the reference product, and calculating relative size of the reference product in the posture reference picture; obtaining joint positions in contact with the reference product in the model joint position data, calculating expected size of a model holding / dressing area, and generating the target scale factor. 4.The method of claim 2, wherein, The method for designing the conditional diffusion model comprises the following steps: setting input conditions: extracting model joint position data, scaled product scaling features and a mask of the reference product; adding a geometric condition embedding layer to a basic U-Net structure, encoding position, scale and angle into a 128-dimensional vector in the geometric condition embedding layer, inputting the vector in parallel with a text embedding layer to obtain a modified U-Net structure; the geometric conditions in the geometric condition embedding layer include target position, rotation angle, scale factor and human body joint nodes; inputting the input conditions into the modified U-Net structure to generate an image of the digital model holding / dressing the product in the posture reference picture as a holding model image. 5.The method of claim 1, wherein, After obtaining the holding model image, edge fusion optimization is performed on the holding model image using Poisson fusion to obtain a clearer holding model image. 6.The method of claim 1, wherein, The model picture is input into a human posture model to perform human body joint detection and obtain model joint position data.

7. The conditional diffusion model based on the holding digital model generation method of claim 6, wherein, The human posture model adopts OpenPose. 8.The method of claim 1, wherein, After receiving the model picture, the product picture and the posture reference picture input by the user, size normalization is performed on the model picture, the product picture and the posture reference picture. 9.The method of claim 2, wherein, The method for obtaining the target position data comprises the following steps: Computing relative sizes of control items in a posture reference map To: ; According to the joint position data of the model and the joint position data of the commodity in contact, the expected size of the model holding / dressing the commodity is calculated, a target scale factor is generated, the target scale factor The calculation formula is: ； wherein is the width in the pose reference image, is the height in the pose reference image; is the width in the model image, is the height in the model image; mapping relative position relationship between the holding key points and the key points of the reference product in the posture reference picture to corresponding joint nodes of the model holding / dressing area in the model picture to obtain target position data of the corresponding product.

10. A system for generating a digital mannequin based on a conditional diffusion model, employing the method of any one of claims 1-9, wherein, The method comprises the following steps: An input module is configured to receive a model image, a product image and a pose reference image input by a user; The pose reference image has a model and a reference object; A detection module is configured to extract model joint position data in the model image and relative sizes of the reference object in the pose reference image, and determine a target scale factor of a target held / dressed object; The relative position relationship between the model joints and the reference object in the pose reference image is extracted; A calculation module is configured to map the relative position relationship between the model joints and the reference object to corresponding joint nodes in the model image, to obtain target position data of a corresponding product; The product to be displayed in the product image is scaled according to the target scale factor to obtain product scaling features; A diffusion generation module is configured to input the model joint position data, the product scaling features and the target position data into a conditional diffusion model, perform conditional diffusion, and generate an image of the digital model holding / dressing the product in the pose reference image as a held model image.

Images