Fusion-enhanced e-commerce commodity graph generation method
By constructing data sets and training graph generic graph models, using multimodal large models and texture features, the problem of poor integration between products and backgrounds in e-commerce product graph generation is solved, and the edge fusion is stronger and the product invariance is achieved. The generation of e-commerce product maps has improved the generation efficiency and quality.
Patent Information
- Application Number
- CN202411302069.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-18
- Publication Date
- 2025-01-03
AI Technical Summary
When generating e-commerce product maps, the sense of integration between the product and the background is poor, which affects the quality of generation. The method of directly generating backgrounds based on the product map can easily lead to more product peripherals, making it difficult to maintain the authenticity of the product.
Build a data set, use multimodal large model and image segmentation model to find the product location, generate segmentation maps, and adjust the background through the inpainting method, combine canny's controlnet and texture features to train the graph model to enhance the sense of edge fusion and ensure the product Invariance.
It improves the sense of integration between products and background, and the edges of the generated e-commerce product map are more natural, maintains the authenticity and immutability of products, and improves the generation efficiency and quality.
Smart Images
Figure CN119251498A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to image processing technology, and more particularly to an image generating method. Background Art
[0002] With the rapid development of the Internet and e-commerce, more and more businesses are selling goods online. However, product images on e-commerce platforms often require professional photographers to shoot and edit, which is an expensive cost for small businesses or individual stores. At the same time, for e-commerce platforms, how to improve product display effects to promote sales has always been a challenge. Automatically generating product display images that meet the needs of e-commerce platforms through AI image generation technology can effectively improve efficiency and greatly reduce costs.
[0003] Generating product display images that fit e-commerce scenarios mainly involves two challenges. On the one hand, it is necessary to generate a background that fits the e-commerce scenario, and on the other hand, it is necessary to keep the main product unchanged. There are currently two mainstream technologies. One method is based on synthesis. By placing the main product in a pre-generated background, an e-commerce product image that fits the e-commerce scenario is obtained. However, there are many problems with the fusion of the product and the background, including perspective, product location and size, light and shadow, and fusion of edges and backgrounds. The other method is to directly generate a background for the product. This method usually uses an inpainting method based on a diffusion model to generate a background for the product, thereby obtaining a more realistic and natural e-commerce product image. When only inpainting is used, it often results in the generation of more products around the product, which leads to commodity This does not conform to the actual situation. The mainstream approach to this problem is to add canny-based controlnet constraints to effectively control the multiple generation around the product, and finally paste the product image back to the generated image to ensure the authenticity of the product.
[0004] The method of directly generating backgrounds based on product images is more realistic and natural than the method of synthesizing product scenes. However, in order to ensure the invariance of the product, the product image needs to be pasted back to the generated image in the end. This process often leads to the problem of product edges not being integrated with the background, thus affecting the final generated quality. Summary of the invention
[0005] In view of the shortcomings of the prior art, an object of the present invention is to provide a method for generating e-commerce product images with enhanced sense of integration.
[0006] In order to achieve the above-mentioned purpose, the technical solution of the present invention is: a method for generating an e-commerce product image with enhanced sense of integration, comprising:
[0007] (1) The steps of constructing a data set include: (101) collecting real e-commerce product image data, and using a multimodal large model to find the product body in the image, using the product name as a prompt word to find the position of the product in the image, and using the product position in the image as a prompt word to input into the image segmentation model to obtain a segmentation map of the product in the image, and extracting the product image corresponding to the segmentation map for subsequent use; (102) using the segmentation map as a mask and adjusting the prompt word to generate a background to fill the position where the product body is located as the background, and performing appropriate expansion processing on the segmentation map to increase the probability of covering all product bodies; (103) pasting the product image corresponding to the extracted segmentation map back to the e-commerce product image after background filling to obtain an e-commerce product image with poor edge fusion, thereby obtaining an e-commerce product image with poor edge fusion and a real e-commerce product image sample pair (X, Y); (104) through manual screening, the e-commerce product image (X1, Y1) with the entire product body cut out is used for subsequent precise fine-tuning;
[0008] (2) The step of training the graph-generated graph model includes: (201) using X as a condition and Y as a generated result, using a training code to train the model, and obtaining a graph-generated graph model capable of repairing edges; (202) using the e-commerce product graph (X1, Y1) to further fine-tune the graph-generated graph model;
[0009] (3) The step of generating an e-commerce product image with enhanced sense of integration includes: (301) generating an e-commerce product image with poor sense of integration by directly generating a background based on the product image; (302) enhancing the obtained e-commerce product image with poor sense of integration by using a trained graph-generated model to generate an e-commerce product image with stronger edge fusion.
[0010] Preferably, in step (3), step (304) is also included: further pasting the main body of the product back into the generated e-commerce product image, and performing appropriate image corrosion operations on the main body of the product when pasting it back.
[0011] Preferably, in step (101), it also includes: extracting multi-scale texture features from the background part of the real e-commerce product image; and storing the extracted texture features in a feature library.
[0012] Preferably, in step (301), it also includes: selecting a group of texture features in the feature library, combining the selected features into a new feature set, gradually migrating the selected texture features to the background image, and adjusting the migration order and intensity of the features in real time according to the intermediate results in the migration process.
[0013] Preferably, the texture features include color pattern, line direction, and texture repeatability.
[0014] Preferably, the feature combination process includes superposition, mixing and deformation of features.
[0015] Compared with the prior art, the present invention constructs a data set with enhanced fusion of product images and backgrounds, thereby generating e-commerce product images with more natural edge fusion and keeping the products unchanged. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The structure of the present invention and its beneficial technical effects are described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0017] Figure 1 is a flow chart of the method of the present invention;
[0018] Figure 2 is a flow chart of the steps of constructing a data set of the present invention;
[0019] Figure 3 It is a flowchart of the steps of training the graph generation model of the present invention. DETAILED DESCRIPTION
[0020] In order to make the invention purpose, technical solution and beneficial technical effects of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be understood that the specific implementation methods described in this specification are only for explaining the present invention, not for limiting the present invention.
[0021] See also Figure 1 , Figure 2 and Figure 3 The method for generating an e-commerce product image with enhanced fusion sense of the present invention includes:
[0022] 1. Dataset construction:
[0023] Collect real e-commerce product image data, represented by Y here, and use a multimodal large model such as llava to find the main body of the product in the image, use the product name as a prompt and use the Grounding DINO method to find the position of the product in the image, and input the position of the product in the image as a prompt into segment anything to obtain the segmentation map of the product in the image, and extract the product image corresponding to the segmentation map for subsequent use. Extract multi-scale texture features from the background part of the real e-commerce product image; store the extracted texture features in a feature library; these features can include color mode, line direction, texture repeatability, etc.
[0024] Using the inpainting method (such as the inpainting model provided by kandingsky3), the segmentation map is used as a mask and the prompt word is adjusted to generate the background to fill the position of the product body as the background. Since the obtained segmentation map may not accurately cover the product body, the segmentation map is appropriately expanded to increase the probability of covering the entire product body.
[0025] Paste the product image corresponding to the extracted segmentation image back to the e-commerce product image after background filling to obtain an e-commerce product image with poor edge fusion, represented by X here, thereby obtaining an e-commerce product image with poor edge fusion and a real e-commerce product image sample pair (X, Y).
[0026] In order to enable the model to generate edges better, we manually select e-commerce product images (X1, Y1) with better cutout effects, that is, those that can cut out the entire product body, for subsequent precise fine-tuning.
[0027] 2. Training the graph-based graph model with enhanced fusion sense:
[0028] Use ControlNet to perform graph-generated graph training, take X as the condition and Y as the generated result, and use the training code provided by diffusers to train the model to obtain a graph-generated graph model that can repair edges.
[0029] Use (X1, Y1) to fine-tune the model obtained in the first step to further improve the generation effect of the model.
[0030] 3. The process of generating e-commerce product images with enhanced fusion while keeping the product images unchanged:
[0031] First, the e-commerce product image with poor fusion is generated by directly generating the background based on the product image. Here, the e-commerce product image is generated by combining the controlnet based on canny and inpainting, and the product image is pasted back to the generated image to ensure the authenticity of the product, and the e-commerce product image with poor fusion is obtained. Specifically, the product is first placed on a completely black background as the initial image, and the mask image of the image is obtained to fill the background while keeping the product unchanged. Then, the edge image of the image is obtained by using the canny operator of opencv as the condition of the controlnet. At the same time, the description of the image to be generated is used as a prompt, and the above initial image, mask image, edge image and image description are input into the controlnet and impainting algorithm based on canny. Here, the diffsers library is mainly used to provide the StableDiffusionXLControlNetPipeline related methods to obtain the generated product image. Since the original real product on the product image will lose some information during the generation process, the real product image is pasted back to the generated image after the generated image is obtained to ensure the authenticity of the product, and finally the e-commerce product image with poor fusion is obtained. During the background generation process, a group of texture features in the feature library is selected, the selected features are combined into a new feature set, the selected texture features are gradually migrated to the background image, and the migration order and intensity of the features are adjusted in real time according to the intermediate results in the migration process; according to the intermediate results in the migration process, the migration order and intensity of the features are adjusted in real time to optimize the final image quality.
[0032] The trained fusion enhancement model is used to enhance the e-commerce product image with poor fusion obtained in the first step to obtain an e-commerce product image with stronger edge fusion.
[0033] Since the main body of the product will have slight changes after passing through the model, in order to ensure that the details of the product remain unchanged, the main body of the product is further pasted back into the generated product image. In order to make good use of the generated edges with a strong sense of fusion, appropriate image corrosion operations are performed on the main body of the product when pasting it back, so as to better utilize the generated edges and finally obtain an e-commerce product image with a better sense of fusion.
[0034] According to the disclosure and teaching of the above description, those skilled in the art to which the present invention belongs may also make appropriate changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the present invention should also fall within the scope of protection of the claims of the present invention. In addition, although some specific terms are used in this specification, these terms are only for the convenience of description and do not constitute any limitation to the present invention.
Claims
1. A method for generating an e-commerce product image with enhanced sense of integration, characterized in that: include: (1) The steps of constructing the data set include: (101) Collect real e-commerce product image data, and use the multimodal large model to find the product body in the image, use the product name as a prompt word to find the position of the product in the image, and input the position of the product in the image as a prompt word into the image segmentation model to obtain a segmentation map of the product in the image, and extract the product image corresponding to the segmentation map for subsequent use; (102) using the segmented image as a mask and adjusting the prompt word to generate a background to fill the location of the main body of the product as the background, and performing appropriate expansion processing on the segmented image to increase the probability of covering the entire main body of the product; (103) Paste the product image corresponding to the extracted segmentation image back to the e-commerce product image after background filling to obtain an e-commerce product image with poor edge fusion, thereby obtaining an e-commerce product image with poor edge fusion and a real e-commerce product image sample pair (X, Y); (104) Through manual screening, the e-commerce product image (X1, Y1) of the entire product body is cut out for subsequent precise fine-tuning; (2) The steps of training the graph-generated graph model include: (201) Taking X as a condition and Y as a generated result, a model is trained using a training code to obtain a graph-generated graph model capable of repairing edges; (202) further fine-tuning the graph-generated graph model using the e-commerce product graph (X1, Y1); (3) The steps of generating an e-commerce product map with enhanced fusion sense include: (301) A method of directly generating a background based on a product image is used to generate an e-commerce product image with a poor sense of integration; (302) The trained graph-based model is used to enhance the e-commerce product graph with poor fusion, so as to generate an e-commerce product graph with stronger edge fusion.
2. The method for generating e-commerce product images with enhanced sense of integration according to claim 1, characterized in that: In step (3), step (303) is also included: further pasting the main body of the product back into the generated e-commerce product image, and performing appropriate image corrosion operations on the main body of the product when pasting it back.
3. The method for generating e-commerce product images with enhanced sense of integration according to claim 1, characterized in that: In step (101), it also includes: extracting multi-scale texture features from the background part of the real e-commerce product image; and storing the extracted texture features in a feature library.
4. The method for generating e-commerce product images with enhanced sense of integration according to claim 3, characterized in that: In step (301), it also includes: selecting a group of texture features in the feature library, combining the selected features into a new feature set, gradually migrating the selected texture features to the background image, and adjusting the migration order and intensity of the features in real time according to the intermediate results of the migration process.
5. The method for generating e-commerce product images with enhanced sense of integration according to claim 3, characterized in that: The texture features include color pattern, line direction, and texture repeatability.
6. The method for generating an e-commerce product image with enhanced sense of integration according to claim 4, characterized in that: The feature combination process includes superposition, mixing and deformation of features.