Commodity picture generation method and device, computer equipment and storage medium

Abstract

The invention relates to a commodity picture generation method and device, computer equipment and a storage medium. The method comprises the following steps: receiving a to-be-processed commodity graph; determining a target commodity from the to-be-processed commodity graph, and extracting a commodity main body image of the target commodity; obtaining first description information of the commodity main body image, wherein the first description information is used for describing the target commodity; the commodity main body image and the first description information are input into a diffusion model to generate a target commodity graph containing the commodity main body image, the diffusion model comprises a pre-trained LoRA model and a ControlNet model, the LoRA model is used for generating a scene in the target commodity graph, and the ControlNet model is used for generating the scene in the target commodity graph. The ControlNet model is used for controlling the output quality of the diffusion model. By adopting the method, the high-quality commodity display image can be accurately generated, and the high consistency of the commodity main body in the generated image and the original image is ensured.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, computer equipment and storage medium for generating product images. Background Art

[0002] The traditional methods of generating e-commerce images mainly include designer design, programmatic design platform and simple application based on generative algorithm. Although the widely adopted designer design method can ensure creativity and personalization, and is suitable for the production of banner advertising images at many important promotional nodes, its disadvantages are that it is time-consuming and laborious, and the number of images produced is limited, making it difficult to respond quickly to market changes and large-scale advertising needs. On the other hand, although the traditional programmatic design platform has realized the automation of the design process and reduced labor costs, due to its reliance on template design and the support of expert systems, the generated images are relatively simple in style, lacking creativity, and have high requirements on the quality and quantity of original materials. In essence, it is still a combination and optimization of existing materials.

[0003] In addition, although simple applications based on generative algorithms can automatically produce design drawings through algorithms, the generated images are not ideal. There are problems such as poor light and shadow processing, stiff content, and low resolution. Designers are still required to provide certain prior knowledge, similar to the "card drawing" mode, which is difficult to meet the high standards of creative e-commerce graphic design. Among them, Stable Diffusion is a diffusion model developed by Stability AI, CompVis and Runway in 2022 based on LatentDiffusion. Due to its open source nature, it has spawned many creative drawing tools and workflows based on this model, but the image generation of Stable Diffusion is not stable and is a common "card drawing" mode. Therefore, when generating e-commerce graphics related to specific brands, models and products, it is difficult to achieve controllable generation of content. Summary of the invention

[0004] Based on this, it is necessary to provide a product image generation method, device, computer equipment and storage medium that can improve the quality of image generation in response to the above technical problems.

[0005] In one aspect, a method for generating a product image is provided, the method comprising:

[0006] Receive the product image to be processed;

[0007] Determine a target commodity from the commodity image to be processed, and extract a commodity main body image of the target commodity;

[0008] Acquire first description information of the commodity main image, where the first description information is used to describe the target commodity;

[0009] The main image of the product and the first description information are input into a diffusion model to generate a target product image containing the main image of the product, wherein the diffusion model includes a pre-trained LoRA model and a ControlNet model, the LoRA model is used to generate the scene in the target product image, and the ControlNet model is used to control the output quality of the diffusion model.

[0010] In one embodiment, determining a target commodity from the commodity image to be processed and extracting a commodity main body image of the target commodity includes:

[0011] Generate a ternary graph of the commodity graph to be processed by a deep learning model;

[0012] Based on the to-be-processed commodity image and the ternary image, obtaining a commodity main body image of the target commodity through a depth image mask and / or a guided context attention mask; or,

[0013] The commodity image to be processed is segmented by a pre-trained segmentation model to obtain a commodity main body image of the target commodity.

[0014] In one embodiment, before segmenting the commodity image to be processed by using the pre-trained segmentation model, the process further includes:

[0015] Collecting sample image data of different commodities to obtain a segmented data set, wherein the segmented data set includes sample images of different commodities and commodity main images obtained by segmenting the sample images;

[0016] The segmentation model is trained using the segmentation data set to obtain a first training result;

[0017] Adjust the learning rate and number of iterations of the segmentation model according to the accuracy information and loss information of the first training result.

[0018] In one embodiment, the obtaining of first description information of the main image of the commodity includes:

[0019] The attribute features of the target product are identified in the product main image, and first description information of the target product is generated based on the identified attribute features.

[0020] In one embodiment, before inputting the main product image and the description information into a diffusion model to generate a target product image including the main product image, the method further includes:

[0021] Collecting sample image data of different scenes to obtain a scene training set, wherein the scene training set includes sample images of different scenes and second description information obtained by annotating the sample images;

[0022] Training the LoRA model using the scenario training set to obtain a second training result;

[0023] Adjust the learning rate and number of iterations of the LoRA model according to the accuracy information and loss information of the second training result.

[0024] In one embodiment, the step of inputting the main product image and the first description information into a diffusion model to generate a target product image containing the main product image includes:

[0025] Determine a target scenario, and determine a target LoRA model based on the target scenario;

[0026] The main image of the product image and the first description information are used as conditional inputs of the ControlNet model, so that the ControlNet model guides the diffusion model to fuse the main image of the product with the scene generated by the target LoRA model according to the edge features or depth features of the main image of the product, so as to obtain the target product image.

[0027] In one embodiment, after inputting the main product image and the first description information into a diffusion model to generate a target product image including the main product image, the method further includes:

[0028] Segment the target product image to obtain multiple image blocks;

[0029] Performing at least one super-resolution process on the multiple image blocks through a super-resolution model to obtain multiple target image blocks;

[0030] The multiple target images are stitched together to obtain a final product image.

[0031] On the other hand, a device for generating a product image is provided, the device comprising:

[0032] A receiving module, used for receiving a product image to be processed;

[0033] An extraction module, used to determine a target commodity from the commodity image to be processed, and extract a commodity main body image of the target commodity;

[0034] An acquisition module, used to acquire description information of the main image of the commodity, wherein the description information is used to describe the target commodity;

[0035] A generation module is used to input the main image of the product and the description information into a diffusion model to generate a target product image containing the main image of the product, wherein the diffusion model includes a ControlNet model and a pre-trained LoRA model, the LoRA model is used to generate the scene in the target product image, and the ControlNet model is used to control the output quality of the diffusion model.

[0036] In another aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0037] Receive the product image to be processed;

[0038] Determine a target commodity from the commodity image to be processed, and extract a commodity main body image of the target commodity;

[0039] Acquire first description information of the commodity main image, where the first description information is used to describe the target commodity;

[0040] The main image of the product and the first description information are input into a diffusion model to generate a target product image containing the main image of the product, wherein the diffusion model includes a pre-trained LoRA model and a ControlNet model, the LoRA model is used to generate the scene in the target product image, and the ControlNet model is used to control the output quality of the diffusion model.

[0041] In another aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0042] Receive the product image to be processed;

[0043] Determine a target commodity from the commodity image to be processed, and extract a commodity main body image of the target commodity;

[0044] Acquire first description information of the commodity main image, where the first description information is used to describe the target commodity;

[0045] The main image of the product and the first description information are input into a diffusion model to generate a target product image containing the main image of the product, wherein the diffusion model includes a pre-trained LoRA model and a ControlNet model, the LoRA model is used to generate the scene in the target product image, and the ControlNet model is used to control the output quality of the diffusion model.

[0046] The above-mentioned product image generation method, device, computer equipment and storage medium can generate the target product image by determining the target product from the product image to be processed and extracting its main product image, and inputting the first description information corresponding to the target product into the diffusion model. It can not only accurately generate high-quality product display images, but also ensure the high consistency between the main product in the generated image and the original image, and realize the reconstruction of the traditional e-commerce image synthesis by the AIGC large model. In addition, the LoRA model is used to fine-tune specific scenes to enhance the realism and immersion of the scenes in the generated images, while the ControlNet model effectively avoids image defects such as rough edges or holes caused by extracting the main product image through precise control of the diffusion model, thereby significantly improving the overall quality and visual effects of the generated images. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a process of generating a product image in one embodiment;

[0048] Figure 2 is a flowchart of a method for generating a product image in another embodiment;

[0049] Figure 3 A schematic diagram of a process for extracting a main image of a commodity in one embodiment;

[0050] Figure 4 is a structural block diagram of a picture generating device in one embodiment;

[0051] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] In one embodiment, Figure 1-3 As shown, a method for generating a product image is provided, comprising the following steps:

[0054] Step S1, receiving a product image to be processed.

[0055] Step S2, determining a target commodity from the commodity images to be processed, and extracting a commodity main body image of the target commodity.

[0056] Among them, since the product image to be processed uploaded by the user contains background and other elements that are irrelevant to the target product, it is necessary to cut out or segment the product image to be processed and extract the main image of the target product as the basis for subsequent image generation.

[0057] In one embodiment, the specific steps of determining the target product from the product image to be processed and extracting the product main image of the target product include: generating a ternary graph of the product image to be processed through a deep learning model; based on the product image to be processed and the ternary graph, obtaining the product main image of the target product through a deep image mask and / or a guided context attention mask; or, segmenting the product image to be processed through a pre-trained segmentation model to obtain the product main image of the target product.

[0058] like Figure 3 As shown in the figure, the deep learning model uses BASNet (Boundary Awareness Segmentation Network). Through the two-stage cutout algorithm based on BASNet, simple products such as white background images and transparent images uploaded by users can be quickly cutout to obtain product images. Specifically, BASNet is used to obtain the ternary image tripmap of the product image to be processed, and then the product image to be processed and the ternary image are sent to the DIM (Deep Image Matting) or GCA module (Guided Contextual Attention) to obtain the final alpha image.

[0059] The segmentation model adopts an interactive segmentation model. Users can click on the products they want to select. The segmentation model identifies the target product through the set of points clicked by the current user, and feeds the segmentation results back to the front end. Among them, the commonly used SAM model has several obvious disadvantages, such as the existence of rough mask boundaries, and in some multi-subject product images, the understanding of the subject details is insufficient, which easily leads to mis-segmentation of some detail structures. Therefore, the segmentation model in this embodiment adopts the hq_sam model improved based on the SAM model, which introduces the hq output token and a high-quality mask prediction layer (three-layer MLP). By reusing and fixing the mask decoder of SAM, a learnable hq output token (1x256) is connected with the output token (4x256) and prompt token (Nx256) of SAM and passed as input to the mask decoder of SAM. Similar to the original output token, in each attention layer, the hq output token is first self-attention calculated with other tokens, and then reverse attention of token to image and image to token is performed to update its features. The hq output token uses the point-by-point MLP shared with other tokens in each decoder layer. After passing through two decoder layers, the updated hq output token can access the global image context, prompt The important geometric type information of the token and the hidden mask information of other output tokens are added. Finally, a new three-layer MLP is added to generate dynamic convolution kernel weights from the updated hq-output token, and then point-by-point multiplication with the fused hq features is performed to generate high-quality masks.

[0060] In one embodiment, before segmenting the product image to be processed by a pre-trained segmentation model, it also includes: collecting sample image data of different products to obtain a segmentation data set, wherein the segmentation data set includes sample images of different products and product main images obtained by segmenting the sample images; training the segmentation model through the segmentation data set to obtain a first training result; and adjusting the learning rate and number of iterations of the segmentation model according to the accuracy information and loss information of the first training result.

[0061] Specifically, a batch of badcase data of product image cutouts from e-commerce platforms are collected and used as segmentation data sets for training the hq_sam model after screening, cleaning and expansion. It should be noted that the data selected here only needs to be the cutout data of the target product, that is, the main image of the product, and there is no need to use the full-image cutout data like the sam model.

[0062] The hq_sam model is trained by splitting the data set and setting the training parameters, where the learning rate is 1e-3, the input image size is 1024x1024, the batch_size is 6, the vit_h model is used, the StepLR training method is used, and the Adam optimizer is used. According to the average accuracy change and loss change trend of the cross-validation of the training set and the validation set, the learning rate and the number of iterations are adjusted until the accuracy change and the loss change gradually tend to a stable state, and the final learning rate and number of iterations are determined to obtain a well-converged hq_sam model.

[0063] Step S3: obtaining first description information of the main image of the commodity, where the first description information is used to describe the target commodity.

[0064] In one embodiment, the acquiring the first description information of the main image of the commodity includes: identifying attribute features of the target commodity in the main image of the commodity, and generating the first description information of the target commodity based on the identified attribute features.

[0065] After the main image of the target product is obtained, it will be sent to the image-generating algorithm of the diffusion model. Before that, it is necessary to determine the prompt (i.e., the first description information) for inputting the product image. The user can choose to input the prompt by himself or use the blip algorithm to generate the prompt corresponding to the main image of the product.

[0066] Among them, the blip algorithm combines image and text processing to achieve efficient multimodal understanding and generation. In the blip algorithm, a complete N-layer ViT (Vision Transformer) model is used on the image side, and three models are designed on the text side. Each model is responsible for calculating different objective functions and selecting different parts of the model according to different objective functions. The modules of these models share parameters through color coding, that is, the modules with the same color mark share parameters to ensure that only one set of model parameters needs to be updated during training. The first text model is also composed of N layers. Its main task is to understand the input text and perform classification tasks to extract text features, and then combine these features with visual features to calculate ITC (cross-modal consistency) loss. The second text model is the image-guided text encoder (Image-groundedText Encoder), which is a multimodal encoder that uses image information to perform tasks involving multimodality and then calculates ITM (image-text matching) loss. Compared with the first model, this module adds a cross-attention mechanism to fuse the information on the image side, while the self-attention and feedforward neural network (FFN) remain the same. The third model is a decoder (Decoder), the design of which is closer to the GPT model, and the input sentences will be partially masked during its training process. This means that during the training phase, the model can only predict the masked part based on the existing partial information, rather than directly using the complete sentence. Therefore, the first layer of the model uses the causal self-attention mechanism (Causal Self-Attention) instead of sharing parameters with bidirectional self-attention (Bi Self-Attention). Nevertheless, in addition to the causal self-attention mechanism, others such as cross attention and feedforward neural network still share parameters with previous modules.

[0067] Furthermore, the blip output is improved. When generating prompts, in addition to using the native text of blip, relevant words are introduced from the predefined search library to enrich the prompt output. Among them, blip-large is selected as the basic model, the text output length is set to 32, and clip ViT-L-14 is used as the search module to build five search libraries: artists, flavors, mediums, movement, and negative. The final generated prompt not only includes the output of blip, but also integrates the relevant words in these five search libraries to enhance the accuracy and diversity of the prompt. Through such a design, high-quality prompts corresponding to the target products can be automatically generated without manual input by the user, and then the user's subsequent product image generation process.

[0068] Step S4: input the main image of the product and the first description information into a diffusion model to generate a target product image containing the main image of the product, wherein the diffusion model includes a pre-trained LoRA model and a ControlNet model, the LoRA model is used to generate the scene in the target product image, and the ControlNet model is used to control the output quality of the diffusion model.

[0069] Specifically, the diffusion model uses a stablediffusion model. After inputting the main image of the product and the corresponding prompt, the inpaint mode of the stablediffusion model can be used to generate images. However, the product images generated through this link are usually unstable and it is difficult to ensure the details such as light and shadow, texture, etc., which will appear stiff. Therefore, based on the image generation, it is necessary to use the pre-trained LoRA model to fine-tune the stablediffusion model so that it can generate high-quality images in specific scenarios, such as indoor scenes such as kitchens, living rooms, and bedrooms, as well as scene images such as urban landscapes, street shots, beaches, and villages, and achieve precise control through the ControlNet model to stabilize the output results.

[0070] In one embodiment, before inputting the main image of the product and the description information into the diffusion model to generate a target product image containing the main image of the product, it also includes: collecting sample image data of different scenes to obtain a scene training set, the scene training set including sample images of different scenes and second description information obtained by annotating the sample images; training the LoRA model through the scene training set to obtain a second training result; adjusting the learning rate and number of iterations of the LoRA model according to the accuracy information and loss information of the second training result.

[0071] Specifically, the sample image data collected needs to cover different scenes, including internal scenes of e-commerce, such as e-commerce design pictures and banner advertisements, as well as various external scenes. After screening and cleaning, about 100 pictures are retained for each scene. Then the collected pictures are labeled. The labeling tool can use blip to generate text descriptions corresponding to the sample pictures of each scene, which helps the model better understand the image content and provide necessary text information for subsequent training.

[0072] The LoRA model is trained through the above scenario training set. The training of this model is based on the pre-trained stablediffusion model for fine-tuning, obtaining the LoRA weights, and setting the training parameters. The training image size is unified to 512x512 pixels, the learning rate is 1e-4, the cosine annealing strategy with restart function (cosine_with_restarts) is adopted, the optimizer selects adamw, the network dimension is set to 32, the training cycle (epoch) is 2, and only the UNet part is trained during the training process without training the text encoder. After the initial training is completed, according to the average accuracy change and loss change trend of the cross-validation of the training set and the validation set, the learning rate and number of iterations are adjusted until the accuracy change and loss change gradually tend to a stable state, and the final learning rate and number of iterations are determined to improve the accuracy and robustness of the model.

[0073] Through the above training process, a LoRA model with good convergence and adaptation to specific scenarios can be obtained. This model can generate more realistic and demand-compliant images for specific scenarios while maintaining the original model capabilities.

[0074] In one embodiment, the inputting the main image of the product and the first description information into a diffusion model to generate a target product image containing the main image of the product includes: determining a target scene, and determining a target LoRA model based on the target scene; using the main image of the product image and the first description information as conditional inputs of the ControlNet model, so that the ControlNet model guides the diffusion model to fuse the main image of the product with the scene generated by the target LoRA model according to edge features or depth features of the main image of the product, so as to obtain the target product image.

[0075] It should be noted that the generated product images sometimes have rough edges or holes due to the cutout or segmentation of the segmentation model. Such images sent to the raw image often produce strange additional outputs, so the ControlNet model is introduced to stabilize the output. Among them, the ControlNet model is an end-to-end neural network structure that controls the diffusion model to learn the input conditions of a specific task. ControlNet clones the weights of a large diffusion model into trainable copies and locked copies. The locked copies retain the model's ability to learn from a large number of images, while the trainable copies are trained on a dataset of a specific task to learn conditional control. The two modules are connected by zero convolution, in which the convolution weights gradually grow from zero to optimized parameters in a learning manner. This method retains production-ready weights and makes the training process robust on datasets of different sizes. In addition, since zero convolution does not introduce new noise to deep features, the training speed is comparable to fine-tuning the diffusion model compared to training new layers from scratch, and can even achieve efficient training on hardware such as RTX 3090Ti. Even on smaller datasets (such as 50K or even 1K), ControlNet can also show good performance.

[0076] In order to further improve the output stability, two modes of the ControlNet model are selected: Depth mode and Canny edge detection mode. The Canny mode ensures that the boundaries of the output image will not be strangely deformed or stretched, while the Depth mode ensures that the generated image has obvious layering and depth information. These two modes can be used alone or in combination according to actual needs, and can be combined with the LoRA model and applied to different scenarios.

[0077] Based on the above steps, after obtaining the main image of the product and the corresponding prompt, the user selects the corresponding scene LoRA, and then combines it with the corresponding ControlNet to generate a very stable e-commerce graph output. For example, to configure the generation parameters for this purpose, you can set the sampling steps (Sampling Steps) to 30, the sampling method (Sampling Method) to DPM++2MKarras, the image height (Height) to 512 pixels, the image width (Width) to 512 pixels, the conditional fusion scale (CFGScale) to 7, the denoising strength (Denoising Strength) to 0.45, the repair area (Inpaint Area) to 0, and the mask mode (Mask Mode) to 1.

[0078] In one embodiment, after inputting the main image of the product and the first description information into a diffusion model to generate a target product image containing the main image of the product, the method further includes: segmenting the target product image to obtain a plurality of image blocks; performing at least one super-resolution processing on the plurality of image blocks through a super-resolution model to obtain a plurality of target image blocks; and splicing the plurality of target images to obtain a final product image.

[0079] After the above steps, we can get the 512x512 pixel target product image output by the diffusion model. In order to improve the detail expression of the image and adapt it to the display requirements of higher resolution, the Real-ESRGAN algorithm is used to perform super-resolution processing of the image, that is, to expand the target product image to a larger size, usually 800x800 pixels. Among them, Real-ESRGAN is improved on the basis of ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks), aiming to solve some limitations of ESRGAN in practical applications. Specifically, Real-ESRGAN proposed four improvements to ESRGAN: 1. A new dataset construction method to enhance the complexity of the reduced-order image; 2. The introduction of a sinc filter when constructing the dataset solves the artifact problem that may be generated by traditional methods when processing images; 3. The original VGG discriminator in ESRGAN is replaced with a discriminator based on the UNet architecture. The UNet discriminator has stronger spatial perception capabilities and helps to capture subtle features in the image; 4. Spectral Normalization is introduced to stabilize the training instability caused by complex datasets and the UNet discriminator.

[0080] Specifically, the target product image is segmented, usually into 192x192 pixel blocks, with an 8-pixel overlap between each block to ensure a natural transition at the seam of the final synthesized image and maintain overall consistency and continuity. These image blocks are super-resolution processed using the R-ESRGAN 4x+ model, which can enlarge each image block to four times its original size. In order to further enhance the texture of the image, two serial links are designed to perform super-resolution processing on the original image twice to enhance its texture, and then all the super-resolution processed image blocks are reassembled to obtain the final product image, which not only improves the detail richness and clarity of the image, but also ensures the overall coordination and aesthetics of the image.

[0081] It should be understood that although Figure 1-3The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0082] In one embodiment, Figure 4 As shown, a commodity image generation device is provided, including: a receiving module, an extraction module, an acquisition module and a generation module, wherein:

[0083] A receiving module, used for receiving a product image to be processed;

[0084] An extraction module, used to determine a target commodity from the commodity image to be processed, and extract a commodity main body image of the target commodity;

[0085] An acquisition module, used to acquire description information of the main image of the commodity, wherein the description information is used to describe the target commodity;

[0086] A generation module is used to input the main image of the product and the description information into a diffusion model to generate a target product image containing the main image of the product, wherein the diffusion model includes a ControlNet model and a pre-trained LoRA model, the LoRA model is used to generate the scene in the target product image, and the ControlNet model is used to control the output quality of the diffusion model.

[0087] In one embodiment, the extraction module is also used to generate a ternary graph of the product image to be processed through a deep learning model; based on the product image to be processed and the ternary graph, obtain the product main body image of the target product through a deep image mask and / or a guided context attention mask; or, segment the product image to be processed through a pre-trained segmentation model to obtain the product main body image of the target product.

[0088] In one embodiment, the device also includes a first training module, which is used to collect sample image data of different commodities to obtain a segmented data set, wherein the segmented data set includes sample images of different commodities and commodity main images obtained by segmenting the sample images; the segmentation model is trained using the segmented data set to obtain a first training result; and the learning rate and number of iterations of the segmentation model are adjusted according to the accuracy information and loss information of the first training result.

[0089] In one embodiment, the acquisition module is further used to identify attribute features of the target product in the main product image, and generate first description information of the target product based on the identified attribute features.

[0090] In one embodiment, the device also includes a second training module, which is used to collect sample image data of different scenes to obtain a scene training set, wherein the scene training set includes sample images of different scenes and second description information obtained by annotating the sample images; the LoRA model is trained using the scene training set to obtain a second training result; and the learning rate and number of iterations of the LoRA model are adjusted according to the accuracy information and loss information of the second training result.

[0091] In one embodiment, the generation module is also used to determine the target scene and determine the target LoRA model based on the target scene; the main image of the product image and the first description information are used as conditional inputs of the ControlNet model, so that the ControlNet model guides the diffusion model to fuse the main image of the product with the scene generated by the target LoRA model according to the edge features or depth features of the main image of the product to obtain the target product image.

[0092] In one embodiment, the device also includes a super-resolution module, which is used to segment the target product image to obtain multiple image blocks; perform at least one super-resolution processing on the multiple image blocks through a super-resolution model to obtain multiple target image blocks; and splice the multiple target images to obtain a final product image.

[0093] For the specific limitations of the product image generation device, please refer to the limitations of the product image generation method above, which will not be repeated here. Each module in the above-mentioned product image generation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0094] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store commodity and scene image data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a commodity image generation method is implemented.

[0095] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0096] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0097] Receive the product image to be processed;

[0098] Determine a target commodity from the commodity image to be processed, and extract a commodity main body image of the target commodity;

[0099] Acquire first description information of the commodity main image, where the first description information is used to describe the target commodity;

[0100] The main image of the product and the first description information are input into a diffusion model to generate a target product image containing the main image of the product, wherein the diffusion model includes a pre-trained LoRA model and a ControlNet model, the LoRA model is used to generate the scene in the target product image, and the ControlNet model is used to control the output quality of the diffusion model.

[0101] In one embodiment, when the processor executes the computer program, the following steps are also implemented: generating a ternary graph of the product image to be processed through a deep learning model; based on the product image to be processed and the ternary graph, obtaining the product main body image of the target product through a deep image mask and / or a guided context attention mask; or, segmenting the product image to be processed through a pre-trained segmentation model to obtain the product main body image of the target product.

[0102] In one embodiment, when the processor executes the computer program, the following steps are also implemented: collecting sample image data of different commodities to obtain a segmented data set, wherein the segmented data set includes sample images of different commodities and commodity main images obtained by segmenting the sample images; training the segmentation model through the segmented data set to obtain a first training result; and adjusting the learning rate and number of iterations of the segmentation model according to the accuracy information and loss information of the first training result.

[0103] In one embodiment, when the processor executes the computer program, the following steps are further implemented: identifying attribute features of the target product in the main image of the product, and generating first description information of the target product based on the identified attribute features.

[0104] In one embodiment, when the processor executes the computer program, the following steps are also implemented: collecting sample image data of different scenes to obtain a scene training set, wherein the scene training set includes sample images of different scenes and second description information obtained by annotating the sample images; training the LoRA model through the scene training set to obtain a second training result; adjusting the learning rate and number of iterations of the LoRA model according to the accuracy information and loss information of the second training result.

[0105] In one embodiment, when the processor executes the computer program, the following steps are also implemented: determining a target scene, and determining a target LoRA model based on the target scene; using the main image of the product image and the first description information as conditional inputs of the ControlNet model, so that the ControlNet model guides the diffusion model to fuse the main image of the product with the scene generated by the target LoRA model according to the edge features or depth features of the main image of the product, so as to obtain the target product image.

[0106] In one embodiment, when the processor executes the computer program, the following steps are also implemented: segmenting the target product image to obtain multiple image blocks; performing at least one super-resolution processing on the multiple image blocks through a super-resolution model to obtain multiple target image blocks; and splicing the multiple target images to obtain a final product image.

[0107] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0108] Receive the product image to be processed;

[0109] Determine a target commodity from the commodity image to be processed, and extract a commodity main body image of the target commodity;

[0110] Acquire first description information of the commodity main image, where the first description information is used to describe the target commodity;

[0111] The main image of the product and the first description information are input into a diffusion model to generate a target product image containing the main image of the product, wherein the diffusion model includes a pre-trained LoRA model and a ControlNet model, the LoRA model is used to generate the scene in the target product image, and the ControlNet model is used to control the output quality of the diffusion model.

[0112] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: generating a ternary graph of the product image to be processed through a deep learning model; based on the product image to be processed and the ternary graph, obtaining the product main body image of the target product through a deep image mask and / or a guided context attention mask; or, segmenting the product image to be processed through a pre-trained segmentation model to obtain the product main body image of the target product.

[0113] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: sample image data of different commodities are collected to obtain a segmented data set, wherein the segmented data set includes sample images of different commodities and commodity main images obtained by segmenting the sample images; the segmentation model is trained using the segmented data set to obtain a first training result; and the learning rate and number of iterations of the segmentation model are adjusted according to the accuracy information and loss information of the first training result.

[0114] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: identifying attribute features of the target product in the main image of the product, and generating first description information of the target product based on the identified attribute features.

[0115] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: collecting sample image data of different scenes to obtain a scene training set, wherein the scene training set includes sample images of different scenes and second description information obtained by annotating the sample images; training the LoRA model through the scene training set to obtain a second training result; adjusting the learning rate and number of iterations of the LoRA model according to the accuracy information and loss information of the second training result.

[0116] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: determining a target scene, and determining a target LoRA model based on the target scene; using the main image of the product image and the first description information as conditional inputs of the ControlNet model, so that the ControlNet model guides the diffusion model to fuse the main image of the product with the scene generated by the target LoRA model according to the edge features or depth features of the main image of the product, so as to obtain the target product image.

[0117] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: segmenting the target product image to obtain multiple image blocks; performing at least one super-resolution processing on the multiple image blocks through a super-resolution model to obtain multiple target image blocks; and splicing the multiple target images to obtain a final product image.

[0118] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0119] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for generating a product image, characterized in that: include: Receive the product image to be processed; Determine a target commodity from the commodity image to be processed, and extract a commodity main body image of the target commodity; Acquire first description information of the commodity main image, where the first description information is used to describe the target commodity; The main image of the product and the first description information are input into a diffusion model to generate a target product image containing the main image of the product, wherein the diffusion model includes a pre-trained LoRA model and a ControlNet model, the LoRA model is used to generate the scene in the target product image, and the ControlNet model is used to control the output quality of the diffusion model.

2. The method for generating a product image according to claim 1, characterized in that: The step of determining a target commodity from the commodity image to be processed and extracting a commodity main body image of the target commodity includes: Generate a ternary graph of the commodity graph to be processed by a deep learning model; Based on the to-be-processed commodity image and the ternary image, obtaining a commodity main body image of the target commodity through a depth image mask and / or a guided context attention mask; or, The commodity image to be processed is segmented by a pre-trained segmentation model to obtain a commodity main body image of the target commodity.

3. The method for generating a product image according to claim 2, characterized in that: Before segmenting the commodity image to be processed by the pre-trained segmentation model, the method further includes: Collecting sample image data of different commodities to obtain a segmented data set, wherein the segmented data set includes sample images of different commodities and commodity main images obtained by segmenting the sample images; The segmentation model is trained using the segmentation data set to obtain a first training result; Adjust the learning rate and number of iterations of the segmentation model according to the accuracy information and loss information of the first training result.

4. The method for generating a product image according to any one of claims 1 to 3, characterized in that: The obtaining of first description information of the commodity main image includes: The attribute features of the target product are identified in the product main image, and first description information of the target product is generated based on the identified attribute features.

5. The method for generating a product image according to claim 1, characterized in that: Before inputting the commodity main image and the description information into a diffusion model to generate a target commodity image including the commodity main image, the method further includes: Collecting sample image data of different scenes to obtain a scene training set, wherein the scene training set includes sample images of different scenes and second description information obtained by annotating the sample images; Training the LoRA model using the scenario training set to obtain a second training result; Adjust the learning rate and number of iterations of the LoRA model according to the accuracy information and loss information of the second training result.

6. The method for generating a product image according to claim 1 or 5, characterized in that: The step of inputting the main product image and the first description information into a diffusion model to generate a target product image containing the main product image includes: Determine a target scenario, and determine a target LoRA model based on the target scenario; The main image of the product image and the first description information are used as conditional inputs of the ControlNet model, so that the ControlNet model guides the diffusion model to fuse the main image of the product with the scene generated by the target LoRA model according to the edge features or depth features of the main image of the product, so as to obtain the target product image.

7. The method for generating a product image according to claim 1, characterized in that: After inputting the main product image and the first description information into a diffusion model to generate a target product image including the main product image, the method further includes: Segment the target product image to obtain multiple image blocks; Performing at least one super-resolution process on the multiple image blocks through a super-resolution model to obtain multiple target image blocks; The multiple target images are stitched together to obtain a final product image.

8. A product image device, characterized in that: The device comprises: A receiving module, used for receiving a product image to be processed; An extraction module, used to determine a target commodity from the commodity image to be processed, and extract a commodity main body image of the target commodity; An acquisition module, used to acquire description information of the main image of the commodity, wherein the description information is used to describe the target commodity; A generation module is used to input the main image of the product and the description information into a diffusion model to generate a target product image containing the main image of the product, wherein the diffusion model includes a ControlNet model and a pre-trained LoRA model, the LoRA model is used to generate the scene in the target product image, and the ControlNet model is used to control the output quality of the diffusion model.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images