Lora model training-based commodity graph-to-background method, apparatus and device, and medium

Abstract

The invention provides a Lora model training-based commodity graph background changing method and device, equipment and a medium, and the method comprises the steps: obtaining first commodity graphs of various commodities, and obtaining training data; each first commodity graph is labeled, and a corresponding label file is formed; setting a training round number, a learning rate and training image pixels, inputting training data and a label file, and performing model training to obtain a required background Lora model; carrying out matting on the second commodity graph of which the background needs to be replaced, and obtaining a commodity main body mask in the second commodity graph; extracting edge information of the second commodity graph; loading the background Lora model in a graph-to-graph algorithm, and inputting the edge information, the background cue word and the commodity main body mask into the graph-to-graph algorithm to generate a commodity replacement graph; the background cue word corresponds to the background label in the set format, so that the drawing efficiency is improved, and the drawing cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field, and in particular to a method, device, equipment and medium for changing the background of a product image based on Lora model training. Background Art

[0002] In the e-commerce industry, the production of product images mainly relies on professional photography teams or artists using Photoshop to synthesize backgrounds, all of which are manually edited. These traditional methods have significant defects: long production cycles, lack of timeliness, and difficulty in quickly adapting to market changes; at the same time, excessive reliance on manual operations leads to high costs and low efficiency, and the results are limited by personal skill levels, resulting in unstable efficiency and quality, which in turn affects the progress of product launches and store operations. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a method, device, equipment and medium for changing the background of a product image based on Lora model training, which improves the efficiency of mapping and reduces the cost of mapping.

[0004] In a first aspect, the present invention provides a method for changing the background of a product image based on Lora model training, comprising the following steps:

[0005] Step 1: Obtain first product images of various products, where the first product images are images taken at a set shooting angle, and the first product images include multiple images of the same product in different proportions in the image, to obtain training data;

[0006] Step 2: Label each of the first product images according to the set format background label through the multimodal model to form a corresponding label file;

[0007] Step 3: Set the number of training rounds, learning rate, and training image pixels, then input the training data and label file to perform model training to obtain the required background Lora model.

[0008] Step 4: Cut out the second product image whose background needs to be replaced to obtain a product main body mask; extract edge information of the second product image;

[0009] Step 5: Load the background Lora model into the image generation algorithm, input the edge information, background prompt words and product body mask into the image generation algorithm, and generate a product replacement image; the background prompt words correspond to the set format background label.

[0010] In a second aspect, the present invention provides a device for changing the background of a product image based on Lora model training, comprising:

[0011] A training data acquisition module acquires first product images of various products. The first product images are pictures taken at a set shooting angle, and each first product image includes multiple pictures of the same product at different scales in the picture, thereby obtaining training data;

[0012] A label setting module tags each of the first product images with background labels in a set format through a multi-modal model, forming corresponding label files;

[0013] A model training module sets the number of training rounds, learning rate, and training image pixels, and then inputs the training data and label files for model training to obtain the required background Lora model;

[0014] An image acquisition module performs matting on a second product image whose background needs to be replaced to obtain a product main body mask therein; and extracts the edge information of the second product image;

[0015] An image generation module loads the background Lora model in the image-to-image generation algorithm, inputs the edge information, background prompt words, and product main body mask into the image-to-image generation algorithm to generate a product replacement image; the background prompt words correspond to the background labels in the set format.

[0016] In a third aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in the first aspect is implemented.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect is implemented.

[0018] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0019] The present invention successfully solves the need of product sellers to change the background of product images. Sellers only need to simply take pictures by themselves to generate beautiful product images, greatly improving the efficiency of making product images, reducing the operation costs of sellers, and no longer requiring the use of professional camera teams and graphic designers for matting editing, etc.;

[0020] The lighting optimization of the present invention can greatly improve the efficiency of product synthesis, and can effectively solve the problem of insufficient lighting that is prone to occur in traditional lighting synthesis algorithms; making the generated product images have high authenticity, fusion effect, and lighting effect.

[0021] The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable, the specific embodiments of the present invention are specifically given below. Brief Description of the Drawings

[0022] The present invention will be further described below with reference to the drawings in conjunction with embodiments.

[0023] Figure 1 It is the flowchart of the method in Embodiment 1 of the present invention;

[0024] Figure 2 It is the structural schematic diagram of the device in Embodiment 2 of the present invention. Detailed Embodiments

[0025] The overall idea of the technical solution in the embodiments of this application is as follows:

[0026] (1) Data preprocessing. Here, various product pictures are required, which need to include multiple angles such as upward shooting, flat shooting, and downward shooting, and also cover various backgrounds, products of different sizes and proportions, that is, the proportion of the partial area of the product occupying the entire picture. Perform bicubic sharpening interpolation scaling (shrinking or enlarging) on these product pictures, scale all the images to 1024*1024, and perform operations such as clarity optimization. This step requires screening various product pictures, including various different backgrounds.

[0027] (2) Data labeling. Then, the main body and background of the processed product pictures are described in detail in a structured format, and its format includes: shooting angle, detailed list of background elements, color of the product main body, and name of the product main body. Formatting the labeling can greatly improve the role of the prompt words in the use of the Lora model.

[0028] (3) Build a dataset, and map the text labels and product images one by one.

[0029] (4) Model training. Use the Lora training script to train the Lora model. Before training the model, the model training parameters need to be set. Set the total number of training rounds to 15000, the learning rate to 0.0001, and the training image size to 1024. Just run the script to start the model training with one key.

[0030] (5) In the image-to-image generation algorithm, load the product picture Lora trained by us, set the Lora loading weight to 0.75, call the canny algorithm, and adjust the weight parameter weight to 0.9. The main purpose of this parameter is to control the influence intensity of the canny algorithm on the extraction of edge information. Input the mask image and the prompt words with scene and lighting descriptions into the image-to-image generation for mask redrawing to obtain the product replacement picture.

[0031] (6) Transfer the product replacement image into Ic-light and use the prompt words again to optimize the illumination of the image; or obtain the foreground-removed image and the background image in the product replacement image; enhance the foreground-removed image through Gamma transformation, increase the brightness of the product main body in the foreground-removed image by a first set value to obtain an enhanced image; map the background image to the HSV space, reduce the V space in the HSV space by a second set value, and then map it back to the RGB space to obtain a background-darkened image; synthesize the enhanced image and the background-darkened image to obtain a first synthesized image, and extract the first latent information from the first synthesized image through VAE decoding; synthesize the foreground-removed image and the background image to obtain a second synthesized image, and extract the first set information from the second synthesized image using the ControlNet model; send the first latent information and the first set information to the diffusion model to generate a first light and shadow guidance map; extract the second latent information from the first light and shadow guidance map, and send the second latent information and the first set information to the diffusion model to generate a second light and shadow guidance map; transfer the texture information of the first light and shadow guidance map to the second light and shadow guidance map through the high-contrast retention algorithm to obtain an intermediate image, and then use the welsh algorithm to transfer the color information of the background image to the intermediate image to obtain an optimized product image.

[0032] Model training: Refers to the training process of the Lora model. Lora is a model fine-tuning technology that reduces the number of parameters required for fine-tuning by inserting low-rank matrices into a pre-trained large model, thereby improving training efficiency and avoiding overfitting. In the application scenario of product images, Lora training is used to generate or optimize e-commerce product images so that the product images can better adapt to specific backgrounds.

[0033] Mask image: Extract the mask image of the product image to be replaced as the mask for image generation to ensure that the product main body remains unchanged during redrawing.

[0034] Canny: The Canny algorithm is an image processing technology mainly used for edge detection, that is, identifying the edges in the image, and can extract clear and accurate edge information from the image. Combined with img2img image generation to limit the edges of the product main body and ensure the consistency of the edge information of the product main body after changing the background with the edges of the product image to be replaced.

[0035] Ic-light: Can manipulate image illumination through prompt words, making the illumination of the foreground main body consistent with the background environment illumination, so that the two are integrated. Here, we use the method of prompt words and appropriately give illumination prompt words to affect the illumination of the image and improve the fusion effect between the product main body and the background.

[0036] Example 1

[0037] AsFigure 1 As shown in the figure, this embodiment provides a method for changing the background of a product image based on Lora model training, including the following steps:

[0038] Step 1: Obtain the first product images of various products. The first product images are pictures taken at a set shooting angle, and the first product images include multiple pictures of the same product in different proportions in the picture, so as to obtain training data;

[0039] Step 2: Tag each of the first product images with background labels in a set format through a multi-modal model to form corresponding label files;

[0040] Step 3: Set the number of training epochs, learning rate, and training image pixels, and then input the training data and label files for model training to obtain the required background Lora model;

[0041] Step 4: Cut out the second product image whose background needs to be changed to obtain the product main body mask therein; extract the edge information of the second product image;

[0042] Step 5: Load the background Lora model in the image-to-image generation algorithm, and input the edge information, background prompt words, and product main body mask into the image-to-image generation algorithm to generate a product replacement image; the background prompt words correspond to the background labels in the set format.

[0043] In this embodiment, preferably, it further includes Step 6: Obtain the background-removed image and the background image in the product replacement image. The background-removed image is the image of the remaining product main body part after the background is deleted in the product replacement image; enhance the background-removed image through Gamma transformation, increase the brightness of the product main body in the background-removed image by a first set value to obtain an enhanced image; map the background image to the HSV space, reduce the V space in the HSV space by a second set value, and then map it back to the RGB space to obtain a background-darkened image; synthesize the enhanced image and the background-darkened image to obtain a first synthesized image, and extract the first latent information from the first synthesized image through VAE decoding; synthesize the background-removed image and the background image to obtain a second synthesized image, and extract the first set information from the second synthesized image by using the ControlNet model; send the first latent information and the first set information to the diffusion model to generate a first light and shadow guidance image; extract the second latent information from the first light and shadow guidance image, and send the second latent information and the first set information to the diffusion model to generate a second light and shadow guidance image; transfer the texture information of the first light and shadow guidance image to the second light and shadow guidance image through the high-contrast retention algorithm to obtain an intermediate image, and then use the welsh algorithm to transfer the color information of the background image to the intermediate image to obtain an optimized product image.

[0044] In this embodiment, preferably, step 1 is specifically as follows: Obtain the first product images of various products. The first product images are pictures taken at a set shooting angle, and the first product images include multiple pictures of the same product in different proportions in the picture; Scale all the first product images so that the size of each first product image is 1024*1024 to obtain training data.

[0045] In this embodiment, preferably, step 4 is specifically as follows: Cut out the second product image whose background needs to be replaced to obtain the product main body mask therein; Call the canny algorithm to extract the edge information of the second product image, and the weight parameter weight of the canny algorithm is 0.9;

[0046] Step 5 is specifically as follows: Load the background Lora model in the image-to-image generation algorithm, and set its loading weight to 0.75. Input the edge information, background prompt words, and product main body mask into the image-to-image generation algorithm to generate a product replacement image; The background prompt words correspond to the set format background tags.

[0047] Based on the same inventive concept, the present application also provides an apparatus corresponding to the method in Embodiment 1. For details, see Embodiment 2.

[0048] Embodiment 2

[0049] As Figure 2 shown, in this embodiment, a product image background replacement apparatus based on Lora model training is provided, including:

[0050] A training data acquisition module, which acquires the first product images of various products. The first product images are pictures taken at a set shooting angle, and the first product images include multiple pictures of the same product in different proportions in the picture, to obtain training data;

[0051] A label setting module, which tags each of the first product images according to the set format background tags through a multi-modal model to form a corresponding label file;

[0052] A model training module, which sets the number of training rounds, learning rate, and training image pixels, and then inputs the training data and the label file for model training to obtain the required background Lora model;

[0053] A picture acquisition module, which cuts out the second product image whose background needs to be replaced to obtain the product main body mask therein; Extract the edge information of the second product image;

[0054] The image generation module loads the background Lora model in the image-to-image generation algorithm, inputs the edge information, background prompt, and commodity main body mask into the image-to-image generation algorithm to generate a commodity replacement image; the background prompt corresponds to the background label in the set format.

[0055] In this embodiment, preferably, it further includes an optimized lighting and shadow module, which obtains the background-removed image and the background image in the commodity replacement image. The background-removed image is the image of the commodity main body part remaining after removing the background in the commodity replacement image; the background-removed image is enhanced through Gamma transformation, and the brightness of the commodity main body in the background-removed image is increased by a first set value to obtain an enhanced image; the background image is mapped to the HSV space, the V space in the HSV space is reduced by a second set value, and then mapped back to the RGB space to obtain a background-darkened image; the enhanced image and the background-darkened image are synthesized to obtain a first synthesized image, and the first latent information is extracted from the first synthesized image through VAE decoding; the background-removed image and the background image are synthesized to obtain a second synthesized image, and the first set information is extracted from the second synthesized image by using the ControlNet model; the first latent information and the first set information are sent to the diffusion model to generate a first lighting and shadow guidance image; the second latent information is extracted from the first lighting and shadow guidance image, and the second latent information and the first set information are sent to the diffusion model to generate a second lighting and shadow guidance image; the texture information of the first lighting and shadow guidance image is migrated to the second lighting and shadow guidance image through the high-contrast retention algorithm to obtain an intermediate image, and then the color information of the background image is migrated to the intermediate image by using the welsh algorithm to obtain an optimized commodity image.

[0056] In this embodiment, preferably, the training data acquisition module is specifically: acquiring the first commodity images of various commodities, the first commodity images being images at a set shooting angle, and the first commodity images including multiple images of the same commodity with different proportions in the image; scaling all the first commodity images so that the size of each first commodity image is 1024*1024 to obtain training data.

[0057] In this embodiment, preferably, the image acquisition module is specifically: performing matte extraction on the second commodity image that needs to replace the background to obtain the commodity main body mask therein; calling the canny algorithm to extract the edge information of the second commodity image, and the weight parameter weight of the canny algorithm is 0.9.

[0058] The image generation module is specifically: loading the background Lora model in the image-to-image generation algorithm, with its loading weight set to 0.75, inputting the edge information, background prompt, and commodity main body mask into the image-to-image generation algorithm to generate a commodity replacement image; the background prompt corresponds to the background label in the set format.

[0059] Since the device described in the second embodiment of the present invention is the device adopted for implementing the method of the first embodiment of the present invention, based on the method described in the first embodiment of the present invention, those skilled in the art can understand the specific structure and variations of the device, so it will not be elaborated herein. Any device adopted for the method of the first embodiment of the present invention falls within the scope of protection of the present invention.

[0060] Based on the same inventive concept, this application provides an embodiment of an electronic device corresponding to the first embodiment, as detailed in the third embodiment.

[0061] Embodiment Three

[0062] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, any implementation manner in the first embodiment can be realized.

[0063] Since the electronic device described in this embodiment is the device adopted for implementing the method in the first embodiment of this application, based on the method described in the first embodiment of this application, those skilled in the art can understand the specific implementation manners and various variations of the electronic device in this embodiment. Therefore, how the electronic device realizes the method in the embodiments of this application will not be introduced in detail herein. Any device adopted by those skilled in the art for implementing the method in the embodiments of this application falls within the scope of protection of this application.

[0064] Based on the same inventive concept, this application provides a storage medium corresponding to the first embodiment, as detailed in the fourth embodiment.

[0065] Embodiment Four

[0066] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, any implementation manner in the first embodiment can be realized.

[0067] The technical solutions provided in the embodiments of this application at least have the following technical effects or advantages:

[0068] This embodiment successfully solves the need of commodity sellers to change the background of commodity pictures. Sellers only need to simply take pictures by themselves, and then use this technical solution to generate the required composite pictures from the taken commodity pictures, greatly improving the efficiency of making commodity pictures, reducing the operation costs of sellers, and no longer requiring the use of professional camera teams and graphic designers for padding picture editing, etc.

[0069] The light and shadow optimization of the present invention can greatly improve the efficiency of commodity synthesis, and can effectively solve the problem of insufficient light and shadow that easily occurs in traditional light and shadow synthesis algorithms; making the generated commodity pictures have high authenticity, fusion effect, and lighting effect.

[0070] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0071] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0072] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0073] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0074] Although the specific embodiments of the present invention have been described above, those skilled in the art of this technology should understand that the specific embodiments we described are illustrative rather than used to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered by the scope protected by the claims of the present invention.

Claims

1. A method for changing the background of a product image based on Lora model training, characterized in that: The steps include: Step 1: Obtain first product images of various products, where the first product images are images taken at a set shooting angle, and the first product images include multiple images of the same product in different proportions in the image, to obtain training data; Step 2: Label each of the first product images according to the set format background label through the multimodal model to form a corresponding label file; Step 3: Set the number of training rounds, learning rate, and training image pixels, then input the training data and label file to perform model training to obtain the required background Lora model. Step 4: Cut out the second product image whose background needs to be replaced to obtain a product main body mask; extract edge information of the second product image; Step 5: Load the background Lora model into the image generation algorithm, input the edge information, background prompt words and product body mask into the image generation algorithm, and generate a product replacement image; the background prompt words correspond to the set format background label.

2. The method for changing the background of a product image based on Lora model training according to claim 1, characterized in that: The method further includes step 6, obtaining a background image and a background image in the product replacement image; enhancing the background image by Gamma transformation, increasing the brightness of the product body in the background image by a first set value, and obtaining an enhanced image; mapping the background image to the HSV space, reducing the V space in the HSV space by a second set value, and then mapping it back to the RGB space to obtain a background darkening image; synthesizing the enhanced image and the background darkening image to obtain a first synthesized image, and extracting first latent information from the first synthesized image by VAE decoding; The background-removed image and the background image are synthesized to obtain a second synthesized image, and the first setting information is extracted from the second synthesized image using the ControlNet model; the first latent information and the first setting information are sent to the diffusion model to generate a first light and shadow guidance map; the second latent information is extracted from the first light and shadow guidance map, and the second latent information and the first setting information are sent to the diffusion model to generate a second light and shadow guidance map; the texture information of the first light and shadow guidance map is transferred to the second light and shadow guidance map through a high-contrast retention algorithm to obtain an intermediate image, and then the Welsh algorithm is used to transfer the color information of the background image to the intermediate image to obtain a product optimization image.

3. The method for changing the background of a product image based on Lora model training according to claim 1, characterized in that: The step 1 specifically includes: obtaining first product images of various products, where the first product images are images of a set shooting angle, and the first product images include multiple images of the same product in different proportions in the images; scaling all the first product images so that the size of each first product image is 1024*1024, to obtain training data.

4. The method for changing the background of a product image based on Lora model training according to claim 1, characterized in that: The step 4 specifically includes: cutting out the second product image whose background needs to be replaced to obtain a product main body mask; calling the Canny algorithm to extract edge information of the second product image, wherein the weight parameter weight of the Canny algorithm is 0.9; The step 5 is specifically as follows: loading the background Lora model into the image generation algorithm, setting its loading weight to 0.75, inputting the edge information, background prompt words and product body mask into the image generation algorithm, and generating a product replacement image; the background prompt words correspond to the set format background label.

5. A background changing device for a product image based on Lora model training, characterized in that: include: A training data acquisition module is provided to acquire first product images of various products, wherein the first product images are images taken at a set shooting angle and include multiple images of the same product in different proportions in the images, thereby obtaining training data; Setting a labeling module to label each of the first product images according to a set format background label through a multimodal model to form a corresponding label file; In the training model module, set the number of training rounds, learning rate, and training image pixels, then input the training data and label file to perform model training to obtain the required background Lora model; The image acquisition module cuts out the second product image whose background needs to be replaced to obtain a product main body mask; and extracts edge information of the second product image; Generate an image module, load the background Lora model in the image generation algorithm, input the edge information, background prompt words and product body mask into the image generation algorithm, and generate a product replacement image; the background prompt words correspond to the set format background label.

6. The device for changing the background of a product image based on Lora model training according to claim 5, characterized in that: The method also includes optimizing a light and shadow module, obtaining a background image and a background image in a product replacement image; enhancing the background image by Gamma transformation, increasing the brightness of the product body in the background image by a first set value, and obtaining an enhanced image; mapping the background image to an HSV space, reducing the V space in the HSV space by a second set value, and then mapping it back to an RGB space to obtain a background darkening image; synthesizing the enhanced image and the background darkening image to obtain a first synthesized image, and extracting first latent information from the first synthesized image by VAE decoding; The background-removed image and the background image are synthesized to obtain a second synthesized image, and the first setting information is extracted from the second synthesized image using the ControlNet model; the first latent information and the first setting information are sent to the diffusion model to generate a first light and shadow guidance map; the second latent information is extracted from the first light and shadow guidance map, and the second latent information and the first setting information are sent to the diffusion model to generate a second light and shadow guidance map; the texture information of the first light and shadow guidance map is transferred to the second light and shadow guidance map through a high-contrast retention algorithm to obtain an intermediate image, and then the Welsh algorithm is used to transfer the color information of the background image to the intermediate image to obtain a product optimization image.

7. The device for changing the background of a product image based on Lora model training according to claim 5, characterized in that: The module for obtaining training data specifically comprises: obtaining first product images of various products, where the first product images are images with a set shooting angle, and the first product images include multiple images of the same product in different proportions in the images; scaling all the first product images so that the size of each first product image is 1024*1024, to obtain training data.

8. The device for changing the background of a product image based on Lora model training according to claim 5, characterized in that: The image acquisition module specifically includes: cutting out the second product image whose background needs to be replaced to obtain a product main body mask; calling the Canny algorithm to extract edge information of the second product image, wherein the weight parameter weight of the Canny algorithm is 0.9; The image generation module specifically includes: loading the background Lora model in the image generation algorithm, setting its loading weight to 0.75, inputting the edge information, background prompt words and product body mask into the image generation algorithm, and generating a product replacement image; the background prompt words correspond to the set format background label.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images