Method, device and equipment for generating commodity poster with enhanced light and shadow, and medium

Abstract

The invention provides a shadow-enhanced commodity poster generation method and device, equipment and a medium, and the method comprises the steps: carrying out the matting of a commodity image, and obtaining a commodity main body image; processing to obtain a first grey base map with a grey base as a background; processing the first grey base map through a Depth algorithm to obtain a first depth image; processing the first grey base graph through a Lineart algorithm to obtain a first line draft graph; the first depth image and the first line draft image are transmitted into Controlnet, a set Lora light and shadow model and set cue words are input into a diffusion model, the diffusion model is guided to draw through the Controlnet, and a first intermediate image is generated; inputting a set light and shadow reference image into an IC-Light model, extracting to obtain light and shadow features, inputting the light and shadow features and the first intermediate image into a diffusion model for drawing, and generating a result image; the light and shadow effect of the result image is remarkable, and user use is met.

Description

Technical Field

[0001] The present invention relates to the technical field of generating pictures, and particularly relates to a method, device, equipment and medium for generating a product poster with enhanced light and shadow. Background Art

[0002] In the fields of e-commerce and advertising, it is necessary to promote products by setting product posters. A product poster not only requires setting a corresponding background, but also needs to optimize its light and shadow effects to bring visual attraction and a sense of reality to buyers. However, the existing technology is to combine the product with a specific background manually through graphic design and optimize the light and shadow, which results in very low efficiency. When a large number of pictures need to be processed, a large number of graphic designers need to be recruited or outsourced.

[0003] In the prior art, there are methods of generating product posters by using StableDiffusion, which requires technical personnel to write specific prompts. The disadvantage of this method is that the generated pictures often have problems such as floating and very low light and shadow intensity. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method, device, equipment and medium for generating a product poster with enhanced light and shadow. Through this method, the product in the generated poster can be naturally combined with the background, and further light and shadow optimization can be carried out.

[0005] In the first aspect, the present invention provides a method for generating a product poster with enhanced light and shadow, including the following steps:

[0006] Step 1: Cut out the product image to obtain the main product image.

[0007] Step 2: Process the main product image to obtain a first gray-bottom image with a gray background.

[0008] Step 3: Process the first gray-bottom image through the Depth algorithm to obtain a first depth image; process the first gray-bottom image through the Lineart algorithm to obtain a first line drawing.

[0009] Step 4: Input the first depth image and the first line drawing into ControlNet, and input the set Lora light and shadow model and the set prompt into the diffusion model. Guide the diffusion model to draw through ControlNet to generate a first intermediate image.

[0010] Step 5: Input the set light and shadow reference image into the IC-Light model to extract the light and shadow features, and input the light and shadow features and the first intermediate image into the diffusion model for drawing to generate a result image.

[0011] In a second aspect, the present invention provides a device for generating a product poster with enhanced light and shadow, including:

[0012] A main body extraction module that performs matting on a product image to obtain a product main body image;

[0013] A gray background setting module that processes the product main body image to obtain a first gray background image with a gray background;

[0014] An acquisition guidance module that processes the first gray background image through the Depth algorithm to obtain a first depth image; the first gray background image is processed through the Lineart algorithm to obtain a first line drawing;

[0015] An intermediate image generation module that inputs the first depth image and the first line drawing into Controlnet, and inputs a set Lora light and shadow model and a set prompt into a diffusion model, and guides the diffusion model to draw through Controlnet to generate a first intermediate image;

[0016] A light and shadow enhancement module that inputs a set light and shadow reference image into the IC-Light model, extracts light and shadow features, and inputs the light and shadow features and the first intermediate image into the diffusion model for drawing to generate a result image.

[0017] In a third aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in the first aspect is implemented.

[0018] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect is implemented.

[0019] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0020] The method of the present invention can optimize the light and shadow effect of the generated poster, effectively solve the problem of insufficient light and shadow that easily occurs in traditional light and shadow synthesis algorithms. This method does not require manual participation and can greatly improve work efficiency.

[0021] The generation method of the present invention can also greatly improve the stability of generating product posters, making them free from problems such as suspension or mismatch between the product and the background, making the generated product posters closer to real life, and further increasing the usage rate of generating product posters by users.

[0022] The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the content of the specification. And in order to make the above and other objects, features and advantages of the present invention more obvious and understandable, the specific embodiments of the present invention are specifically described below. Brief Description of the Drawings

[0023] The present invention will be further described below with reference to the drawings in conjunction with embodiments.

[0024] Figure 1 It is a flowchart of the method in Embodiment 1 of the present invention;

[0025] Figure 2 It is a schematic structural diagram of the device in Embodiment 2 of the present invention. Detailed Embodiments

[0026] Embodiment 1

[0027] As Figure 1 shown, this embodiment provides a method for generating a product poster with enhanced light and shadow, including the following steps:

[0028] Step 1: Cut out the product image to obtain the main product image;

[0029] Step 2: Process the main product image to obtain a first gray-bottom image with a gray background;

[0030] Step 3: Process the first gray-bottom image through the Depth algorithm to obtain a first depth image; the first gray-bottom image is processed through the Lineart algorithm to obtain a first line drawing;

[0031] Step 4: Input the first depth image and the first line drawing into Controlnet, and input the set Lora light and shadow model and the set prompt words into the diffusion model. The diffusion model is guided by Controlnet to perform drawing to generate a first intermediate image;

[0032] Step 5: Input the set light and shadow reference image into the IC-Light model to extract the light and shadow features, and input the light and shadow features and the first intermediate image into the diffusion model for drawing to generate a result image.

[0033] In this embodiment, preferably, step 1 is specifically as follows: Determine the pixels of the product image. If the pixels of the product image are less than 2000×2000, directly perform a matting operation through the Visual Intelligence Open Platform to obtain the required product main image. If the pixels of the image are greater than or equal to 2000×2000, use the Imgproc.resize in OpenCV with the Imgproc.INTER_LANCZ0S4 algorithm to scale the image proportionally to obtain a scaled image. Perform a matting operation on the scaled image through the Visual Intelligence Open Platform to obtain a first mask image. Use the Imgproc.resize in OpenCV with the Imgproc.INTER_LANCZ0S4 algorithm to scale the first mask image proportionally back to the original size to obtain a second mask image. Read the alpha channel of each pixel point in the user-uploaded image to obtain a first matrix, read the alpha channel of each pixel point in the second mask image to obtain a second matrix, and call Core.min to merge the first matrix and the second matrix to obtain a third matrix. Replace the alpha channel in the user-uploaded image with the third matrix to obtain the required product main image.

[0034] Due to the matting size limit of the Alibaba Cloud Visual Intelligence Open Platform, when the longest side exceeds 2000 pixels, it is necessary to perform proportional scaling. Call Imgproc.resize to use the Imgproc.INTER_LANCZOS4 algorithm for image proportional scaling;

[0035] Call Imgproc.resize to use the Imgproc.INTER_LANCZOS4 algorithm for image proportional scaling. This algorithm can reduce the influence of artifacts while maintaining the edge sharpness.

[0036] Obtain the matting result according to the black and white image + original image:

[0037] a. Take out the alpha channel of the original image. Call Core.min, which will take the minimum value of the transparency channels of the mask and the original image, ensuring that the original information of the transparency channel is retained. Through this step of processing, the lines of the obtained matting can be made more perfect;

[0038] b. Remove the alpha channel of the original image and add the black and white image as the new alpha channel for layer merging;

[0039] c. Set the color value of the transparent area to black. This step is to reduce the size of the matting result image and save storage costs.

[0040] The core code is as follows:

[0041] / / Create a fully transparent matrix with the same size as the original image for comparison;

[0042] Mat compareAlpha = new Mat(outImg.size(), CvType.CV_8UC1, Scalar.all(0.0));

[0043] / / Used to save the comparison result;

[0044] Mat compareResult = new Mat();

[0045] / / Compare the alpha channel of the original image with alpha. The positions with a value of 0 in the obtained mask are the transparent positions in the original image;

[0046] Core.compare(outPlanes.get(3), compareAlpha, compareResult, Core.CMP_EQ);

[0047] / / Create a completely black matrix with the same size as the original image;

[0048] Mat black = new Mat(outImg.size(), outImg.type(), Scalar.all(0));

[0049] / / Copy the black color in black to the corresponding positions in outImg according to the mask;

[0050] Core.bitwise_and(black, outImg, outImg, compareResult);

[0051] In this embodiment, preferably, step 4 is specifically: input the first depth image and the first line drawing into Controlnet, and set the reference weights of the lineart module and the depth module in Controlnet to 0.5 and 0.6 respectively, and input the set Lora lighting model and the set prompt words into the diffusion model. The weight of the Lora lighting model is set to 0.7, and draw through the diffusion model, set the redrawing amplitude to 0.85, and generate the first intermediate image;

[0052] The training method of the Lora lighting model is: obtain various commercial poster images with set lighting, where the commercial poster images include multiple images of the same product in different proportions in the picture, to obtain training data; label each of the commercial poster images through a multi-modal model according to the set format lighting labels to form corresponding label files; set the number of training rounds, learning rate, and the number of pixels of the training images, and then input the training data and the label files for Lora model training to obtain the required Lora lighting model.

[0053] In this embodiment, preferably, step 5 is specifically as follows: The generated first intermediate image is processed by the depth algorithm to obtain a second depth image; the commodity gray background image is processed by the lineart algorithm to obtain a second line drawing; the second depth image and the second line drawing are respectively sent to Controlnet, the redrawing amplitude is set to 1.0, and the reference weights of the lineart module and the depth module in Controlnet are set to 0.45 and 0.55 respectively. Then, drawing is performed through the diffusion model to generate a second intermediate image; the set light and shadow reference image is input into the IC-Light model, the light and shadow features are extracted, and the light and shadow features and the second intermediate image are input into the diffusion model for drawing to generate the result image.

[0054] In this embodiment, preferably, step 2 is specifically as follows: The area of the commodity main body in the commodity main body image is combined with a grayscale image with a grayscale value of 127 to obtain a commodity gray background image; the commodity gray background image is processed to obtain a first gray background image with a resolution of 640*640.

[0055] Based on the same inventive concept, the present application also provides a device corresponding to the method in Embodiment 1, as detailed in Embodiment 2.

[0056] Embodiment 2

[0057] As Figure 2 shown, in this embodiment, a device for generating an enhanced light and shadow commodity poster is provided, including:

[0058] A main body extraction module that performs matte extraction on the commodity image to obtain a commodity main body image;

[0059] A gray background setting module that processes the commodity main body image to obtain a first gray background image with a gray background;

[0060] A guidance acquisition module that processes the first gray background image through the Depth algorithm to obtain a first depth image; the first gray background image is processed through the Lineart algorithm to obtain a first line drawing;

[0061] An intermediate image generation module that inputs the first depth image and the first line drawing into Controlnet, and inputs the set Lora light and shadow model and the set prompt words into the diffusion model, and guides the diffusion model to perform drawing through Controlnet to generate a first intermediate image;

[0062] A light and shadow enhancement module that inputs the set light and shadow reference image into the IC-Light model, extracts the light and shadow features, and inputs the light and shadow features and the first intermediate image into the diffusion model for drawing to generate the result image.

[0063] In this embodiment, preferably, the main body extraction module is specifically as follows: judge the pixels of the product image. If the pixels of the product image are less than 2000×2000, directly perform the matting operation through the Visual Intelligence Open Platform to obtain the required product main body image; if the pixels of the image are greater than or equal to 2000×2000, use the Imgproc.resize in OpenCV with the Imgproc.INTER_LANCZ0S4 algorithm to scale the image proportionally to obtain a scaled image; perform the matting operation on the scaled image through the Visual Intelligence Open Platform to obtain a first mask image; use the Imgproc.resize in OpenCV with the Imgproc.INTER_LANCZ0S4 algorithm to restore the first mask image proportionally to the original size to obtain a second mask image; read the alpha channel of each pixel point in the user-uploaded image to obtain a first matrix, read the alpha channel of each pixel point in the second mask image to obtain a second matrix, and call Core.min to merge the first matrix and the second matrix to obtain a third matrix; replace the alpha channel in the user-uploaded image with the third matrix to obtain the required product main body image.

[0064] In this embodiment, preferably, the intermediate image generation module is specifically as follows: input the first depth image and the first line drawing into Controlnet, and set the reference weights of the lineart module and the depth module in Controlnet to 0.5 and 0.6 respectively, and input the set Lora light and shadow model and the set prompt words into the diffusion model. The weight of the Lora light and shadow model is set to 0.7, draw through the diffusion model, and set the redrawing amplitude to 0.85 to generate a first intermediate image;

[0065] The training method of the Lora light and shadow model is as follows: obtain various product poster images with set light and shadow, where the product poster images include multiple images of the same product with different proportions in the picture to obtain training data; label each of the product poster images with a light and shadow label in a set format through a multimodal model to form a corresponding label file; set the number of training epochs, learning rate, and training image pixels, and then input the training data and the label file for Lora model training to obtain the required Lora light and shadow model.

[0066] In this embodiment, preferably, the light and shadow enhancement module is specifically as follows: The generated first intermediate image is processed by the depth algorithm to obtain a second depth image; the commodity gray background image is processed by the lineart algorithm to obtain a second line drawing; the second depth image and the second line drawing are respectively sent to Controlnet, the redrawing amplitude is set to 1.0, and the reference weights of the lineart module and the depth module in Controlnet are set to 0.45 and 0.55 respectively, and then drawing is performed through the diffusion model to generate a second intermediate image; the set light and shadow reference image is input into the IC-Light model to extract light and shadow features, and the light and shadow features and the second intermediate image are input into the diffusion model for drawing to generate a result image.

[0067] In this embodiment, preferably, the gray background setting module is specifically as follows: The area of the commodity main body map except for the commodity main body part is synthesized with a gray scale image with a gray scale value of 127 to obtain a commodity gray background image; the commodity gray background image is processed to obtain a first gray background image with a resolution of 640*640.

[0068] Since the device introduced in the second embodiment of the present invention is the device adopted for implementing the method of the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device, so it will not be elaborated here. Any device adopted for the method of the first embodiment of the present invention belongs to the scope protected by the present invention.

[0069] Based on the same inventive concept, this application provides an electronic device embodiment corresponding to the first embodiment, as detailed in the third embodiment.

[0070] Embodiment Three

[0071] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, any implementation manner in the first embodiment can be realized.

[0072] Since the electronic device introduced in this embodiment is the device adopted for implementing the method in the first embodiment of this application, based on the method introduced in the first embodiment of this application, those skilled in the art can understand the specific implementation manner of the electronic device in this embodiment and its various variations. Therefore, how this electronic device implements the method in the embodiments of this application will not be introduced in detail here. Any device adopted by those skilled in the art for implementing the method in the embodiments of this application belongs to the scope protected by this application.

[0073] Based on the same inventive concept, this application provides a storage medium corresponding to the first embodiment, as detailed in the fourth embodiment.

[0074] Embodiment Four

[0075] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, any implementation manner in the first embodiment can be realized.

[0076] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0078] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device realizes the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0080] Although the specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments we described are illustrative only and not used to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should all be covered by the scope protected by the claims of the present invention.

Claims

1. A method for generating a product poster with enhanced light and shadow, characterized in that: The steps include: Step 1: Cut out the product image to obtain the main image of the product; Step 2: Process the main image of the product to obtain a first gray background image with a gray background; Step 3: Process the first gray background image by using a Depth algorithm to obtain a first depth image; process the first gray background image by using a Lineart algorithm to obtain a first line drawing; Step 4: The first depth image and the first line drawing are input into Controlnet, and the set Lora light and shadow model and the set prompt words are input into the diffusion model, and the diffusion model is guided by Controlnet to draw, so as to generate a first intermediate image; Step 5: Input the set light and shadow reference image into the IC-Light model to extract the light and shadow features, input the light and shadow features and the first intermediate image into the diffusion model for drawing, and generate a result image.

2. The method for generating a product poster with enhanced light and shadow according to claim 1, characterized in that: The step 1 is specifically as follows: judging the pixels of the product image, if the pixels of the product image are less than 2000×2000, directly performing a cutout operation through the visual intelligence open platform to obtain the required product main body image; if the pixels of the image are greater than or equal to 2000×2000, using Imgproc.resize in OpenCV and using the Imgproc.INTER_LANCZ0S4 algorithm to scale the image in proportion to obtain a scaled image; performing a cutout operation on the scaled image through the visual intelligence open platform to obtain a first mask image; using the Imgproc.resize in OpenCV and using the Imgproc.INTER_LANCZ0S4 algorithm to proportionally restore the image to a second mask image of the original size; reading out the transparent channel of each pixel in the image uploaded by the user to obtain a first matrix, reading out the transparent channel of each pixel in the second mask image to obtain a second matrix, calling Core.min to merge the first matrix with the second matrix to obtain a third matrix; The transparent channel in the picture uploaded by the user is replaced by the third matrix to obtain the required main picture of the product.

3. The method for generating a product poster with enhanced light and shadow according to claim 1, characterized in that: The step 4 is specifically as follows: the first depth image and the first line drawing are passed into Controlnet, and the reference weights of the lineart module and the depth module in Controlnet are set to 0.5 and 0.6 respectively, and the Lora light and shadow model and the prompt words are set to be input into the diffusion model, the weight of the Lora light and shadow model is set to 0.7, and the diffusion model is used for drawing, and the redrawing amplitude is set to 0.85 to generate the first intermediate image; The training method of the Lora light and shadow model is as follows: obtain product poster images with various set light and shadow, wherein the product poster images include multiple images of the same product in different proportions in the images, to obtain training data; label each of the product poster images according to the set format of light and shadow labels through a multimodal model to form a corresponding label file; set the number of training rounds, learning rate, and training image pixels, and then input the training data and label file to perform Lora model training to obtain the required Lora light and shadow model.

4. The method for generating a product poster with enhanced light and shadow according to claim 1, characterized in that: The step 5 is specifically as follows: the generated first intermediate image is processed by the depth algorithm to obtain a second depth image; the gray background image of the product is processed by the lineart algorithm to obtain a second line drawing image; the second depth image and the second line drawing image are respectively sent to Controlnet, the redrawing amplitude is set to 1.0, and the reference weights of the lineart module and the depth module in Controlnet are set to 0.45 and 0.55 respectively, and then drawn by the diffusion model to generate a second intermediate image; the set light and shadow reference image is input into the IC-Light model to extract the light and shadow features, and the light and shadow features and the second intermediate image are input into the diffusion model for drawing to generate a result image.

5. A device for generating a product poster with enhanced light and shadow, characterized in that: include: The main body extraction module cuts out the product image to obtain the main body image of the product; A gray background module is set to process the main image of the product to obtain a first gray background image with a gray background; The acquisition guidance module processes the first gray background image by using a Depth algorithm to obtain a first depth image; the first gray background image is processed by using a Lineart algorithm to obtain a first line drawing; Generate an intermediate image module, input the first depth image and the first line drawing into Controlnet, and input the set Lora light and shadow model and the set prompt words into the diffusion model, guide the diffusion model to draw through Controlnet, and generate a first intermediate image; The light and shadow enhancement module inputs the set light and shadow reference image into the IC-Light model, extracts the light and shadow features, inputs the light and shadow features and the first intermediate image into the diffusion model for drawing, and generates a result image.

6. The device for generating a product poster with enhanced light and shadow according to claim 5, characterized in that: The extraction main body module is specifically as follows: the pixels of the product image are judged. If the pixels of the product image are less than 2000×2000, the image is directly cut out through the visual intelligence open platform to obtain the required product main body image; if the pixels of the image are greater than or equal to 2000×2000, the image is scaled in proportion using the Imgproc.resize in OpenCV and the Imgproc.INTER_LANCZ0S4 algorithm to obtain a scaled image; the scaled image is cut out through the visual intelligence open platform to obtain a first mask image; the first mask image is restored to a second mask image of the original size through the Imgproc.resize in OpenCV and the Imgproc.INTER_LANCZ0S4 algorithm; the transparent channel of each pixel in the image uploaded by the user is read out to obtain a first matrix, the transparent channel of each pixel in the second mask image is read out to obtain a second matrix, and Core.min is called to merge the first matrix with the second matrix to obtain a third matrix; The transparent channel in the picture uploaded by the user is replaced by the third matrix to obtain the required main picture of the product.

7. The device for generating a product poster with enhanced light and shadow according to claim 5, characterized in that: The module for generating the intermediate image is specifically as follows: the first depth image and the first line drawing are input into Controlnet, and the reference weights of the lineart module and the depth module in Controlnet are set to 0.5 and 0.6 respectively, and the Lora light and shadow model and the prompt words are input into the diffusion model, the weight of the Lora light and shadow model is set to 0.7, and the diffusion model is used for drawing, and the redrawing amplitude is set to 0.85 to generate the first intermediate image; The training method of the Lora light and shadow model is as follows: obtain product poster images with various set light and shadow, wherein the product poster images include multiple images of the same product in different proportions in the images, to obtain training data; label each of the product poster images according to the set format of light and shadow labels through a multimodal model to form a corresponding label file; set the number of training rounds, learning rate, and training image pixels, and then input the training data and label file to perform Lora model training to obtain the required Lora light and shadow model.

8. The device for generating a product poster with enhanced light and shadow according to claim 5, characterized in that: The light and shadow enhancement module is specifically as follows: the generated first intermediate image is processed by the depth algorithm to obtain a second depth image; the gray background image of the product is processed by the lineart algorithm to obtain a second line drawing image; the second depth image and the second line drawing image are sent to Controlnet respectively, the redrawing amplitude is set to 1.0, and the reference weights of the lineart module and the depth module in Controlnet are set to 0.45 and 0.55 respectively, and then drawn by the diffusion model to generate a second intermediate image; the set light and shadow reference image is input into the IC-Light model to extract the light and shadow features, and the light and shadow features and the second intermediate image are input into the diffusion model for drawing to generate a result image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images