Method, device and equipment for replacing commodity with commercial picture, and medium

Abstract

The invention provides a method, a device, equipment and a medium for replacing a commodity with a commercial picture, and the method comprises the steps: carrying out the matting operation of a first commodity picture uploaded by a user, obtaining a first commodity main body picture, and extracting a first commodity mask; processing to obtain an empty scene graph; carrying out matting operation on a second commodity image uploaded by the user to obtain a second commodity main body image; placing the second commodity main body image at a set position of the empty scene image, performing synthesis to obtain a composite image, and extracting a first commodity main body mask from the composite image; performing negation on the first commodity main body mask to obtain a second commodity main body mask; extracting a total image contour line graph and a depth information graph of the composite image; performing cue word reverse deduction on the composite graph to obtain language description of the composite graph; inputting the second commodity main body mask, the full-graph contour line graph, the depth information graph and the language descriptor into a diffusion model to generate a required replacement graph; the efficiency of merchants is greatly improved, and the cost of the merchants is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field, and particularly to a method, apparatus, device, and medium for replacing a product in a commercial photo Background Art

[0002] In the e-commerce industry, after a seller upgrades a product, it is necessary to replace the product in the previous product photo with the upgraded product to quickly achieve batch production of commercial photos. Commercial photos are product pictures, which can reduce a large amount of cumbersome post-processing and improve efficiency. However, the traditional method requires a graphic artist to perform image matting, then manually repair it, then photograph the upgraded product, then perform image matting on it, and finally replace it into the original product photo. This image editing method is inefficient and is subject to the professional skills of the graphic artist, making it difficult to achieve natural and seamless product replacement. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method, apparatus, device, and medium for replacing a product in a commercial photo, which realizes product replacement based on the image elimination, repair, and redrawing functions of deep learning. This method can quickly and accurately replace the product in the product photo, and the replaced product seamlessly blends with the original background.

[0004] In a first aspect, the present invention provides a method for replacing a product in a commercial photo, including the following steps:

[0005] Step 1: Perform image matting on the first product photo uploaded by the user to extract the product main body, obtaining a first product main body photo, and extract a first product mask from the first product main body photo;

[0006] Step 2: Perform dilation on the first product mask n times with a kernel size of m*m using the dilation algorithm to obtain a second product mask; perform elimination and repair on the area corresponding to the second product mask in the first product photo using the elimination algorithm to obtain an empty scene photo;

[0007] Step 3: Perform image matting on the second product photo uploaded by the user to extract the product main body, obtaining a second product main body photo;

[0008] Step 4: Place the second product main body photo at a set position in the empty scene photo for synthesis to obtain a synthesized photo, and extract a first product main body mask from the synthesized photo; invert the first product main body mask to obtain a second product main body mask;

[0009] Step 5: Extract the full image contour line drawing and depth information map of the synthesized photo; and perform reverse prompting on the synthesized photo to obtain the language description of the synthesized photo;

[0010] Step 6: Input the second commodity main body mask, the full - map contour line drawing, the depth information map, and the language description words into the diffusion model to generate the required replacement map.

[0011] In a second aspect, the present invention provides an apparatus for replacing a commodity in a commercial photo, including:

[0012] A first mask extraction module that performs a matting operation on the first commodity photo uploaded by the user, extracts the commodity main body therefrom to obtain a first commodity main body photo, and extracts a first commodity mask from the first commodity main body photo;

[0013] An empty - scene acquisition module that performs dilation on the first commodity mask n times with a kernel size of m*m using the dilation algorithm to obtain a second commodity mask; performs elimination and repair on the area corresponding to the second commodity mask in the first commodity photo using the elimination algorithm to obtain an empty - scene photo;

[0014] A matting module that performs a matting operation on the second commodity photo uploaded by the user, extracts the commodity main body therefrom to obtain a second commodity main body photo;

[0015] A second mask extraction module that places the second commodity main body photo at a set position in the empty - scene photo for synthesis to obtain a synthesized photo, extracts a first commodity main body mask from the synthesized photo; takes the inverse of the first commodity main body mask to obtain a second commodity main body mask;

[0016] An information acquisition and generation module that extracts the full - map contour line drawing and the depth information map of the synthesized photo; and performs reverse inference of the prompt words on the synthesized photo to obtain the language description of the synthesized photo;

[0017] A generated - picture module that inputs the second commodity main body mask, the full - map contour line drawing, the depth information map, and the language description words into the diffusion model to generate the required replacement map.

[0018] In a third aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in the first aspect is implemented.

[0019] In a fourth aspect, the present invention provides a computer - readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect is implemented.

[0020] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0021] The present invention extracts the main body in the commercial photo, then eliminates the commodity main body in the commercial photo and repairs it. After that, a new commodity is placed in the repaired picture for redrawing, so that the new commodity is more naturally integrated into the repaired scene, realizing commodity replacement. This method makes the background of the commodity complete without defects. Through redrawing, the new commodity is seamlessly integrated, making the generated picture not obtrusive and unnatural, and greatly improving the efficiency of commodity replacement.

[0022] The above description is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable, the specific embodiments of the present invention are specifically listed below. Brief Description of the Drawings

[0023] The present invention will be further described below with reference to the accompanying drawings in conjunction with embodiments.

[0024] Figure 1 It is a flowchart of the method in Embodiment 1 of the present invention;

[0025] Figure 2 It is a structural schematic diagram of the device in Embodiment 2 of the present invention. Detailed Embodiments

[0026] By providing a method, device, equipment and medium for replacing commodities in commercial photos in the embodiments of the present application, the commodities in the commodity photos can be replaced quickly and accurately, and the replaced commodities are seamlessly integrated with the original background.

[0027] Embodiment 1

[0028] As Figure 1 shown, this embodiment provides a method for replacing commodities in commercial photos, including the following steps:

[0029] Step 1: Perform a matte extraction operation on the first commodity picture uploaded by the user to extract the commodity main body therein, obtaining a first commodity main body picture, and extracting a first commodity matte from the first commodity main body picture;

[0030] Step 2: Use the dilation algorithm to perform dilation on the first commodity matte n times with a kernel size of m*m, where both n and m are positive integers, n can be 3 times, and m can be 5; obtaining a second commodity matte, and expanding the main body area matte range in this way; use the elimination algorithm to perform elimination on the second commodity matte, that is, perform elimination on the matte area, obtaining a third commodity matte; repair the eliminated area in the third commodity matte to obtain an empty scene picture, and the eliminated area is seamlessly connected with the surrounding image content;

[0031] Step 3: Perform matte extraction on the second product image uploaded by the user to extract the product main body therein, obtaining the second product main body image;

[0032] Step 4: Place the second product main body image at the set position of the empty scene image for synthesis, obtaining a composite image, and extract the first product main body mask from the composite image; Invert the first product main body mask to obtain the second product main body mask, that is, the masked area becomes the background. The inversion means subtracting the value of each pixel point in the first product main body mask from 255;

[0033] Step 5: Extract the full image contour line drawing and the depth information map of the composite image; and perform prompt reverse inference on the composite image to obtain the language description of the composite image;

[0034] Step 6: Input the second product main body mask, the full image contour line drawing, the depth information map, and the language description words into a diffusion model to generate the required replacement image. This diffusion model can be stable diffusion, and redrawing is performed through this diffusion model. In order to prevent changes in the edges of the product main body, the full image contour line drawing is used to control the product edges during the redrawing process; In order to allow the new product to readjust the depth of field and light and shadow information in the repaired background image, the depth information map is used for control; And combined with the language description words, finally, it can guide the diffusion model to perform accurate redrawing, making the fusion of the background and the main body more natural.

[0035] In this embodiment, preferably, step 2 is specifically: Perform dilation on the first product mask n times with a kernel size of m*m using the dilation algorithm of OpenCV to obtain the second product mask; Use the LaMa algorithm to perform elimination and repair on the area corresponding to the second product mask in the first product image to obtain the empty scene image.

[0036] In this embodiment, preferably, step 5 is specifically: Use the canny edge extraction algorithm of OpenCV to extract the full image contour line drawing of the composite image; Use the open-source depth estimation algorithm MiDas to extract the depth information map of the composite image; Use the open-source multimodal model BLIP to perform prompt reverse inference on the composite image to obtain the language description of the composite image.

[0037] In this embodiment, preferably, the matting operation is specifically as follows: judge the pixels of the image uploaded by the user. If the pixels of the image are less than 2000×2000, directly perform the matting operation through the Visual Intelligence Open Platform to obtain the required main image of the picture; if the pixels of the image are greater than or equal to 2000×2000, use Imgproc.resize in OpenCV with the Imgproc.INTER_LANCZ0S4 algorithm to scale the image proportionally to obtain a scaled image; perform the matting operation on the scaled image through the Visual Intelligence Open Platform to obtain the first mask image; use Imgproc.resize in OpenCV with the Imgproc.INTER_LANCZ0S4 algorithm to proportionally restore the first mask image to the second mask image of the original size; read the alpha channel of each pixel point in the image uploaded by the user to obtain the first matrix, read the alpha channel of each pixel point in the second mask image to obtain the second matrix, and call Core.min to merge the first matrix and the second matrix to obtain the third matrix; replace the alpha channel in the image uploaded by the user with the third matrix to obtain the required main image of the picture.

[0038] Due to the matting size limit of the Alibaba Cloud Visual Intelligence Open Platform, when the longest side exceeds 2000 pixels, proportional scaling is required. Call Imgproc.resize to use the Imgproc.INTER_LANCZOS4 algorithm for image proportional scaling;

[0039] Call Imgproc.resize to use the Imgproc.INTER_LANCZOS4 algorithm for image proportional scaling. This algorithm can reduce the influence of artifacts while maintaining the edge sharpness.

[0040] Obtain the matting result according to the black and white image + original image:

[0041] a. Take out the alpha channel of the original image. Call Core.min, which will take the minimum value of the transparency channels of the mask and the original image, ensuring that the original information of the transparency channel is retained. Through this step of processing, the lines of the obtained matting can be made more perfect;

[0042] b. Remove the alpha channel of the original image and add the black and white image as the new alpha channel for layer merging;

[0043] c. Set the color value of the transparent area to black. This step is to reduce the size of the matting result image and save storage costs.

[0044] The core code is as follows:

[0045] / / Create a fully transparent matrix with the same size as the original image for comparison;

[0046] Mat compareAlpha = new Mat(outImg.size(), CvType.CV_8UC1, Scalar.all(0.0));

[0047] / / Used to save the comparison result;

[0048] Mat compareResult = new Mat();

[0049] / / Compare the alpha channel of the original image with alpha. The positions with a value of 0 in the obtained mask are the transparent positions in the original image;

[0050] Core.compare(outPlanes.get(3), compareAlpha, compareResult, Core.CMP_EQ);

[0051] / / Create a completely black matrix with the same size as the original image;

[0052] Mat black = new Mat(outImg.size(), outImg.type(), Scalar.all(0));

[0053] / / Copy the black color in black to the corresponding positions in outImg according to the mask;

[0054] Core.bitwise_and(black, outImg, outImg, compareResult);

[0055] Based on the same inventive concept, this application also provides a device corresponding to the method in Embodiment 1. For details, see Embodiment 2.

[0056] Embodiment 2

[0057] As Figure 2 shown, in this embodiment, a device for replacing a product in a commercial photo is provided, including:

[0058] A first mask extraction module that performs a matte extraction operation on the first product image uploaded by the user, extracts the product main body therefrom to obtain a first product main body image, and extracts a first product mask from the first product main body image;

[0059] An empty scene acquisition module that performs dilation on the first product mask n times with a kernel size of m*m using the dilation algorithm to obtain a second product mask; performs elimination and repair on the area corresponding to the second product mask in the first product image using the elimination algorithm to obtain an empty scene image;

[0060] The matte extraction module performs matte extraction on the second product image uploaded by the user, extracts the product main body therein, and obtains the second product main body image;

[0061] The second matte extraction module places the second product main body image at the set position of the empty scene image, performs synthesis to obtain a synthesized image, and extracts the first product main body matte from the synthesized image; the first product main body matte is inverted to obtain the second product main body matte;

[0062] The generation information acquisition module extracts the full-image contour line drawing and the depth information map of the synthesized image; and performs reverse prompting on the synthesized image to obtain the language description of the synthesized image;

[0063] The image generation module inputs the second product main body matte, the full-image contour line drawing, the depth information map, and the language description words into a diffusion model to generate the required replacement image.

[0064] In this embodiment, preferably, the empty scene acquisition module is specifically: performing dilation on the first product matte n times with a kernel size of m*m using the dilation algorithm of OpenCV to obtain the second product matte; using the LaMa algorithm to eliminate and repair the area corresponding to the second product matte in the first product image to obtain the empty scene image.

[0065] In this embodiment, preferably, the generation information acquisition module is specifically: extracting the full-image contour line drawing of the synthesized image using the canny edge extraction algorithm of OpenCV; extracting the depth information map of the synthesized image using the open-source depth estimation algorithm MiDas; performing reverse prompting on the synthesized image using the open-source multimodal model BLIP to obtain the language description of the synthesized image.

[0066] In this embodiment, preferably, the matting operation is specifically as follows: judge the pixels of the picture uploaded by the user. If the pixels of the picture are less than 2000×2000, directly perform the matting operation through the Visual Intelligence Open Platform to obtain the required main picture of the picture; if the pixels of the picture are greater than or equal to 2000×2000, use Imgproc.resize in OpenCV with the Imgproc.INTER_LANCZ0S4 algorithm to scale the picture proportionally to obtain a scaled picture; perform the matting operation on the scaled picture through the Visual Intelligence Open Platform to obtain a first mask picture; use Imgproc.resize in OpenCV with the Imgproc.INTER_LANCZ0S4 algorithm to restore the first mask picture proportionally to the original size to obtain a second mask picture of the original size; read the alpha channel of each pixel point in the picture uploaded by the user to obtain a first matrix, read the alpha channel of each pixel point in the second mask picture to obtain a second matrix, and call Core.min to merge the first matrix and the second matrix to obtain a third matrix; replace the alpha channel in the picture uploaded by the user with the third matrix to obtain the required main picture of the picture.

[0067] Since the device introduced in the second embodiment of the present invention is the device adopted to implement the method of the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, those skilled in the art can understand the specific structure and variations of the device, so it will not be elaborated here. Any device adopted by the method of the first embodiment of the present invention belongs to the scope protected by the present invention.

[0068] Based on the same inventive concept, this application provides an electronic device embodiment corresponding to the first embodiment, as detailed in the third embodiment.

[0069] Embodiment Three

[0070] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, any implementation manner in the first embodiment can be realized.

[0071] Since the electronic device introduced in this embodiment is the device adopted to implement the method in the first embodiment of this application, based on the method introduced in the first embodiment of this application, those skilled in the art can understand the specific implementation manner of the electronic device in this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiments of this application will not be introduced in detail here. As long as the device adopted by those skilled in the art to implement the method in the embodiments of this application belongs to the scope protected by this application.

[0072] Based on the same inventive concept, this application provides a storage medium corresponding to the first embodiment, as detailed in the fourth embodiment.

[0073] Example 4

[0074] This embodiment provides a computer-readable storage medium with a computer program stored thereon. When the computer program is executed by a processor, any implementation manner in Embodiment 1 can be realized.

[0075] The technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0076] In this embodiment, through image processing technology, the original commodity main body is eliminated and seamlessly repaired to maintain the integrity and beauty of the background. Subsequently, the new commodity is integrated into the repaired background, and through meticulous redrawing work, it is ensured that the new commodity naturally blends with the surrounding environment. This implementation method not only improves the efficiency of commodity replacement but also ensures the seamless connection and natural transition of the final image, avoiding any abrupt or unnatural effects, thereby greatly enhancing the professionalism and attractiveness of commodity display.

[0077] Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram can be realized by computer program instructions, and the combination of processes and / or blocks in the flowchart and / or block diagram can also be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0079] These computer program instructions can also be stored in a computer-readable memory capable of guiding a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device realizes the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0080] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are executed on the computer or other programmable apparatus to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable apparatus provide steps for realizing the functions specified in one process or a plurality of processes and / or blocks Figure 1 one process or a plurality of processes and / or blocks Figure 1 or steps for realizing the functions specified in a plurality of blocks.

[0081] Although the specific embodiments of the present invention have been described above, those skilled in the art of this technology should understand that the specific embodiments we described are illustrative only and not used to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A method for replacing a product with a commercial photo, characterized in that: The steps include: Step 1: Perform a cutout operation on the first product image uploaded by the user to cut out the product body therein to obtain a first product body image, and extract a first product mask from the first product body image; Step 2: Use an expansion algorithm to expand the first product mask n times with a kernel size of m*m to obtain a second product mask; eliminate and repair the area in the first product image corresponding to the second product mask to obtain an empty scene image; Step 3: Cut out the second product image uploaded by the user, cut out the main body of the product, and obtain the main body image of the second product; Step 4: placing the second product main body image at a set position of the empty scene image, synthesizing it to obtain a synthesized image, and extracting the first product main body mask from the synthesized image; Invert the first product main body mask to obtain the second product main body mask; Step 5: extracting the full-image contour line map and the depth information map of the composite image; and using the composite image as a prompt word to reversely infer the language description of the composite image; Step 6: Input the second product main body mask, the full image contour line map, the depth information map and the language description words into the diffusion model to generate the required replacement map.

2. A method for replacing goods with commercial photos according to claim 1, characterized in that: The step 2 specifically includes: using the dilation algorithm of OpenCV to dilate the first product mask n times with a kernel size of m*m to obtain the second product mask; using the LaMa algorithm to eliminate and repair the area in the first product image corresponding to the second product mask to obtain an empty scene image.

3. A method for replacing goods with commercial photos according to claim 1, characterized in that: The step 5 specifically comprises: using the canny edge extraction algorithm of OpenCV to extract the contour line map of the whole image of the composite image; using the open source depth estimation algorithm MiDas to extract the depth information map of the composite image; The open source multimodal model BLIP is used to reverse the prompt words of the synthetic image to obtain a language description of the synthetic image.

4. A method for replacing goods with commercial photos according to claim 1, characterized in that: The cutout operation is specifically as follows: the pixels of the picture uploaded by the user are judged. If the pixels of the picture are less than 2000×2000, the cutout operation is directly performed through the visual intelligence open platform to obtain the required main picture of the picture; if the pixels of the picture are greater than or equal to 2000×2000, the picture is scaled in proportion using the Imgproc.resize in OpenCV and the Imgproc.INTER_LANCZ0S4 algorithm to obtain a scaled picture; the scaled picture is cutout through the visual intelligence open platform to obtain a first mask picture; the first mask picture is proportionally restored to a second mask picture of the original size through the Imgproc.resize in OpenCV and the Imgproc.INTER_LANCZ0S4 algorithm; the transparent channel of each pixel in the picture uploaded by the user is read out to obtain a first matrix, the transparent channel of each pixel in the second mask picture is read out to obtain a second matrix, and Core.min is called to merge the first matrix with the second matrix to obtain a third matrix; The transparent channel in the picture uploaded by the user is replaced by the third matrix to obtain the required main picture of the picture.

5. A device for replacing a product with a commercial photo, characterized in that: include: A first mask extraction module performs a cutout operation on the first product image uploaded by the user, cuts out the product body therein, obtains a first product body image, and extracts a first product mask from the first product body image; The empty scene acquisition module uses an expansion algorithm to expand the first product mask n times with a kernel size of m*m to obtain a second product mask; and uses an elimination algorithm to eliminate and repair the area in the first product image corresponding to the second product mask to obtain an empty scene image; A cutout module cuts out the second product image uploaded by the user, cuts out the main body of the product, and obtains the main body image of the second product; A second mask extraction module is configured to place the second product main body image at a set position of the empty scene image, synthesize the image, obtain a synthesized image, and extract the first product main body mask from the synthesized image; Invert the first product main body mask to obtain the second product main body mask; Obtaining a generation information module, extracting a full-image contour line map and a depth information map of the composite image; and inferring the composite image as a prompt word to obtain a language description of the composite image; Generate an image module, input the second product main body mask, the full image contour line map, the depth information map and the language description words into the diffusion model to generate the required replacement map.

6. The device for replacing a commodity with a commercial photo according to claim 5, characterized in that: The empty scene acquisition module specifically comprises: using the expansion algorithm of OpenCV to expand the first product mask n times with a kernel size of m*m to obtain the second product mask; using the LaMa algorithm to eliminate and repair the area corresponding to the second product mask in the first product image to obtain the empty scene image.

7. The device for replacing a commodity with a commercial photo according to claim 5, characterized in that: The acquisition generation information module specifically comprises: using the canny edge extraction algorithm of OpenCV to extract the contour line map of the whole image of the synthetic image; using the open source depth estimation algorithm MiDas to extract the depth information map of the synthetic image; The open source multimodal model BLIP is used to reverse the prompt words of the synthetic image to obtain a language description of the synthetic image.

8. The device for replacing a commodity with a commercial photo according to claim 5, characterized in that: The cutout operation is specifically as follows: the pixels of the picture uploaded by the user are judged. If the pixels of the picture are less than 2000×2000, the cutout operation is directly performed through the visual intelligence open platform to obtain the required main picture of the picture; if the pixels of the picture are greater than or equal to 2000×2000, the picture is scaled in proportion using the Imgproc.resize in OpenCV and the Imgproc.INTER_LANCZ0S4 algorithm to obtain a scaled picture; the scaled picture is cutout through the visual intelligence open platform to obtain a first mask picture; the first mask picture is proportionally restored to a second mask picture of the original size through the Imgproc.resize in OpenCV and the Imgproc.INTER_LANCZ0S4 algorithm; the transparent channel of each pixel in the picture uploaded by the user is read out to obtain a first matrix, the transparent channel of each pixel in the second mask picture is read out to obtain a second matrix, and Core.min is called to merge the first matrix with the second matrix to obtain a third matrix; The transparent channel in the picture uploaded by the user is replaced by the third matrix to obtain the required main picture of the picture.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images