Commodity background generation method and device based on FLUX.1 model, equipment and medium

Abstract

The invention provides a commodity background generation method and device based on an FLUX.1 model, equipment and a medium, and the method comprises the steps: carrying out the pixel-level semantic segmentation of a commodity image, generating a foreground mask containing soft edge information, and carrying out the processing of the commodity image, and obtaining a white-background commodity image; performing semantic feature reverse analysis on the original background of the commodity image based on a preset cue word template through a Janus-Pro-7B multi-mode model, and generating a background description vector containing an illumination direction and a material attribute; performing visual feature preprocessing, and inputting a processing result and a background description vector into a control network to generate a multi-modal control condition; and inputting the background description vector, the foreground mask and the multi-modal control condition into an FLUX.1 diffusion model to generate a background image, and realizing high-precision background generation and natural fusion through multi-modal feature decoupling, cross-model parameter linkage and a dynamic illumination matching algorithm.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment and medium for generating a commodity background based on a FLUX.1 model. Background Art

[0002] Existing image background generation technology suffers from the following core technical bottlenecks: Insufficient detail preservation: Traditional segmentation methods lose edge details of the product (especially in areas with complex textures like transparent materials and hair) due to excessive blurring or edge positioning errors, resulting in pixel-level deviations between the segmentation mask and the true boundary. Style consistency flaws: The lighting direction, material reflectivity, color tone, and texture of the replacement background and the product body are difficult to accurately match. This creates a noticeable sense of disconnection at the fusion boundary, resulting in poor visual consistency across the entire image. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method, device, equipment and medium for generating product background based on the FLUX.1 model, which realizes high-precision background generation and natural fusion through multimodal feature decoupling, cross-model parameter linkage and dynamic lighting matching algorithm.

[0004] In a first aspect, the present invention provides a method for generating product background based on the FLUX.1 model, comprising the following steps:

[0005] Step 1: Perform pixel-level semantic segmentation on the product image to generate a foreground mask containing soft edge information, and then process the product image to obtain a white background product image;

[0006] Step 2: Using the Janus-Pro-7B multimodal model, reversely analyze the semantic features of the original background of the product image based on a preset prompt word template to generate a background description vector that includes lighting direction and material properties;

[0007] Step 3: Perform visual feature preprocessing and input the processing results and background description vector into the control network to generate multimodal control conditions;

[0008] Step 4: Input the background description vector, foreground mask and multimodal control conditions into the FLUX.1 diffusion model to generate a background image.

[0009] In a second aspect, the present invention provides a product background generation device based on the FLUX.1 model, comprising:

[0010] The image processing module performs pixel-level semantic segmentation on the product image, generates a foreground mask containing soft edge information, and processes the product image to obtain a white background product image;

[0011] The reverse parsing module uses the Janus-Pro-7B multimodal model to perform reverse semantic feature analysis of the original background of the product image based on a preset prompt word template, generating a background description vector that includes lighting direction and material properties.

[0012] The control condition module performs visual feature preprocessing and inputs the processing results and background description vector into the control network to generate multimodal control conditions;

[0013] The generation module inputs the background description vector, foreground mask and multimodal control conditions into the FLUX.1 diffusion model to generate a background image.

[0014] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the method described in the first aspect when the program is executed by a processor.

[0016] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0017] The present invention can achieve pixel-level precise segmentation for complex texture areas such as transparent materials and hair, with a natural transition between the foreground and background boundaries, effectively avoiding the edge blur and detail loss problems caused by traditional methods, and significantly enhancing the integrity of the subject's contour.

[0018] The background generated by the present invention is highly matched with visual elements such as the lighting direction, material reflection characteristics, color tone and texture of the product body, eliminating the sense of picture fragmentation commonly seen in traditional methods and achieving a natural fusion of the product and the background.

[0019] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0021] Figure 1 This is a flowchart of the method in Example 1 of the present invention;

[0022] Figure 2 This is a schematic diagram of the structure of the device in Example 2 of the present invention. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the application have the following general ideas:

[0024] FLUX.1 text-to-image model is used to generate high-quality images from text descriptions.

[0025] FLUX.1-dev-ControlNet-Union-Pro-2.0 model: is a unified ControlNet model suitable for FLUX.1-dev, which optimizes Canny and pose control effects, adds soft edge support and cancels the tile function.

[0026] Janus-Pro-7B model is based on DeepSeek-LLM-7B base model, which decouples understanding and generation tasks by separating visual coding paths, uses SigLIP-L visual encoder and generation marker with a downsampling rate of 16; supports text-to-image generation and other tasks.

[0027] TTP partition (tile-based texture processing) is a partitioning technique for high-resolution images, mainly used to solve the problem of memory limit and computing efficiency when generating or editing large-size images. Direct full-image loading may cause memory overflow or model crash, while partitioning can cut the image into multiple small tiles, optimize each tile and seamlessly splice them together to achieve efficient and stable high-quality output.

[0028] BiRefNet (Bilateral Reference Network) is a high-resolution binary image segmentation model, which is used to solve the fine segmentation task of foreground and background, and supports background removal, mask generation, COD (Clothing Object Detection), SOD (Salient Object Detection) and other applications.

[0029] Depth-Anything-V2 model is a monocular depth estimation model, which is trained based on 595K synthetic labeled images + 62M + real unlabeled images.

[0030] The specific implementation is as follows:

[0031] 1. Fine segmentation stage

[0032] The BiRefNet high-resolution segmentation model is used to perform pixel-level semantic segmentation on the product image to generate a foreground mask (Mask) containing soft edge information, solving the problem of blurred segmentation of transparent objects and complex textures.

[0033] 2. Background semantic back-propagation stage

[0034] Using the Janus-Pro-7B multimodal model (developed by Deep Quest), based on preset prompt word templates (such as "product type + material + environmental keywords"), the semantic features of the original background of the input image are reversely parsed to generate a background description vector that includes lighting direction and material properties.

[0035] 3. Multimodal conditional control stage

[0036] Visual feature preprocessing:

[0037] Perform Canny edge detection on the foreground mask to generate a structural feature map;

[0038] Generate high-precision depth maps through the Depth-Anything-V2 monocular depth model to build spatial geometric constraints.

[0039] Conditional generation model: Utilize the FLUX.1-dev-ControlNet-Union-Pro-2.0 control network to fuse Canny edges, depth maps, and semantic description vectors to achieve multi-dimensional conditional control of the FLUX.1 text-based graph model.

[0040] 4. Background generation and optimization stage

[0041] Based on the semantic cue words and multimodal control conditions obtained by inverse deduction, the background image is generated through the FLUX.1 diffusion model. The TTP blocking technology (Tile Texture Processing) is used to solve the video memory limitation problem of high-resolution images, achieving block-by-block optimization and seamless splicing.

[0042] The BiRefNet high-resolution segmentation model uses a bilateral reference network architecture to preserve transparent material and hair details. The Janus-Pro-7B multimodal reverse parsing extracts background description vectors from the original background pixels based on a preset prompt word template ("product type + material + environmental keywords"). The FLUX.1-dev-ControlNet-Union-Pro-2.0 control network: Based on the FLUX.1 model, it is easy to cause problems such as product structure deformation and changes in details. When changing the background of a product, it is definitely hoped that the product will not be changed and a high-quality background will be generated. The fusion of Canny edge maps (geometric constraints), depth maps (spatial constraints), and semantic vectors (style constraints) can better control the structure of the product. At the same time, the FLUX.1 model can generate high-quality backgrounds, which are higher in quality than the images generated by the previous SD1.5 and XL models.

[0043] Example 1

[0044] like Figure 1 As shown, this embodiment provides a method for generating product background based on the FLUX.1 model, including the following steps:

[0045] Step 1: Perform pixel-level semantic segmentation on the product image to generate a foreground mask containing soft edge information, and then process the product image to obtain a white background product image;

[0046] Step 2: Using the Janus-Pro-7B multimodal model, reversely analyze the semantic features of the original background of the product image based on a preset prompt word template to generate a background description vector that includes lighting direction and material properties;

[0047] Step 3: Perform visual feature preprocessing and input the processing results and background description vector into the control network to generate multimodal control conditions;

[0048] Step 4: Input the background description vector, foreground mask and multimodal control conditions into the FLUX.1 diffusion model to generate a background image.

[0049] This embodiment preferably further includes step 5, dividing the generated background image into a set number of small blocks through TTP blocking, inputting each small block into the Flux model for detail enhancement, and merging all the generated small blocks through TTP blocking to obtain the desired background final image.

[0050] In this embodiment, preferably, the step 3 is specifically: visual feature preprocessing:

[0051] Perform Canny edge detection on the foreground mask to generate a structural feature map;

[0052] Generate a depth map based on the product image with a white background using the Depth-Anything-V2 monocular depth model;

[0053] The FLUX.1-dev-ControlNet-Union-Pro-2.0 control network is used to fuse the structural feature map, depth map, and background description vector to obtain multimodal control conditions.

[0054] In this embodiment, preferably, step 1 specifically includes: using the BiRefNet high-resolution segmentation model to perform pixel-level semantic segmentation on the product image, generating a foreground mask containing soft edge information, and processing the product image to obtain a white background product image.

[0055] Based on the same inventive concept, this application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0056] Example 2

[0057] like Figure 2 As shown, in this embodiment, a product background generation device based on the FLUX.1 model is provided, including:

[0058] The image processing module performs pixel-level semantic segmentation on the product image, generates a foreground mask containing soft edge information, and processes the product image to obtain a white background product image;

[0059] The reverse parsing module uses the Janus-Pro-7B multimodal model to perform reverse semantic feature analysis of the original background of the product image based on a preset prompt word template, generating a background description vector that includes lighting direction and material properties.

[0060] The control condition module performs visual feature preprocessing and inputs the processing results and background description vector into the control network to generate multimodal control conditions;

[0061] The generation module inputs the background description vector, foreground mask and multimodal control conditions into the FLUX.1 diffusion model to generate a background image.

[0062] This embodiment preferably also includes a detail enhancement module, which divides the generated background image into a set number of small blocks through TTP blocking, inputs each small block into the Flux model for detail enhancement, and merges all the generated small blocks through TTP blocking to obtain the required background final image.

[0063] In this embodiment, preferably, the control condition module is specifically: visual feature preprocessing:

[0064] Perform Canny edge detection on the foreground mask to generate a structural feature map;

[0065] Generate a depth map based on the product image with a white background using the Depth-Anything-V2 monocular depth model;

[0066] The FLUX.1-dev-ControlNet-Union-Pro-2.0 control network is used to fuse the structural feature map, depth map, and background description vector to obtain multimodal control conditions.

[0067] In this embodiment, preferably, the image processing module specifically: uses the BiRefNet high-resolution segmentation model to perform pixel-level semantic segmentation on the product image, generates a foreground mask containing soft edge information, and processes the product image to obtain a white background product image.

[0068] Since the device described in the second embodiment of the present invention is used to implement the method of the first embodiment of the present invention, those skilled in the art will be able to understand the specific structure and variations of the device based on the method described in the first embodiment of the present invention, and therefore will not be described in detail here. All devices used in the method of the first embodiment of the present invention fall within the scope of protection of the present invention.

[0069] Based on the same inventive concept, this application provides an electronic device embodiment corresponding to the first embodiment, see the third embodiment for details.

[0070] Example 3

[0071] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any implementation method in the first embodiment can be implemented.

[0072] Since the electronic device described in this embodiment is the device used to implement the method in Example 1 of this application, based on the method described in Example 1 of this application, those skilled in the art will be able to understand the specific implementation of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of this application will not be described in detail here. As long as the device used by those skilled in the art to implement the method in the embodiment of this application falls within the scope of protection to be provided by this application.

[0073] Based on the same inventive concept, this application provides a storage medium corresponding to Example 1, see Example 4 for details.

[0074] Example 4

[0075] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any implementation method in the first embodiment can be implemented.

[0076] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0078] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0080] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for generating product background based on the FLUX.1 model, characterized by: The steps include: Step 1: Perform pixel-level semantic segmentation on the product image to generate a foreground mask containing soft edge information, and then process the product image to obtain a white background product image; Step 2: Using the Janus-Pro-7B multimodal model, reversely analyze the semantic features of the original background of the product image based on a preset prompt word template to generate a background description vector that includes lighting direction and material properties; Step 3: Perform visual feature preprocessing and input the processing results and background description vector into the control network to generate multimodal control conditions; Step 4: Input the background description vector, foreground mask and multimodal control conditions into the FLUX.1 diffusion model to generate a background image.

2. The method for generating product background based on the FLUX.1 model according to claim 1, characterized in that: The process also includes step 5, dividing the generated background image into a set number of small blocks through TTP blocking, inputting each small block into the Flux model for detail enhancement, and merging all the generated small blocks through TTP blocking to obtain the desired background final image.

3. The method for generating product background based on the FLUX.1 model according to claim 1, characterized in that: The step 3 is specifically as follows: visual feature preprocessing: Perform Canny edge detection on the foreground mask to generate a structural feature map; Generate a depth map based on the product image with a white background using the Depth-Anything-V2 monocular depth model; The FLUX.1-dev-ControlNet-Union-Pro-2.0 control network is used to fuse the structural feature map, depth map, and background description vector to obtain multimodal control conditions.

4. The method for generating product background based on the FLUX.1 model according to claim 1, characterized in that: Specifically, step 1 includes: using the BiRefNet high-resolution segmentation model to perform pixel-level semantic segmentation on the product image, generating a foreground mask containing soft edge information, and processing the product image to obtain a white-background product image.

5. A product background generation device based on the FLUX.1 model, characterized by: include: The image processing module performs pixel-level semantic segmentation on the product image, generates a foreground mask containing soft edge information, and processes the product image to obtain a white background product image; The reverse parsing module uses the Janus-Pro-7B multimodal model to perform reverse semantic feature analysis of the original background of the product image based on a preset prompt word template, generating a background description vector that includes lighting direction and material properties. The control condition module performs visual feature preprocessing and inputs the processing results and background description vector into the control network to generate multimodal control conditions; The generation module inputs the background description vector, foreground mask and multimodal control conditions into the FLUX.1 diffusion model to generate a background image.

6. The device for generating product background based on the FLUX.1 model according to claim 5, characterized in that: It also includes a detail enhancement module, which divides the generated background image into a set number of small blocks through TTP blocking, inputs each small block into the Flux model for detail enhancement, and merges all the generated small blocks through TTP blocking to obtain the required background final image.

7. The device for generating product background based on the FLUX.1 model according to claim 5, characterized in that: The control condition module is specifically: visual feature preprocessing: Perform Canny edge detection on the foreground mask to generate a structural feature map; Generate a depth map based on the product image with a white background using the Depth-Anything-V2 monocular depth model; The FLUX.1-dev-ControlNet-Union-Pro-2.0 control network is used to fuse the structural feature map, depth map, and background description vector to obtain multimodal control conditions.

8. The device for generating product background based on the FLUX.1 model according to claim 5, characterized in that: The image processing module specifically uses the BiRefNet high-resolution segmentation model to perform pixel-level semantic segmentation on the product image, generate a foreground mask containing soft edge information, and process the product image to obtain a white background product image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images