E-commerce image harmonization and re-lighting method based on flow matching diffusion Transform
By using a flow-matching diffusion Transformer architecture, combined with self-attention and cross-attention mechanisms, the problems of foreground-background disharmony and detail loss in e-commerce image compositing are solved, achieving efficient and stable compositing of lighting and shadow effects, and improving the image processing efficiency and quality of e-commerce platforms.
Patent Information
- Application Number
- CN202511570005.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-30
- Publication Date
- 2026-02-06
AI Technical Summary
Existing technologies for e-commerce image synthesis suffer from problems such as disharmony between foreground and background, severe loss of detail, limited understanding of lighting, and unstable generation, resulting in unnatural synthesized images that fail to meet the high-efficiency processing requirements of e-commerce platforms.
Employing a flow-matching diffusion Transformer architecture that combines self-attention and cross-attention mechanisms, the system extracts deep features through an image encoder, adjusts foreground lighting and color using background illumination information, and generates shadows that match the background, achieving efficient end-to-end harmonization processing.
It preserves the original high-definition texture and structure of the product, generates highly realistic lighting and shadow effects that match the background environment, improves the efficiency and quality of e-commerce image synthesis, and ensures the stability and controllability of the generation process.
Smart Images
Figure CN121481908A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and image processing technology, specifically relating to an image synthesis method based on generative artificial intelligence, and in particular an automated method that uses a flow matching diffusion model and a Transformer architecture to harmonize the lighting, color, and shadow of foreground products and new background environments in e-commerce scenarios, and to achieve high-quality relighting. Background Technology
[0002] In the e-commerce industry, to enhance product display and appeal, merchants often composite extracted product images with diverse scene backgrounds. However, simple image overlay can lead to significant visual inconsistencies between the foreground and background in terms of lighting direction, intensity, color temperature, and shadows. The composite image appears fake and unnatural, severely impacting the professional image of the product and consumer trust.
[0003] Traditional solutions primarily rely on professional graphic designers manually adjusting images using software such as Adobe Photoshop. Designers need to spend a significant amount of time analyzing the background lighting environment and redrawing highlights, shadows, contact shadows, and ambient light reflections for the main product. This is an extremely time-consuming, costly, and highly skill-dependent process, making it difficult to meet the rapid processing needs of e-commerce platforms with their massive volumes of product images.
[0004] In recent years, deep learning-based image harmonization methods have made some progress, such as using normalizing models or diffusion models to learn image style transfer and color correction. However, these methods still have many limitations:
[0005] Poor ability to retain details: In the process of adjusting lighting and color, some generated models are prone to blurring or distorting the original texture and material details of the product, and even changing the size and shape of the product. This is unacceptable for e-commerce applications that require accurate display of product details.
[0006] Limited ability to understand lighting: Due to the inherent limitations of their local receptive field, past image generation models have difficulty fully understanding and modeling complex global lighting environments, resulting in unrealistic generated lighting effects (especially soft shadows and fine ambient light reflections).
[0007] Insufficient controllability and stability: The training process of diffusion and flow models is unstable, and the controllability of the generated results is poor, sometimes producing strange artifacts that do not conform to the laws of physics.
[0008] We propose a flow-matching diffusion Transformer as a generative model, achieving a breakthrough in image generation quality. However, its traditional iterative denoising and sampling process is typically slow. Flow matching, as a novel interpretation and training paradigm of the diffusion model, enables more efficient and stable training, allowing for the generation of high-quality images in a single step or with fewer steps. Furthermore, the Transformer architecture, with its powerful global dependency modeling capabilities, demonstrates great potential in understanding the overall contextual information of images.
[0009] Therefore, in order to address the pain points of disharmony between foreground and background in e-commerce image synthesis and the tendency of existing AI methods to lose details, there is an urgent need to study a new method that can accurately preserve the original features of foreground objects, deeply understand and simulate complex lighting environments, and efficiently generate highly realistic harmonization effects. Summary of the Invention
[0010] The purpose of this invention is to provide an e-commerce image harmonization and relighting method based on flow matching diffusion Transformer, in order to solve the problems mentioned in the background art, such as low efficiency of manual processing, damage to product details and limited harmonization effect of existing AI methods.
[0011] To achieve the above objectives, the present invention provides the following technical solution: a method for harmonizing and relighting e-commerce images based on flow matching diffusion Transformer, the method comprising the following steps:
[0012] S1. Image Input and Feature Encoding
[0013] The system receives a foreground product image and a target background image, and uses an image editing tool to place the foreground product image at the target position in the target background image. Then, it uses a shared image encoder to extract deep feature representations.
[0014] S2. Lighting and Color Harmonization Based on Flow Matching Diffusion Transformer
[0015] The features extracted from S1 are input into a core harmonization module. This module adopts a flow matching diffusion Transformer architecture, which adjusts the lighting and color of the foreground features based on background features to match the background environment. The Transformer layer deeply analyzes the global lighting environment of the background image and the geometric and material features of the foreground product through self-attention and cross-attention mechanisms. Based on this understanding, the flow matching diffusion process generates an optimal "harmonization field" or directly generates harmonized foreground features based on background features. This process is specifically used to adjust the lighting and color of the foreground to seamlessly match the background environment.
[0016] S3, Shadow Generation and Processing
[0017] By leveraging the Transformer module's understanding of the lighting environment, features of shadows (including contact shadows and cast shadows) that match the harmonized foreground are generated. The foreground image is standardized, and a matting model is used for mask acquisition. The input mask is expanded by pixels and Gaussian blurred, and the output mask and shadow features are output.
[0018] S4, Final Image Synthesis
[0019] The image decoder reconstructs the harmonized foreground mask and generated shadow features, accurately synthesizes them onto the target background image, and outputs the final, highly realistic e-commerce display image.
[0020] Preferably, the image encoder of S1 employs a pre-trained visual model SigLip2 to obtain high-level semantic features that are robust to image content and style.
[0021] Preferably, the S2 flow matching diffusion Transformer module includes a multi-head self-attention layer and a cross-attention mechanism. The cross-attention mechanism uses foreground features as a query and background features as keys and values, thereby injecting background illumination information into each spatial location of the foreground features to achieve pixel-level fine-grained illumination adjustment. Furthermore, the flow matching diffusion Transformer module embeds time step and conditional information by introducing adaptive layer normalization and performs noise prediction in the latent space. This module includes a multi-head self-attention layer and a point-by-point feedforward network, employing gated attention units to enhance feature interaction capabilities. It also combines channel dimension scaling and residual connections to maintain training stability. The specially designed cross-attention mechanism in the module can inject background illumination information into each spatial location of the foreground features, achieving pixel-level fine-grained illumination adjustment while maintaining the integrity of the foreground structure and texture.
[0022] Preferably, the shadow generation process of S3 and the harmonization process of S2 are performed jointly in the model and share some network layers to ensure that the generated shadows and the foreground after relighting maintain physical consistency in direction, softness, and color.
[0023] Preferably, the image pixel expansion in S3 is performed using a traditional interpolation method.
[0024] Preferably, the image decoder of S4 employs a variational autoencoder (VAE) decoder to generate the final synthesized image.
[0025] Preferably, it also includes constructing a large-scale, high-quality training dataset, which is obtained by professionally processing real scene images through a 3D rendering engine and contains quadruple data of (foreground, mask, background, and composite result) to provide accurate ground truth for supervised learning of the model.
[0026] A computer program product includes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the method described above.
[0027] An electronic device includes a processor and a memory; the memory stores computer program instructions, which, when executed by the processor, implement the method described above.
[0028] The technical effects and advantages provided by the present invention in the above technical solution are as follows:
[0029] (1) Detail fidelity: By decoupling the harmonization task into adjusting the lighting and color, rather than regenerating the entire image, and by utilizing the precise attention mechanism of Transformer, this method can preserve the original high-definition texture, structure and size of the product to a great extent, thus solving the core pain point of existing AI methods.
[0030] (2) Highly realistic harmonization effect: Combining Transformer's deep understanding of global illumination and the powerful generation quality of the flow matching diffusion model, it can create extremely delicate light and shadow effects, ambient light reflection and color blending that are perfectly matched with the background environment and conform to physical laws, with effects comparable to those of professional designers.
[0031] (3) High efficiency and automation: The entire process is an end-to-end automated process, which greatly shortens the image production cycle and reduces hours of manual work to seconds, greatly improving the efficiency of e-commerce content production.
[0032] (4) Stable and controllable generation process: The training paradigm based on flow matching is more stable than traditional diffusion models and GANs, and generates higher quality images with fewer artifacts, ensuring the reliability of commercial applications. Attached Figure Description
[0033] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.
[0034] Figure 1 This is a flowchart of the present invention;
[0035] Figure 2 This is a schematic diagram of the core flow matching diffusion Transformer module structure in the method of this invention;
[0036] Figure 3 This is a schematic diagram of the flow matching data distribution in this invention;
[0037] Figure 4 This is a diagram illustrating the actual effect of the invention. Detailed Implementation
[0038] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.
[0039] Please see Figures 1-4 As shown, the present invention provides the following technical solution:
[0040] A method for harmonization and relighting of e-commerce images based on flow matching diffusion Transformer, such as... Figure 1 As shown, it includes the following steps:
[0041] S1. Image Input and Feature Encoding
[0042] The system receives a foreground product image and a target background image. Using an image editing tool, the foreground product image is placed at the target position in the target background image. Then, a shared image encoder is used to extract deep feature representations. The image encoder uses a pre-trained visual model SigLip2 to obtain high-level semantic features that are robust to image content and style.
[0043] S2. Lighting and Color Harmonization Based on Flow Matching Diffusion Transformer
[0044] The features extracted from S1 are input into a core harmonization module. This module employs a flow matching diffusion Transformer architecture, which adjusts the lighting and color of the foreground features based on background features to match the background environment. The Transformer layer uses self-attention and cross-attention mechanisms to deeply analyze the global lighting environment of the background image and the geometric and material features of the foreground product. Based on this understanding, the flow matching diffusion process generates an optimal "harmonization field" or directly generates harmonized foreground features, conditioned on background features. This process is specifically designed to adjust the lighting and color of the foreground to seamlessly match the background environment. The flow matching diffusion Transformer module includes a multi-head self-attention layer and a cross-attention mechanism. The force mechanism uses foreground features as queries and background features as keys and values, thereby injecting background illumination information into each spatial location of the foreground features to achieve pixel-level fine-grained illumination adjustment. Furthermore, the Flow Matching Diffusion Transformer module embeds time-step and conditional information by introducing adaptive layer normalization and performs noise prediction in the latent space. This module includes a multi-head self-attention layer and a point-by-point feedforward network, employing gated attention units to enhance feature interaction capabilities. It also combines channel dimension scaling and residual connections to maintain training stability. The specially designed cross-attention mechanism within the module can inject background illumination information into each spatial location of the foreground features, achieving pixel-level fine-grained illumination adjustment while maintaining the integrity of the foreground structure and texture.
[0045] S3, Shadow Generation and Processing
[0046] By leveraging the Transformer module's understanding of the lighting environment, features of shadows (including contact shadows and cast shadows) that match the harmonized foreground are generated. The foreground image is standardized, and a matting model is used for mask acquisition. The input mask is pixel-expanded and Gaussian blurred, and the output mask and shadow features are output. Among them, the image pixel expansion is performed using a traditional interpolation method.
[0047] S4, Final Image Synthesis
[0048] The image decoder reconstructs the harmonized foreground mask and generated shadow features, accurately synthesizing them onto the target background image to output a final, highly realistic e-commerce display image (e.g., ...). Figure 4 The image decoder employs a variational autoencoder (VAE) decoder to generate the final synthesized image.
[0049] Furthermore, the shadow generation process of S3 and the harmonization process of S2 are carried out jointly in the model and share some network layers to ensure that the generated shadows and the relit foreground maintain physical consistency in direction, softness, and color.
[0050] Furthermore, it also includes building a large-scale, high-quality training dataset, which is obtained by professionally processing real scene images through a 3D rendering engine. This dataset contains quadruple data (foreground, mask, background, and composite result) to provide accurate ground truth for supervised learning of the model.
[0051] A computer program product includes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the method described above.
[0052] An electronic device includes a processor and a memory; the memory stores computer program instructions, which, when executed by the processor, implement the method described above.
[0053] Implementation Method 1:
[0054] like Figure 1 As shown, after the process begins, an image is first input and its features are encoded; then, the lighting and color of the image are harmonized using a flow matching diffusion Transformer method; subsequently, shadows are generated and processed; finally, all elements are synthesized, the final image is output, and the process ends.
[0055] Implementation Method Two:
[0056] like Figure 2 As shown, the core of this method is the harmonization of lighting and color based on the flow matching diffusion Transformer. The encoded foreground features, background features, and mask features are fed into a specially designed flow matching diffusion Transformer module, as shown below. Figure 2 As shown, this module consists of multiple stacked Transformer blocks.
[0057] Cross-Attention: In each Transformer block, foreground features serve as the query, while background features serve as the key and value. This allows the model to query the lighting information of the entire background for each "pixel block" in the foreground and incorporate it into the foreground features. For example, the model can learn that there is a warm window light source on the right side of the background, and thus generate a soft, warm highlight on the right side of the foreground object accordingly.
[0058] Self-attention: Self-attention is calculated on foreground features that incorporate background information to ensure that the harmonization effect is smooth and consistent across the entire object.
[0059] like Figure 3 As shown, by "patchify" the noisy image latent variables into a sequence, the powerful self-attention mechanism of Transformer is used to learn global image features. At the same time, through the innovative adaLN-Zero mechanism, the model can efficiently incorporate conditional information such as time steps and class labels into each step of the calculation, thereby precisely controlling the image generation process.
[0060] Implementation Method 3:
[0061] The specific method for image editing in S3 is as follows: input image and mask (if any), image normalization (size), output image and mask (if any), perform RMBG2.0 cutout and binarization on the normalized output image to obtain the mask (mask one); if there is an input mask, then perform pixel expansion and Gaussian blur on the input mask (mask two), and select add mode for the output image, the original image and the mask input to perform detail transfer on the mask area (mask two has a higher priority than mask one, choose one of the two).
[0062] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.
Claims
1. A method for harmonization and relighting of e-commerce images based on flow matching diffusion Transformer, characterized in that, The method includes the following steps: S 1. Image Input and Feature Encoding: Receive a foreground product image and a target background image, and use an image editing tool to place the foreground product image at the target position in the target background image. Then, use a shared image encoder to extract deep feature representations. S2. Lighting and color harmonization based on flow matching diffusion Transformer: The features extracted in S1 are input into a core harmonization module. This module adopts the flow matching diffusion Transformer architecture, which adjusts the lighting and color of the foreground features based on the background features to match the background environment. S 3, Shadow Generation and Processing: Utilizing the Transformer module's understanding of the lighting environment, generate shadow features (including contact shadows and cast shadows) that match the harmonized foreground. Standardize the foreground image and use a matting model to obtain a mask. Expand the input mask pixels and apply Gaussian blur to output mask and shadow features. S4. Final Image Synthesis: Using an image decoder, the harmonized foreground mask and generated shadow features are reconstructed and accurately synthesized onto the target background image, outputting the final, highly realistic e-commerce display image.
2. The method for harmonization and relighting of e-commerce images based on flow matching diffusion Transformer according to claim 1, characterized in that: The image encoder of S1 employs a pre-trained visual model SigLip2 to obtain high-level semantic features that are robust to image content and style.
3. The method for harmonization and relighting of e-commerce images based on flow matching diffusion Transformer according to claim 1, characterized in that: The S2 stream matching diffusion Transformer module includes a multi-head self-attention layer and a cross-attention mechanism. The cross-attention mechanism uses foreground features as queries and background features as keys and values, thereby injecting background lighting information into each spatial location of the foreground features to achieve pixel-level fine lighting adjustment.
4. The method for harmonization and relighting of e-commerce images based on flow matching diffusion Transformer according to claim 1, characterized in that: The shadow generation process in S3 and the harmonization process in S2 are performed together in the model and share some network layers to ensure that the generated shadows and the foreground after relighting maintain physical consistency in direction, softness, and color.
5. The method for harmonization and relighting of e-commerce images based on flow matching diffusion Transformer according to claim 1, characterized in that: The image pixel expansion in S3 uses a traditional interpolation method.
6. The method for harmonization and relighting of e-commerce images based on flow matching diffusion Transformer according to claim 1, characterized in that: The image decoder of S4 employs a variational autoencoder (VAE) decoder to generate the final synthesized image.
7. A method for harmonization and relighting of e-commerce images based on flow matching diffusion Transformer according to any one of claims 1-6, characterized in that: It also includes building a large-scale training dataset, which is obtained through 3D rendering and other methods, and contains quadruple data (foreground, mask, background, and composite result) to provide accurate ground truth for supervised learning of the model.
8. A computer program product, characterized in that: The method includes a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the e-commerce image harmonization and relighting method based on flow matching diffusion Transformer as described in any one of claims 1-7.
9. An electronic device, characterized in that: Including processor and memory; The memory stores computer program instructions, which, when executed by the processor, implement the e-commerce image harmonization and relighting method based on flow matching diffusion Transformer as described in any one of claims 1-7.