An image texture transfer method based on a temporally decoupled diffusion model
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTH CHINA UNIV OF TECH
- Filing Date
- 2026-02-12
- Publication Date
- 2026-05-26
Smart Images
Figure CN121685688B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and in particular to an image texture transfer method based on a temporal decoupling diffusion model. Background Technology
[0002] Visual texture transfer refers to the technique of transferring the appearance and material features of a target texture image to a content image while maintaining the geometric structure and semantic layout of the content image. This technique has wide applications in industrial design, e-commerce, game asset creation, and metaverse content generation.
[0003] However, existing image generation and editing techniques, especially those based on Generative Adversarial Networks (GANs) or traditional style transfer, face the classic "structure-texture trade-off" problem when dealing with high-fidelity texture transfer. On the one hand, in pursuit of the ultimate texture fit, the model often destroys the original geometry of the object (for example, distorting the outline of a sofa to fit the texture pattern); on the other hand, if the structure is forcibly constrained, the texture often fails to blend naturally, presenting a stiff "sticker" effect.
[0004] In recent years, diffusion models have made groundbreaking progress in the field of image generation. Existing diffusion-based editing methods (such as Prompt-to-Prompt and Attention Distillation) mostly attempt to simultaneously consider structure and texture at each step of denoising. This simultaneous optimization approach is mathematically prone to getting trapped in local optima, making it difficult to achieve perfect decoupling and recombination of structure and texture in complex scenes. Therefore, a new method that can resolve the conflict between structure preservation and texture alignment from the perspective of the generation mechanism is urgently needed. Summary of the Invention
[0005] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the purpose of this invention is to provide an image texture transfer method based on a temporal decoupling diffusion model.
[0006] The objective of this invention is achieved through the following technical solution:
[0007] An image texture transfer method based on a temporally decoupled diffusion model includes the following:
[0008] Obtain the content image to be processed and the target texture image;
[0009] The content image and the target texture image are processed separately to obtain the content latent variables and texture feature vectors;
[0010] Initialize random noise and proceed to the reverse denoising process of the pre-trained diffusion model. The reverse denoising process includes a structure-dominated stage and a texture rendering stage.
[0011] In the structure-driven stage, a structure guidance module is introduced to calculate the structure consistency loss between the currently generated image and the content image, and convert the structure consistency loss into a gradient value to force the reverse denoising path to follow the geometric contour of the content image.
[0012] In the texture rendering stage, a texture injection module is used to inject the texture feature vector of the target texture image into a pre-trained diffusion model to achieve texture alignment and rendering.
[0013] The denoising process is completed, and the decoded output texture transfer image is generated.
[0014] Furthermore, the initialization of random noise specifically involves:
[0015] The content latent variables are added to the total number of denoising steps T of the pre-trained diffusion model by the denoising diffusion implicit model inversion technique to obtain the initial random noise.
[0016] Furthermore, a time threshold is set. When the denoising time step is greater than the time threshold, the structure-dominated stage is entered; when the denoising time step is less than the time threshold, the texture rendering stage is entered.
[0017] Furthermore, it also includes an adaptive smooth transition step, in which linear interpolation is used within the preset interval between the structure-driven stage and the texture rendering stage, and the weights of the structure guidance module and the texture injection module are adjusted as the denoising time step gradually decreases.
[0018] Furthermore, the time threshold is adaptively adjusted based on the edge complexity of the content image.
[0019] Furthermore, the texture injection module employs an attention mechanism, specifically in the self-attention layer of the pre-trained diffusion model, where the generated image features are used as query vectors, and the target texture image features are used as key vectors and value vectors for calculation.
[0020] Furthermore, the formula for calculating the structural consistency loss is as follows:
[0021]
[0022] in, For the currently generated image, For content images, This is a structural feature extraction function.
[0023] Furthermore, the structure guidance module employs an edge detection network or a depth estimation network to extract the spatial structure map of the content image.
[0024] Furthermore, the weights of the structure guidance module and the texture injection module were adjusted, specifically as follows:
[0025]
[0026] in, It is the corrected noise. It is the prediction noise at time step t. and These are structure-guided gradients and texture-guided gradients, respectively. and These are the weights of the structure guidance module and the texture injection module, respectively.
[0027] Furthermore, the higher the edge density of the content image, the larger the time threshold.
[0028] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0029] High-fidelity structure: By utilizing the temporal characteristics of the diffusion model generation process, the geometric contour is strongly locked during the high-noise stage, effectively avoiding the object deformation problem common in traditional methods.
[0030] Natural texture integration: Focusing on high-frequency detail generation during the low-noise stage allows textures to grow in accordance with the three-dimensional surface of the object, resulting in a more realistic and natural visual effect.
[0031] Plug and play (training-free): This method can be used as an optimization strategy in the inference stage, eliminating the need to retrain a large diffusion model for a specific object, which greatly reduces the cost of use.
[0032] Highly adaptable: By dynamically adjusting the time threshold, it can flexibly meet the migration needs of different types of objects (such as rigid and flexible objects). Attached Figure Description
[0033] Figure 1 is a schematic diagram of the workflow of an image texture transfer method based on a temporal decoupling diffusion model according to the present invention;
[0034] Figure 2 is a schematic diagram of the system hardware architecture based on cloud computing power cluster in an embodiment of the present invention;
[0035] Figure 3 is a schematic diagram comparing the generation effects of the method of the present invention and the prior art on different objects (sofa, teapot). Detailed Implementation
[0036] The present invention will be further described in detail below with reference to the embodiments, but the implementation of the present invention is not limited thereto.
[0037] Example
[0038] like Figure 1 As shown, an image texture transfer method based on a temporal decoupling diffusion model includes four parts: data preprocessing, temporal decoupling setting, staged denoising generation, and output decoding.
[0039] To support the efficient operation of the above method, this embodiment first constructs the hardware system architecture shown in Figure 2. This architecture mainly includes a user interaction terminal, a high-speed internet / 5G communication link, and a cloud-based high-performance computing server cluster, with the specific configuration as follows:
[0040] User interaction terminal: includes a data acquisition module (such as a camera, scanner or file upload interface) for acquiring the content image and texture image to be processed; and a result display module (such as a high-resolution display) for presenting the final generated migration image.
[0041] High-performance computing server cluster: As the core computing power support of the system, it internally deploys a data throughput controller, feature database, model weight loading, and diffusion model inference engine. The inference engine is built on a high-performance GPU array (such as an NVIDIA A100 or RTX 4090 cluster) and is responsible for executing computationally intensive denoising diffusion and dual-stream bootstrapping algorithms.
[0042] High-speed Internet / 5G communication link: The user terminal sends the pre-processed data packets to the server via 5G or high-speed Internet, and the server returns the results after completing the calculation.
[0043] Based on the above hardware environment, the specific processing steps of the image migration method in this embodiment are as follows:
[0044] Step 1: Data preprocessing.
[0045] Obtain the image of the content to be processed and target texture image The encoder that uses a variational autoencoder to encode the content image Mapping to the latent space yields the latent variables of the content. And use a texture feature extractor to extract the target texture image. The texture feature vectors are then used. Then, a denoising diffusion implicit model inversion technique is employed to extract the content latent variables. Total denoising time steps from adding noise to the pre-trained diffusion model , obtain initial random noise .
[0046] Step 2: Timing decoupling settings.
[0047] After initializing random noise, the reverse denoising process of the pre-trained diffusion model is initiated. This reverse denoising process includes a structure-dominated stage and a texture rendering stage.
[0048] By setting a time threshold, the relationship between the current denoising time step and the time threshold is determined, and the structure-dominant stage and texture rendering stage are segmented.
[0049] Specifically:
[0050] The total number of denoising steps for the pre-trained diffusion model is set to T; in this embodiment, 50 steps are used. A time threshold is introduced. ,For example Define the structure weight function. and texture weight function .
[0051]
[0052]
[0053] Where t represents the time step of the back diffusion (from Reduce to ), and This is the preset strength coefficient.
[0054] To further explain, the time threshold is adaptively adjusted based on the edge complexity of the content image. When the edge density of the content image is higher than a preset value, the time threshold is increased to prolong the structure-dominant phase.
[0055] The higher the edge density of the content image (i.e., the more complex the structure), the larger the set time threshold τ. This is a monotonically increasing or proportional relationship.
[0056] For example, simple structures (such as apples): with simple geometric contours, the pre-trained diffusion model can quickly recover a stable structure in the early stages of denoising (high noise stage), so the structure guidance can end earlier, reducing the time threshold τ value and leaving more steps for texture rendering.
[0057] Complex structures (such as hollowed-out bird nests or hair) contain a large amount of high-frequency edge information. Pre-trained diffusion models require more denoising steps to fully construct these intricate geometric structures. If the model switches to the texture stage too early, the complex structure is prone to collapse. Therefore, it is necessary to increase the time threshold τ and extend the structure-dominant stage to ensure that the skeleton is fully formed before applying color.
[0058] In this embodiment, a base threshold of 0.6T is set, and then an increment proportional to the edge density D is added. The time threshold formula is similar to: , where k is a predefined coefficient.
[0059] Step 3: Staged noise reduction and generation
[0060] The image generation process in this step is implemented using a pre-trained diffusion model, which generates images from initial randomized noise. Begin executing the denoising loop. At each denoising time step... Pre-trained diffusion model predicts noise .
[0061] When noise reduction time step Greater than the time threshold Then, it enters the structure-dominated stage; in this stage, the core task is to reconstruct the geometric structure of the content image. Specifically, a structure guidance module is introduced to calculate the structural consistency loss between the currently generated image and the content image and convert it into a gradient value. This gradient value is used as a control parameter to act inversely on the pre-trained diffusion model to guide the denoising direction and force the inverse denoising path to follow the geometric contour of the content image, thus preserving the original image structure.
[0062] Structural consistency loss Defined as:
[0063]
[0064] in, This represents a structural feature extraction operator (e.g., feature map extracted from layer 4 of the U-Net decoder, or using the Canny edge detection operator). This is achieved through computation... And inject it into the sampling process to force latent variables. Keep with content images Consistent spatial layout. At this time, the texture injection channel is closed to prevent messy textures from damaging the structure.
[0065] When noise reduction time step Less than the time threshold Then, in the texture rendering stage, the texture injection module is used to inject the texture feature vector of the target texture image into the pre-trained diffusion model to achieve texture alignment and rendering.
[0066] At this point, the structural constraints are removed, allowing the model to freely modify high-frequency details. This embodiment employs a cross-attention injection method. In the model's attention layer, Image features from the current backbone network, and and It is then replaced with the target texture image. Texture feature vector:
[0067]
[0068] In this way, the generated image retains the macroscopic structure determined in the previous stage, while its pixel details and stylistic features are forcibly replaced with the target texture image. Texture style.
[0069] Step 4: Within the preset interval between the structure-driven stage and the texture rendering stage, use linear interpolation to adjust the weights of the structure-guided module and the texture-injection module as the time step t gradually decreases.
[0070] Further explanation: In order to prevent in Abrupt jumps can occur at any time, causing sudden changes in the control signal and resulting in image artifacts. This embodiment addresses this issue. A linear interpolation strategy is used within the time window. That is, within the transition interval, the corrected noise prediction... The calculation formula is:
[0071]
[0072] in It decays linearly from 1 to 0. It increases linearly from 0 to 1.
[0073] Specifically: Since the generation process of the pre-trained diffusion model is a multi-step iterative denoising process from noise to image (usually involving dozens to hundreds of calculations, with the time step t gradually decreasing from T to 0), we set a preset time transition interval before and after switching the time threshold τ. Within this range, the system dynamically adjusts the weights based on the changes in the current denoising time step t, achieving "gradual control".
[0074] Structure guidance module weight As the denoising iteration process progresses (i.e., the time step t decreases), the weight value decreases linearly from 1 to 0;
[0075] Texture Injection Module Weights As the denoising iteration process progresses (i.e., the time step t decreases), the weight value increases linearly from 0 to 1.
[0076] In this way, by utilizing the continuous change of time steps, a smooth transition of the model's focus from "macrostructure" to "microtexture" is achieved.
[0077] Step 5: Complete the denoising process and decode the output texture transfer image. Specifically, when... When the final latent variable is obtained. After being decoded by the VAE decoder, the final texture transfer image is output on the display.
[0078] The pre-trained diffusion model in this embodiment adopts an existing pre-trained diffusion model such as StableDiffusion, whose core is the standard U-Net architecture, including an encoder, a decoder, and a self-attention layer.
[0079] Structure guidance module (external overlay): As an add-on module, it uses the predicted values output by the pre-trained diffusion model to calculate the gradient, which is then directly overlaid into the noise prediction to externally correct the generation direction.
[0080] Texture Injection Module (Internal Injection): As a plugin mechanism. During inference of the pre-trained diffusion model, it temporarily replaces the keys and values of its internal self-attention layers with texture features, which takes effect without retraining the model.
[0081] Effect verification and comparison:
[0082] To verify the beneficial effects of this invention, a comparative experiment was conducted between this method and existing technologies. For example... Figure 3 The first column contains the input images, the second column (middle column) uses the method described in the existing literature *Attention Distillation: A Unified Approach to Visual Characteristics Transfer*, and the third column represents this method.
[0083] Case 1 (Sofa): The input is a structural diagram of a typical fabric sofa and a crocodile skin texture. While existing technology (middle column) migrates the texture, it causes severe geometric distortion in the sofa armrests and backrest, damaging the original structure. This invention (right column), however, perfectly replicates the crocodile skin texture while strictly maintaining the sofa's industrial design contours, achieving decoupling between structure and texture.
[0084] Case 2 (Teapot): The input consists of a teapot line drawing and a blue-and-white porcelain texture. The result generated by this invention not only accurately colors the teapot but also allows the texture to naturally deform to conform to the curved surface of the teapot, eliminating the "sticker" feel and demonstrating the method's ability to align textures on complex curved surfaces.
[0085] This embodiment also provides an image texture transfer system based on a temporal decoupling diffusion model, including:
[0086] Input module: Used to obtain the content image and target texture image uploaded by the user.
[0087] Timing controller: Used to determine the structure-texture switching time point in the denoising process based on preset parameters or image analysis results.
[0088] Dual-stream guidance module: includes a structure guidance module and a texture injection module, which are used to calculate gradient guidance signals or feature injection signals at different time stages.
[0089] To further explain, the texture injection module uses a VGG-style feature extraction network, and the structure guidance module uses the Canny operator extractor.
[0090] Generation Engine: Based on a pre-trained diffusion model, it performs inverse denoising calculations using signals from a dual-stream guiding module.
[0091] This embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the image texture transfer method based on the temporal decoupling diffusion model described in the above embodiment.
[0092] Those skilled in the art will readily understand that the above description is merely an embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. An image texture transfer method based on a temporally decoupled diffusion model, characterized in that, Including the following: Obtain the content image to be processed and the target texture image; The content image and the target texture image are processed separately to obtain the content latent variables and texture feature vectors; Initialize random noise and proceed to the reverse denoising process of the pre-trained diffusion model. The reverse denoising process includes a structure-dominated stage and a texture rendering stage. A time threshold is set. When the denoising time step is greater than the time threshold, the structure-dominated stage is entered; when the denoising time step is less than the time threshold, the texture rendering stage is entered. It also includes an adaptive smooth transition step, which uses linear interpolation to adjust the weights of the structure guidance module and the texture injection module as the denoising time step gradually decreases within the preset interval between the structure-driven stage and the texture rendering stage. Adjust the weights of the structure guidance module and the texture injection module as follows: in, It is the corrected noise. It is the prediction noise at time step t. and These are structure-guided gradients and texture-guided gradients, respectively. and These are structure-guided weights and texture injection weights, respectively. In the structure-dominated stage, a structure-guided module is introduced to calculate the structure consistency loss between the currently generated image and the content image, and convert the consistency loss into gradient values to force the reverse denoising path to follow the geometric contour of the content image. In the texture rendering stage, a texture injection module is used to inject the texture feature vector of the target texture image into a pre-trained diffusion model to achieve texture alignment and rendering. The denoising process is completed, and the decoded output texture transfer image is generated.
2. The image texture transfer method according to claim 1, characterized in that, The initialization of random noise is specifically as follows: The content latent variables are added to the total number of denoising steps T of the pre-trained diffusion model by the denoising diffusion implicit model inversion technique to obtain the initial random noise.
3. The image texture transfer method according to claim 1, characterized in that, The time threshold is adaptively adjusted based on the edge complexity of the content image.
4. The image texture transfer method according to any one of claims 1-3, characterized in that, The texture injection module employs an attention mechanism. Specifically, in the self-attention layer of the pre-trained diffusion model, the generated image features are used as query vectors, and the target texture image features are used as key and value vectors for calculation.
5. The image texture transfer method according to any one of claims 1-3, characterized in that, The formula for calculating the structural consistency loss is as follows: in, For the currently generated image, For content images, This is a structural feature extraction function.
6. The image texture transfer method according to claim 1, characterized in that, The structure guidance module employs an edge detection network or a depth estimation network to extract the spatial structure map of the content image.
7. The image texture transfer method according to claim 3, characterized in that, The higher the edge density of the content image, the larger the time threshold.