Diffusion style transfer method and system based on latent distribution re-anchoring dynamic injection

Abstract

This invention relates to the field of image style transfer technology, providing a diffusion style transfer method and system based on latent distribution re-anchoring dynamic injection. The method includes: performing second-order statistical alignment of content latent variables and style latent variables using latent covariance recoloring; automatically locating the style injection starting point based on the edge intensity of the content image; achieving latent distribution re-anchoring by injecting latent distribution re-anchoring noise into the initial mixed noise latent variables; fusing content self-attention features and style self-attention features through a soft + dynamic style injection module; and progressively injecting the fused attention features into the attention layer of the diffusion process at the injection starting point during backsampling of the calibrated latent variables, performing diffusion denoising to generate a style-transferred image. This invention preserves images rich in structural detail, achieving excellent content structure preservation while realizing accurate style reproduction.

Description

Technical Field

[0001] This invention relates to the field of image style transfer technology, and in particular to a diffusion style transfer method and system based on latent distribution re-anchoring dynamic injection. Background Technology

[0002] Existing style transfer research typically focuses on explicitly decoupling style from content to achieve interpretable and controllable style transfer. These methods primarily utilize the generative capabilities of pre-trained diffusion models (DMs), but all require additional gradient optimization for fine-tuning, making them time-consuming. To address these challenges, some studies have proposed training-free style transfer schemes. These methods manipulate the self-attention features of pre-trained large-scale DMs to transfer texture from style images to content images. While these methods have shown promising results, they still encounter issues such as ensuring naturalness and color consistency in the generated results, as well as content leakage in the transferred images. The issues of coherence and naturalness in the generated results, as well as color, manifest in some generated images that often exhibit unnatural visual effects. Not only is there a lack of organic integration between style and content, resulting in a noticeable disconnect, but there are also instances of unstable color fluctuations. The core reason for these problems lies in the limitations of the AdaIN method, which relies solely on matching channel means and variances on latent variables. It can only achieve alignment at the first-order statistical level and cannot capture the more crucial correlation information between style and content latent variables, making it difficult to support the requirements for style consistency, content integration, and color stability in the generated results. The problem of content leakage in transferred images specifically manifests as the destruction of some structural elements in the generated images after style injection. For example, facial distortion occurs when generating people, and objects lose their original shape and characteristics when generating still life. The core reason for this problem is that existing style injection methods have several shortcomings: First, even though the self-attention layer is crucial for style injection, it needs optimization. Early injection directly overturns the original structural foundation of the content, lacking protection for the content structure. Second, the timing of style injection is difficult to control precisely. Injecting too early further damages the integrity of the content structure, while injecting too late fails to allow the style to fully integrate, resulting in weak style expression. Summary of the Invention

[0003] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention provides a diffusion style transfer method and system based on latent distribution re-anchoring dynamic injection. By progressively injecting style, it avoids the destructive impact of style information in the early sampling stage on the content structure. It adaptively determines the injection starting point based on the edge complexity of the content image, thus protecting images with rich structural details. This achieves excellent content structure preservation capabilities while realizing accurate style reproduction.

[0004] This invention provides a diffusion style transfer method based on latent distribution re-anchoring dynamic injection, comprising:

[0005] S1: Obtain the content image and style image. Through the denoising diffusion implicit model inverse diffusion process, the content image is inverted to the high-noise state of the latent space to obtain the content latent variables and content self-attention features. Through the denoising diffusion implicit model inverse diffusion process, the style image is inverted to the high-noise state of the latent space to obtain the style latent variables and style self-attention features.

[0006] S2: The latent variables of content and style are second-order statistically aligned using the latent covariance recoloring method to obtain the initial mixed noise latent variables;

[0007] S3: Calculate the grayscale image of the content image using the Sobel gradient operator, calculate the edge intensity using the grayscale image of the content image, and automatically locate the style injection starting point based on the edge intensity;

[0008] S4: Obtain the calibrated latent variables by injecting latent distribution reanchoring noise into the initial mixed noise latent variables;

[0009] S5: The content self-attention features and style self-attention features are fused through the soft + dynamic style injection module to obtain the fused attention features;

[0010] S6: During the backsampling of the calibrated latent variables, the fused attention features are progressively injected into the attention layer of the diffusion process at the injection starting point to perform diffusion denoising and generate a style transfer image.

[0011] Furthermore, step S2 includes:

[0012] S21: Flatten the implicit variables of content and style in the spatial dimension;

[0013] S22: Calculate the decentralized characteristics of the flattened content latent variables and style latent variables;

[0014] S23: Calculate the content covariance matrix based on the decentralized characteristics of the flattened content latent variables, and calculate the style covariance matrix based on the decentralized characteristics of the flattened style latent variables.

[0015] S24: Perform eigenvalue decomposition on the content covariance matrix and style covariance matrix to obtain content eigenvalues ​​and style eigenvalues;

[0016] S25: Construct a whitening matrix based on content feature values, and a coloring matrix based on style feature values;

[0017] A latent covariance recoloring transformation matrix is ​​constructed based on the whitening matrix and the coloring matrix. The content latent variables and style latent variables are then statistically aligned using the latent covariance recoloring transformation matrix to obtain the initial mixed noise latent variables.

[0018] Furthermore, step S3 includes:

[0019] S31: Normalize the content image and calculate the grayscale image of the normalized content image;

[0020] S32: Calculate the horizontal and vertical gradient maps of the normalized grayscale image of the content image using the Sobel gradient operator;

[0021] S33: Calculate the edge strength based on the gradient map in the horizontal direction and the gradient map in the vertical direction;

[0022] S34: Normalize the edge intensity and linearly map it to the injection start point interval to obtain the injection start point.

[0023] Furthermore, the expression for calculating the calibrated latent variables is as follows:

[0024]

[0025] in, for Latent variables after time-calibration for Latent variables in the time-diffusion model For noise intensity, For latent distribution re-anchoring noise, It follows a standard normal distribution.

[0026] Furthermore, the soft + dynamic style injection module includes key-value soft injection and dynamic parameter adjustment.

[0027] Furthermore, key-value soft injection includes:

[0028] S511: Obtain the sampling progress and normalize the sampling progress;

[0029] S512: Construct the mixing coefficients based on the normalized sampling progress;

[0030] S513: Based on the mixing coefficient, perform linear interpolation between the keys in the content self-attention feature and the keys in the style self-attention feature to obtain the keys for fused attention;

[0031] S514: Based on the mixing coefficient, perform linear interpolation between the values ​​in the content self-attention feature and the values ​​in the style self-attention feature to obtain the value of fused attention.

[0032] Furthermore, dynamic parameter adjustment includes:

[0033] S521: Use cosine scheduling to smoothly transition from a high initial value to a low termination value to construct the query retention rate hyperparameter;

[0034] S522: Based on the intermediate results of the mixed content query and migration diffusion model using the query retention rate hyperparameter, obtain the final query in the self-attention process.

[0035] Furthermore, dynamic parameter adjustment also includes:

[0036] Temperature scaling parameters are constructed based on the normalized sampling progress;

[0037] The attention map before softmax in the self-attention layer is magnified based on the temperature scaling parameter.

[0038] Furthermore, the key-value soft injection and the dynamic parameter adjustment only take effect in the N attention layers within the diffusion sampling cycle.

[0039] This invention also provides a diffusion style transfer based on latent distribution re-anchoring dynamic injection, for performing the aforementioned diffusion style transfer method based on latent distribution re-anchoring dynamic injection, comprising:

[0040] The feature extraction module acquires content images and style images. Through the denoising diffusion implicit model inverse diffusion process, the content image is inverted to a high-noise state in the latent space to obtain content latent variables and content self-attention features. Through the denoising diffusion implicit model inverse diffusion process, the style image is inverted to a high-noise state in the latent space to obtain style latent variables and style self-attention features.

[0041] The alignment module performs second-order statistical alignment of content latent variables and style latent variables using the latent covariance recoloring method to obtain the initial mixed noise latent variables.

[0042] An automatic positioning module calculates the grayscale image of the content image using the Sobel gradient operator, calculates the edge intensity using the grayscale image of the content image, and automatically locates the style injection starting point based on the edge intensity.

[0043] The re-anchoring module obtains the calibrated latent variables by injecting latent distribution re-anchoring noise into the initial mixed noise latent variables;

[0044] The soft + dynamic style injection module fuses content self-attention features and style self-attention features to obtain fused attention features.

[0045] The transfer generation module, during the backsampling of the calibrated latent variables, progressively injects the fused attention features into the attention layer of the diffusion process at the injection starting point to perform diffusion denoising and generate a style transfer image.

[0046] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0047] This invention avoids the destructive impact of style information on the content structure in the early sampling stage by progressive style injection. It adaptively determines the injection starting point based on the edge complexity of the content image, thus protecting images with rich structural details. This achieves excellent content structure preservation capabilities while realizing accurate style reproduction.

[0048] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating a diffusion style transfer method based on latent distribution re-anchoring dynamic injection provided by the present invention.

[0051] Figure 2 This is a schematic diagram of the soft + dynamic style injection module provided by the present invention.

[0052] Figure 3 This is a schematic diagram of the structure of a diffusion style migration system based on latent distribution re-anchoring dynamic injection provided by the present invention.

[0053] Figure label:

[0054] 101. Feature Extraction Module; 102. Alignment Module; 103. Automatic Localization Module; 104. Re-anchoring Module; 105. Soft + Dynamic Style Injection Module; 106. Transfer Generation Module. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but cannot be used to limit the scope of this invention.

[0056] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0057] The following is combined Figures 1 to 3 This invention describes a diffusion style transfer method and system based on latent distribution re-anchoring dynamic injection.

[0058] This invention addresses three core issues in diffusion models: the quality of initial latent variables, attention injection strategy and intensity scheduling, and the accuracy and distribution stability of injection starting points. It proposes a complete train-free style transfer framework. The overall process involves performing a Denoising Diffusion Implicit Model Inverse Diffusion (DDIM) inversion on both content and style images, simultaneously collecting self-attention features and latent variables at each step of the inversion process. The core adopts a training-free paradigm of "query (Q) taken from content, key (K) / value (V) taken from style + query retention rate hyperparameter γ + attention temperature scaling parameter τ," and strengthens the statistical correction of initial latent variables. During the sampling phase, backsampling is performed primarily on content, with style information injected only in the later stages of the decoder (focusing on local texture layers), achieving accurate fusion of style and content.

[0059] To address the issue of inconsistent generated results when relying solely on AdaIN, this invention proposes a Latent Covariance Recoloring (LCR) method. Its core is to achieve second-order statistical renormalization on the initial latent variables: by calculating the covariance matrix of content and style latent variables, a coloring matrix is ​​constructed through eigenvalue decomposition to complete the latent variable remapping, aligning the channel association patterns of content latent variables with the style. Compared to StyleID's "initial latent AdaIN," this invention upgrades "channel statistical alignment" to "second-order statistical alignment." Secondly, this invention injects style information through a self-attention layer, while designing a soft injection and dynamic adjustment strategy for query retention / attention temperature. Borrowing the "progressive hybrid" idea of ​​h-space injection, it achieves progressive soft injection from content to style by linear interpolating the cached K / V tensor. Finally, to address the difficulties in controlling the injection timing (too early destroys the structure, too late weakens the style) and the distribution drift caused by multiple latent variable modifications, this invention proposes a combined scheme of automatic localization and latent distribution re-anchoring module (AS-LDR). By performing Sobel edge statistics on the content graph, the detail richness is linearly mapped to the injection start step, and high-frequency loss caused by repeated modifications in the sampling loop is avoided.

[0060] First, the content image and style images By inverting the high-noise state in the latent space through the reverse diffusion process of the denoising diffusion implicit model, we obtain... and The Latent Covariance Recoloring (LCR) module is used to obtain the initial mixed noise by combining the statistical properties of content noise and style noise. At the same time, through the latent distribution re-anchoring mechanism in Injecting enhanced noise effectively restores high-frequency details. In the... During the forward diffusion sampling process, the automatic localization module adaptively calculates the start time of style injection based on the complexity of the content image and progressively injects the self-attention feature information extracted during the style inversion process into the self-attention layer of the diffusion process, thus completing the entire diffusion denoising process and generating a style transfer image.

[0061] like Figure 1 As shown, a diffusion style transfer method based on latent distribution re-anchoring dynamic injection includes:

[0062] S1: Obtain the content image and style image. Through the denoising diffusion implicit model inverse diffusion process, the content image is inverted to the high-noise state of the latent space to obtain the content latent variables and content self-attention features. Through the denoising diffusion implicit model inverse diffusion process, the style image is inverted to the high-noise state of the latent space to obtain the style latent variables and style self-attention features.

[0063] Inversion is performed on the content image and style image respectively, and the self-attention features and latent variables of each step in the inversion process are collected simultaneously.

[0064] S2: The latent variables of content and style are second-order statistically aligned using the latent covariance recoloring method to obtain the initial latent variables;

[0065] The LCR method performs a complete second-order statistical alignment of content and style latent variables at the initial stage of the diffusion process. Its core idea draws inspiration from the whitening-coloring transformation theory in color science. First, it eliminates the original channel correlations of content latent variables through whitening transformation. Then, it applies the covariance structure of style latent variables for coloring transformation, thereby preserving the spatial layout of the content while fully transferring the color distribution characteristics of the style, resulting in a more harmonious image. Compared to AdaIN, LCR not only matches the mean and variance of the channels, but more importantly, it aligns the complete covariance matrix, enabling the stylization result to more accurately reproduce the color relationships of the target style and reflect greater naturalness.

[0066] S21: Flatten the implicit variables of content and style in the spatial dimension;

[0067] Given content, implicit variables , ,in, For the sample size, The number of channels in the image. For height, For width, For dimension The real space, style implicit variables , Flattening in the spatial dimension yields the hidden variables of the flattened content. Style hidden variables after flattening , ,in, For spatial length, , For dimension The real number space;

[0068] S22: Calculate the decentralized characteristics of the flattened content latent variables and style latent variables;

[0069] The expression for calculating the decentralized feature of the flattened latent variables is as follows:

[0070]

[0071] in, To achieve the decentralized characteristics of the latent variables in the flattened content, This is the mean vector of the content latent variables along the channel dimension. , The length of the space after flattening. For dimension The real number space;

[0072] The expression for calculating the decentralized features of the flattened style latent variables is as follows:

[0073]

[0074] in, To achieve the decentralized characteristics of the latent variables in the flattened content, This is the mean vector of the content latent variables along the channel dimension. ;

[0075] S23: Calculate the content covariance matrix based on the decentralized characteristics of the flattened content latent variables, and calculate the style covariance matrix based on the decentralized characteristics of the flattened style latent variables.

[0076] The expression for calculating the content covariance matrix is:

[0077]

[0078] in, For the content covariance matrix, For dimension The real number space;

[0079] The expression for calculating the style covariance matrix is:

[0080]

[0081] in, The style covariance matrix;

[0082] S24: Perform eigenvalue decomposition on the content covariance matrix and style covariance matrix to obtain content eigenvalues ​​and style eigenvalues;

[0083] The covariance matrix of each sample is decomposed into eigenvalues, and the calculation expression is as follows:

[0084]

[0085]

[0086] in, For the first The eigenvalues ​​of the content covariance matrix of each sample For the first The content feature vector matrix of each sample For the first The diagonal eigenvalue matrix of each sample For the first The eigenvalues ​​of the style covariance matrix of each sample. For the first The style feature vector matrix of each sample For the first The style diagonal eigenvalue matrix of each sample. This is the transpose of the matrix;

[0087] S25: Construct a whitening matrix based on content feature values, and a coloring matrix based on style feature values;

[0088] The expression for calculating the whitening matrix is:

[0089]

[0090] in, For whitening matrix, Covariance structures used to eliminate latent variables in content;

[0091] The expression for calculating the coloring matrix is:

[0092]

[0093] in, The coloring matrix;

[0094] S26: Construct a latent covariance recoloring transformation matrix based on the whitening matrix and the coloring matrix. Align the content latent variables and style latent variables using the latent covariance recoloring transformation matrix to obtain the initialized mixed noise latent variables. The calculation expression is:

[0095]

[0096] in, To initialize the hidden variables of mixed noise.

[0097] Transformed and Having the same covariance matrix while retaining Spatial structure information.

[0098] S3: Calculate the grayscale image of the content image using the Sobel operator, calculate the edge intensity using the grayscale image of the content image, and automatically locate the style injection starting point based on the edge intensity;

[0099] The injection start point is adaptively determined based on the structural complexity of the content image. The structural complexity of the content image directly affects the choice of style injection timing: images with rich details and complex edges need to be injected later to fully protect the structure, while images with simple structures can be injected earlier to enhance the style effect.

[0100] S31: Normalize the content image and calculate the grayscale image of the normalized content image. The expression for calculating the grayscale image is:

[0101]

[0102] in, This is the grayscale image of the content image after normalization. The number of channels in the image. For the first The weighting coefficients for each channel, For content images,

[0103] S32: Calculate the horizontal and vertical gradient maps of the normalized grayscale image using the Sobel gradient operator. The calculation expression is as follows:

[0104]

[0105]

[0106] in, This is the gradient plot in the horizontal direction. This is the gradient map in the vertical direction. For the horizontal Sobel operator, For the Sobel operator in the vertical direction, This is a two-dimensional convolution operation;

[0107] S33: Calculate the edge intensity based on the gradient maps in the horizontal and vertical directions. The edge intensity statistic is defined as the spatial average of the gradient magnitude, and the calculation expression is:

[0108]

[0109] in, For edge strength, coordinates The gradient plot in the horizontal direction. coordinates The gradient plot in the vertical direction;

[0110] S34: Normalize the edge intensity and linearly map it to the injection start interval. The calculation expression is as follows:

[0111]

[0112] in, As the starting point for the mapped injection, The lower limit of the injection starting interval, To be the upper limit of the starting interval for injection, For the clipping function, This represents the minimum edge strength. This represents the maximum value of the edge strength.

[0113] The default parameter is set to The greater the edge strength (the richer the structural details), the larger the normalized value E, and the later the injection. The later the injection, the more content-driven the early sampling stage (high signal-to-noise ratio (SNR)) and the more stable the structure. Conversely, images with simple structures can be injected earlier, enhancing the style effect while maintaining the structure.

[0114] S4: Obtain the calibrated latent variables by injecting latent distribution reanchoring noise into the initial mixed noise latent variables;

[0115] Latent distribution re-anchoring is achieved by injecting a tiny amount of Gaussian noise during the initialization phase outside the sampling loop. In the sampling process of the diffusion model, the distribution characteristics of the initial mixed noise latent variables directly affect the quality of the final generated image. The latent variables after LCR statistical alignment may be too deterministic, lacking the necessary randomness to recover high-frequency details. Therefore, LDR-noise is injected into the latent variables after LCR, and the calculation expression is as follows:

[0116]

[0117] in, for Latent variables after time-calibration These are latent variables in the diffusion model. For noise intensity, For LDR-noise;

[0118] The default value is 0.03. The forward process of the diffusion model is defined as follows: At time step T Approximately obey ,in, This represents the conditional probability distribution of the forward process in the diffusion model. In the diffusion model Noisy samples at any given time, This represents the cumulative noise figure for the forward process in the diffusion model. It is an identity matrix. After statistical alignment... Although it matches the statistical properties of the style, it may deviate from the distribution assumptions made during the training of the diffusion model. The injection of LDR-noise will... Migrating from deterministic alignment to a sample point of a Gaussian distribution aligned by covariance enhances the diversity and detail recovery of the distribution while maintaining the effect of statistical alignment.

[0119] like Figure 2 As shown, S5: The content self-attention features and style self-attention features are fused through the Soft & Dynamic Injection inSelf-Attention (SDIS) module to obtain the fused attention features;

[0120] The soft + dynamic style injection module includes key-value soft injection and dynamic parameter adjustment;

[0121] Key-value soft injection includes:

[0122] S511: Obtain the sampling progress and normalize the sampling progress;

[0123] S512: Construct the mixing coefficients based on the normalized sampling progress;

[0124] The blending coefficient controls the intensity of style information injection;

[0125] As the sampling process linearly increases from a small initial value of the mixing coefficient to a larger final value, the expression for calculating the mixing coefficient is:

[0126]

[0127] in, for The mixing coefficient at time 10:00 This is the initial value for the mixing coefficient. This is the final value for the mixing coefficient. for Sampling progress after time-normalization;

[0128] , , Corresponding to the injection start time, The corresponding sampling end time.

[0129] In some specific embodiments of the present invention, the following are set , This design ensures that content is prioritized in the early stages of sampling, while gradually increasing the weight of style information as the sampling process progresses, thus avoiding the destructive impact of hard replacement on the content structure in the early stages.

[0130] S513: Based on the mixing coefficient, perform linear interpolation between the keys in the content self-attention feature and the keys in the style self-attention feature to obtain the keys for fused attention;

[0131] Linear interpolation is performed on the keys in the content branch and the keys in the style branch, and the expression for calculation is:

[0132]

[0133] in, for The key to constantly integrating attention for Attention key to content at any moment for The style attention key of the moment for The mixing coefficient at time;

[0134] S514: Based on the mixing coefficient, perform linear interpolation between the values ​​in the content self-attention feature and the values ​​in the style self-attention feature to obtain the value of fused attention;

[0135] Linear interpolation is performed on the values ​​in the content branch and the style branch, and the calculation expression is:

[0136]

[0137] in, for The value of attention is constantly integrated. for Attention value of content at any given moment for The style attention value at any given moment.

[0138] Linear interpolation is used to smoothly transition between key and value features of content and style, avoiding the sudden introduction of style information in the early stages.

[0139] Hard replacement is equivalent to performing discontinuous jump operations in the feature space: This discontinuity introduces distribution drift during diffusion sampling, leading to instability in the content structure. Soft injection, however, achieves a continuous transition through linear interpolation, forming a smooth path from content to style in the feature space, maintaining the continuity and stability of the sampling process. This idea echoes the progressive blending and latent calibration mechanisms proposed in the h-space injection method, but this invention applies progressive blending to the self-attention feature level, avoiding the distribution mismatch problem that may result from directly modifying latent variables.

[0140] Dynamic parameter scheduling: The query retention rate and attention temperature are designed as functions of time step t to achieve a smooth transition from structure-preserving to strong style.

[0141] Dynamic parameter adjustment includes:

[0142] S521: Use cosine scheduling to smoothly transition from a high initial value to a low termination value to construct the query retention rate hyperparameter;

[0143] The requirements for content preservation and style injection differ at different sampling stages. Early stages require stronger content preservation to stabilize the structure, while later stages can moderately reduce the retention rate to enhance style transfer. Therefore, the query retention rate hyperparameter... The function is designed as a time step, employing cosine scheduling to smoothly transition from a higher initial value to a lower final value. This is because cosine scheduling is smoother at transition boundaries compared to linear scheduling, helping to avoid visual artifacts caused by abrupt parameter changes. The calculation expression is as follows:

[0144]

[0145] in, To query the initial value of the retention rate hyperparameter, To query the termination value of the retention rate hyperparameter, for Sampling progress after time-normalization;

[0146] S522: Based on the intermediate results of the mixed content query and migration diffusion model using the query retention rate hyperparameter, obtain the final query in the self-attention process.

[0147] In self-attention computation, query features determine the spatial layout of the attention graph; therefore, maintaining query stability is crucial for preserving the content structure. This invention uses intermediate results from a hybrid content query and a migration diffusion model to achieve the final query in self-attention, and the computational expression is as follows:

[0148]

[0149] in, for The final query in self-attention at any given moment. for Query the retention rate hyperparameter at all times. To compute the projection layer of the query vector, These are latent variables in the diffusion model. Figure 2 ResbIk is the module for calculating the projection layer of the query vector.

[0150] The need for attention sharpening varies at different sampling stages. In the early stages, moderate sharpening is sufficient to maintain a clear content structure, while later stages require stronger sharpening to highlight stylistic and textural details. Therefore, this invention uses temperature scaling parameters... Extended to a linear scheduling function, this scheduling ensures stronger temperature scaling is applied later in the sampling process (low signal-to-noise ratio stage), synergizing with the increased style injection intensity. The temperature scaling parameter is constructed based on the normalized sampling progress, and its calculation expression is as follows:

[0151]

[0152] in, for Temperature scaling parameters at any given time This is the initial value for the temperature scaling parameter. This is the termination value for the temperature scaling parameter;

[0153] The attention map before softmax in the self-attention layer is magnified according to the temperature scaling parameter;

[0154] A temperature scaling parameter is added to the calculation of self-attention Atten. Enlarging the attention map before softmax can effectively sharpen the attention distribution; the calculation expression is as follows:

[0155]

[0156] in, To integrate attention features, For feature dimension, The softmax activation function is used. This is the transpose of the matrix;

[0157] Key-value soft injection and dynamic parameter adjustment and scheduling only take effect at designated attention layers within the diffusion sampling loop, and are applied by default to layers 6-11 of the decoder. These layers are mainly responsible for generating local texture details, which is highly consistent with the goals of style transfer. Combined with the second-order statistical alignment of the initial latent variables of LCR, the soft injection and dynamic scheduling mechanism form a complete style transfer process: LCR establishes a priori color and naturalness at the diffusion starting point, soft injection progressively transfers texture features during sampling, and dynamic scheduling ensures the optimal balance at different stages.

[0158] S6: During the backsampling of the calibrated latent variables, the fused attention features are progressively injected into the attention layer of the diffusion process at the injection starting point to perform diffusion denoising and generate a style transfer image.

[0159] LCR, SDIS, and adaptive injection timing work synergistically. The second-order statistical alignment of the initial latent variables establishes a good prior for the entire diffusion process, enabling subsequent feature injection to focus on texture and detail transfer, thus realizing the design concept of naturalness, color, and texture separation optimization.

[0160] In the style transfer framework of the diffusion model, the self-attention layer plays a crucial role in transferring texture features from the style image to the content image. The video generation task leverages self-attention (SA) to ensure consistency of subject matter, background, and action coherence between frames; it preserves details by injecting inverted key and value features from the style image, while retaining overall content through query features extracted from the content inverted image. The core idea of ​​the soft + dynamic scheduling mechanism's self-attention module is to extract key and value features from the self-attention layer of the style branch to adjust corresponding features in the stylization branch, thereby achieving effective transfer of high-level style attributes (such as texture). Linear interpolation facilitates a smooth transition between the key features of content and style, avoiding the sudden introduction of style information in the early stages. The key-value soft injection mechanism, given the content self-attention features and style self-attention features collected during the DDIM inversion process at each time step, achieves progressive style transfer through linear interpolation at the key-value feature tensor level.

[0161] Simply using the injection method mentioned in SDIS cannot fully guarantee the correct preservation of content structure. The timing of injection is also crucial for balancing content structure preservation and style transfer. Injecting too early will cause the content structure to be damaged by premature style information; injecting too late will result in insufficient style intensity and difficulty in fully transferring texture details. Existing methods usually use a fixed injection starting point, which limits the generalization ability of the method. In addition, multiple unstructured modifications to latent variables will cause distribution drift and loss of high-frequency details. Injecting quality enhancement noise at the end of sampling can effectively restore details, but gradual injection will disturb the stability of the sampling cycle. We adopt two complementary mechanisms: Automatic Localization and Latent Distribution Re-anchoring Module (AS-LDR). The combination of the two mechanisms ensures the stability of content structure in the high SNR stage (early sampling) and gradually introduces style information in the low SNR stage (late sampling), achieving the optimal balance between structure preservation and style intensity.

[0162] To verify the superiority of the present invention, quantitative comparative experiments were conducted with a variety of advanced algorithms, including six diffusion-based methods (StyleID, StyleAlign, InstantStyle, DiffuseIT, InST, DiffStyle) and three traditional style transfer methods (AesPA-Net, AdaAttN, AdaIN). The quantitative experimental results are shown in Table 1.

[0163] Table 1 Quantitative Experimental Results

[0164]

[0165] In Table 1, LPIPS is an image similarity metric measuring the perceptual level, FID is an index measuring the distribution difference between the generated image and the real image, and ArtFID is a variant of FID for artistic styles. "↓" indicates that the smaller the value, the better. This invention achieves the best score (0.4863) in content fidelity LPIPS, significantly outperforming other methods (StyleID: 0.5055, AesPA-Net: 0.5135, AdaAttN: 0.5439). This advantage is mainly attributed to the soft injection and dynamic scheduling mechanism of this invention. By progressively mixing key-value features instead of hard replacement, it avoids the destructive impact of style information on the content structure in the early sampling stage. The injection starting point is adaptively determined according to the edge complexity of the content image, further protecting images with rich structural details, thereby achieving excellent content structure preservation capabilities.

[0166] In terms of Style Fidelity Index (FID), this invention achieves 18.215, slightly lower than StyleID's 18.131 (the best). This difference mainly stems from the fact that StyleID and AdaIN are more aggressive in style transfer intensity, achieving better results in style feature matching. However, this aggressive style transfer often comes at the cost of content fidelity, resulting in a lower overall performance than this invention.

[0167] Finally, the overall style transfer quality was quantified by jointly evaluating Content Preservation Persistence (LPIPS) and Style Reproduction (FID). Although the present invention performs slightly worse than StyleID in FID, its significant advantage in LPIPS results in the optimal overall ArtFID metric, indicating that the present invention can achieve accurate style reproduction while maintaining the integrity of the content structure in style transfer tasks, providing better overall quality.

[0168] To verify the effectiveness of the proposed component, quantitative ablation experiments were conducted, and the quantitative results of the ablation experiments are shown in Table 2.

[0169] Table 2 Quantitative results of ablation experiments

[0170]

[0171] Table 2 shows that removing the LCR module (without LCR) increases both FID and ArtFID, indicating a significant decrease in style matching ability and overall quality. This demonstrates that LCR is crucial for capturing and transferring the complex color covariance between channels in style images. Removing the SDIS module (without SDIS) increases both FID and LPIPS, reflecting both the critical role of the SA module in injecting style and the importance of dynamic + soft injection in maintaining content structure. Hard or fixed injection strategies cause style information to prematurely or abruptly intrude into content features, leading to structural loss and instability. Therefore, it can be inferred that SDIS is the key to striking a balance between maintaining structural stability and achieving strong style transfer. Finally, removing the AS-LDR module (without AS-LDR) still results in worse performance than the complete model, but the regression is less than when SDIS is removed. This indicates that the main contribution of the AS-LDR mechanism lies in enhancing detail quality and diversity by adding small perturbations to the initial latent variables, thereby generating richer texture details.

[0172] Compared to diffusion-based methods (such as StyleID, CSGO, DiffuseIT, InST, etc.), these methods generally suffer from serious loss of content structure or unreasonable style transfer, or distort the content form (such as changing character features or scene structure), or the stylization effect is disconnected from the content (such as adding irrelevant elements or color imbalance). This invention, however, firmly maintains the essential structure of the content, accurately conveys the visual characteristics of the style, and the generated results show excellent performance in both structural fidelity and style integration.

[0173] Compared to traditional neural network-based methods (such as AdaAttn, AdaIN, and AesPA-Net), this invention excels in balancing content structure preservation and style feature transfer. Traditional methods often suffer from insufficient structure preservation or abrupt style integration, resulting in either loss of content details or jarring stylization effects. This invention can highly restore the core structure of the content (such as character expressions, scene outlines, and portrait features) while naturally and subtly integrating stylistic elements, achieving a harmonious unity between style and content.

[0174] like Figure 3 As shown, a diffusion style transfer system based on latent distribution re-anchoring dynamic injection is used to execute a diffusion style transfer method based on latent distribution re-anchoring dynamic injection, including:

[0175] The feature extraction module 101 acquires the content image and style image, and inverts the content image to the high-noise state of the latent space through the inverse diffusion process of the denoising diffusion implicit model to obtain the content latent variables and content self-attention features. The style image is inverted to the high-noise state of the latent space through the inverse diffusion process of the denoising diffusion implicit model to obtain the style latent variables and style self-attention features.

[0176] Alignment module 102 performs second-order statistical alignment of content latent variables and style latent variables using the latent covariance recoloring method to obtain initial mixed noise latent variables;

[0177] The automatic localization module 103 calculates the grayscale image of the content image using the Sobel gradient operator, calculates the edge intensity using the grayscale image of the content image, and automatically locates the style injection starting point based on the edge intensity.

[0178] The re-anchoring module 104 obtains the calibrated latent variables by injecting latent distributed re-anchoring noise into the initial mixed noise latent variables;

[0179] The soft + dynamic style injection module 105 fuses content self-attention features and style self-attention features to obtain fused attention features;

[0180] During the backsampling of the calibrated latent variables, the transfer generation module 106 progressively injects the fused attention features into the attention layer of the diffusion process at the injection starting point to perform diffusion denoising and generate a style transfer image.

[0181] Through the collaborative work of the above modules, progressive style injection avoids the destructive impact of style information on the content structure in the early sampling stage. The injection starting point is adaptively determined according to the edge complexity of the content image, thus protecting images with rich structural details. This achieves excellent content structure preservation capabilities while realizing accurate style reproduction.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A diffusion-style transfer method based on latent distribution re-anchoring dynamic injection, characterized in that, include: S1: Obtain the content image and style image. Through the denoising diffusion implicit model inverse diffusion process, the content image is inverted to the high-noise state of the latent space to obtain the content latent variables and content self-attention features. Through the denoising diffusion implicit model inverse diffusion process, the style image is inverted to the high-noise state of the latent space to obtain the style latent variables and style self-attention features. S2: The latent variables of content and style are second-order statistically aligned using the latent covariance recoloring method to obtain the initial mixed noise latent variables; S3: Calculate the grayscale image of the content image using the Sobel gradient operator, calculate the edge intensity using the grayscale image of the content image, and automatically locate the style injection starting point based on the edge intensity; S4: Obtain the calibrated latent variables by injecting latent distribution reanchoring noise into the initial mixed noise latent variables; S5: The content self-attention features and style self-attention features are fused through the soft + dynamic style injection module to obtain the fused attention features; The soft + dynamic style injection module includes key-value soft injection and dynamic parameter adjustment; Key-value soft injection includes: S511: Obtain the sampling progress and normalize the sampling progress; S512: Construct the mixing coefficients based on the normalized sampling progress; S513: Based on the mixing coefficient, perform linear interpolation between the keys in the content self-attention feature and the keys in the style self-attention feature to obtain the keys for fused attention; S514: Based on the mixing coefficient, perform linear interpolation between the values ​​in the content self-attention feature and the values ​​in the style self-attention feature to obtain the value of fused attention; Dynamic parameter adjustment includes: S521: Use cosine scheduling to smoothly transition from a high initial value to a low termination value to construct the query retention rate hyperparameter; S522: Based on the query retention rate hyperparameter, the intermediate results of the content query and the migration diffusion model are mixed to obtain the final query in the self-attention; S6: During the backsampling of the calibrated latent variables, the fused attention features are progressively injected into the attention layer of the diffusion process at the injection starting point to perform diffusion denoising and generate a style transfer image.

2. The diffusion style transfer method based on latent distribution re-anchoring dynamic injection according to claim 1, characterized in that, Step S2 includes: S21: Flatten the implicit variables of content and style in the spatial dimension; S22: Calculate the decentralized characteristics of the flattened content latent variables and style latent variables; S23: Calculate the content covariance matrix based on the decentralized characteristics of the flattened content latent variables, and calculate the style covariance matrix based on the decentralized characteristics of the flattened style latent variables. S24: Perform eigenvalue decomposition on the content covariance matrix and style covariance matrix to obtain content eigenvalues ​​and style eigenvalues; S25: Construct a whitening matrix based on content feature values, and a coloring matrix based on style feature values; S26: Construct a latent covariance recoloring transformation matrix based on the whitening matrix and the coloring matrix. Align the content latent variables and style latent variables with second-order statistics based on the latent covariance recoloring transformation matrix to obtain the initial mixed noise latent variables.

3. The diffusion style transfer method based on latent distribution re-anchoring dynamic injection according to claim 1, characterized in that, Step S3 includes: S31: Normalize the content image and calculate the grayscale image of the normalized content image; S32: Calculate the horizontal and vertical gradient maps of the normalized grayscale image of the content image using the Sobel gradient operator; S33: Calculate the edge strength based on the gradient map in the horizontal direction and the gradient map in the vertical direction; S34: Normalize the edge intensity and linearly map it to the injection start point interval to obtain the injection start point.

4. The diffusion style transfer method based on latent distribution re-anchoring dynamic injection according to claim 1, characterized in that, The expression for calculating the calibrated latent variables is: in, for Latent variables after time-calibration for Latent variables in the time-diffusion model For noise intensity, For latent distribution re-anchoring noise, It follows a standard normal distribution.

5. The diffusion style transfer method based on latent distribution re-anchoring dynamic injection according to claim 1, characterized in that, Dynamic parameter adjustment also includes: Temperature scaling parameters are constructed based on the normalized sampling progress; The attention map before softmax in the self-attention layer is magnified based on the temperature scaling parameter.

6. The diffusion style transfer method based on latent distribution re-anchoring dynamic injection according to claim 1, characterized in that, The key-value soft injection and the dynamic parameter adjustment only take effect in the N attention layers within the diffusion sampling loop.

7. A diffusion style transfer system based on latent distribution re-anchoring dynamic injection, characterized in that, To perform a diffusion style transfer method based on latent distribution re-anchoring dynamic injection as described in any one of claims 1 to 6, comprising: The feature extraction module acquires content images and style images. Through the denoising diffusion implicit model inverse diffusion process, the content image is inverted to a high-noise state in the latent space to obtain content latent variables and content self-attention features. Through the denoising diffusion implicit model inverse diffusion process, the style image is inverted to a high-noise state in the latent space to obtain style latent variables and style self-attention features. The alignment module performs second-order statistical alignment of content latent variables and style latent variables using the latent covariance recoloring method to obtain the initial mixed noise latent variables. An automatic positioning module calculates the grayscale image of the content image using the Sobel gradient operator, calculates the edge intensity using the grayscale image of the content image, and automatically locates the style injection starting point based on the edge intensity. The re-anchoring module obtains the calibrated latent variables by injecting latent distribution re-anchoring noise into the initial mixed noise latent variables; The soft + dynamic style injection module fuses content self-attention features and style self-attention features to obtain fused attention features. The transfer generation module, during the backsampling of the calibrated latent variables, progressively injects the fused attention features into the attention layer of the diffusion process at the injection starting point to perform diffusion denoising and generate a style transfer image.

Images