Artistic style transfer method and system based on frequency domain modulation and attention guidance

Abstract

The present application belongs to the technical field of image processing, and particularly relates to an artistic style transfer method and system based on frequency domain modulation and attention guidance, aiming at the difficulty of effectively separating content structure and style features in the prior art, which easily leads to structural damage or insufficient stylization of the generated results. The present application first initializes the strategy based on structure perception frequency domain modulation, effectively decouples content and style from the frequency domain; constructs a decoupling cross-attention mechanism, processes text, content and style features through parallel channels respectively, thereby avoiding feature interference while precisely regulating the generation process; and uses an attention-driven guidance mechanism to model the style transfer as an iterative optimization problem in the latent space, dynamically calibrates the feature distribution through gradient guidance, and effectively alleviates the feature dilution problem in the deep network.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an art style transfer method and system based on frequency domain modulation and attention guidance. Background Technology

[0002] The core objective of art style transfer is to make the visual appearance of the source image reflect the artistic style of the reference image while strictly preserving the content structure of the source image. In the field of image generation, latent diffusion models, represented by Stable Diffusion XL (SDXL), have become the mainstream method due to their excellent ability to generate complex textures. However, how to decouple and re-integrate content structure and style texture within a unified framework remains a key issue in this field.

[0003] Existing training-free methods achieve style transfer by intervening in the self-attention mechanism during the inference phase. These methods perform well in general scenarios, but when dealing with high-resolution tasks or tasks with large style differences, they still face structural limitations in two key stages: the generation process and the sampling initiation point. 1. Feature dilution problem in the generation process: Traditional key-value injection methods replace the key and value of the self-attention layer with features of the reference style image, which performs well in most scenarios. However, from the perspective of feature propagation mechanism, the backbone network of diffusion models generally uses residual connections. This structure preserves the main transmission path for the original content, allowing content features to be directly forwarded through identity mapping, while the injected style signal only acts on the residual branches. This asymmetric structure causes the residual branches to continuously suppress style information, making it difficult for the generated image to reproduce the deep texture of the reference image, often resulting in insufficient stylization.

[0004] 2. Initial Feature Coupling at the Sampling Start Point: The denoising process in the diffusion model depends on the initial state of the latent variables. Standard methods typically use DDIM inversion to obtain the initial noise, but this process couples the high-frequency structure and low-frequency style information of the source image. This source style prior, which is not decoupled during the initialization phase, will strongly conflict with the target style in subsequent generation, resulting in obvious visual artifacts in the generated image. Summary of the Invention

[0005] To address the problems in the background technology, this invention provides an art style transfer method and system based on frequency domain modulation and attention guidance.

[0006] The technical solution of the present invention is as follows: This invention provides a method for art style transfer based on frequency domain modulation and attention guidance, comprising: S1: The image to be processed is compressed and mapped to a low-dimensional latent space. The initial noise-free latent variables are then subjected to DDIM inversion operation to obtain the initial latent variables. S2: Using a two-dimensional fast Fourier transform, the initial latent variables are transformed from the spatial domain to the frequency domain, separating the high-frequency structure component and the low-frequency style component; the low-frequency style component is suppressed according to a set ratio, while the high-frequency structure component is retained, and then transformed back to the spatial domain through an inverse Fourier transform to obtain the initially purified latent variables. Based on the initially purified latent variables, iterative denoising is performed to generate the current latent variables to be optimized. S3: Based on the content image to be processed and the style reference image, extract text features, content features and style features. For each network layer, construct three parallel cross-attention branches. After independently injecting the text features, content features and style features, respectively, the calculation results of the three branches are weighted and fused to obtain the current attention features. S4: Construct style anchors using content query vectors and style key-value vectors; calculate the difference between the current attention features and style anchors to obtain style guidance energy; backpropagate the gradient of style guidance energy back to the current latent variable to be optimized to generate standard latent variables.

[0007] Based on the above-described art style transfer method based on frequency domain modulation and attention guidance, S3 constructs three parallel cross-attention branches for each network layer, which independently inject text features, content features, and style features, respectively. Specifically: For the text attention branch of each network layer, the text features are projected to obtain text key and value vectors, and then cross-attention processing is performed to obtain text attention features. For each network layer's content attention branch, after projecting the content features to obtain content key and value vectors, the content attention features are obtained through cross-attention processing. For each network layer's style attention branch, style features are projected to obtain style keys and value vectors, which are then processed through cross-attention to obtain style attention features.

[0008] Based on the above-described art style transfer method based on frequency domain modulation and attention guidance, step S4 involves backpropagating the gradient of the style guidance energy back to the current latent variable to be optimized, generating a standard latent variable, specifically as follows: The style-guided energy is backpropagated relative to the gradient of the current latent variable to be optimized to obtain the gradient update direction and magnitude of the current latent variable to be optimized. The current latent variable to be optimized is adaptively updated along the gradient descent direction to complete the gradient calibration of the current latent variable to be optimized.

[0009] Based on the above-described art style transfer method based on frequency domain modulation and attention guidance, step S4 describes constructing style anchors using content query vectors and style key-value vectors, which are then used to transfer art styles through formulas. To achieve; In the formula, , , , , These represent style anchors, query matrices, style key vectors, style value vectors, and self-attention functions, respectively.

[0010] Based on the above-described art style transfer method based on frequency domain modulation and attention guidance, step S4 calculates the difference between the current attention feature and the style anchor point to obtain the style guidance energy, which is obtained through the formula: To achieve; In the formula, Style indicates that energy is guided; Indicates the current attention characteristics; Indicates style anchor points; This represents the L1 norm.

[0011] Based on the above-described art style transfer method based on frequency domain modulation and attention guidance, after obtaining the initially purified latent variables, S2 further uses the formula: To inject complementary Gaussian noise; In the formula, , These represent the latent variables after the injection of complementary Gaussian noise and the latent variables after preliminary purification, respectively. Represents the frequency domain modulation coefficients; This represents Gaussian noise with a mean of 0 and a variance of σ².

[0012] Based on the above-described art style transfer method based on frequency domain modulation and attention guidance, S4 also includes a hierarchical asymmetric feature injection operation. To adopt a hierarchical injection strategy, only content features are injected in the downsampling layer and only style features are injected in the upsampling layer.

[0013] This invention also provides an art style transfer system based on frequency domain modulation and attention guidance, comprising: DDIM Inversion Module: The image to be processed is compressed and mapped to a low-dimensional latent space. The resulting initial noise-free latent variables are then subjected to DDIM inversion to obtain the initial latent variables. Frequency domain modulation module: Using two-dimensional fast Fourier transform, the initial latent variables are transformed from the spatial domain to the frequency domain, separating the high-frequency structure component and the low-frequency style component; the low-frequency style component is suppressed according to a set ratio, the high-frequency structure component is retained, and then transformed back to the spatial domain through inverse Fourier transform to obtain the initially purified latent variables; based on the initially purified latent variables, iterative noise reduction is performed to generate the current latent variables to be optimized. Decoupled cross-attention module: Based on the content image to be processed and the style reference image, extract text features, content features and style features. Each network layer constructs three parallel cross-attention branches. After independently injecting the text features, content features and style features, the calculation results of the three branches are weighted and fused to obtain the current attention features. Attention-driven guidance module: Constructs style anchors using content query vectors and style key-value vectors; calculates the difference between the current attention features and style anchors to obtain style guidance energy; backpropagates the gradient of style guidance energy back to the current latent variable to be optimized to generate standard latent variables.

[0014] Based on the above-described art style transfer system based on frequency domain modulation and attention guidance, in the decoupled cross-attention module, each network layer constructs three parallel cross-attention branches, which independently inject text features, content features, and style features, respectively. Specifically: For the text attention branch of each network layer, the text features are projected to obtain text key and value vectors, and then cross-attention processing is performed to obtain text attention features. For each network layer's content attention branch, after projecting the content features to obtain content key and value vectors, the content attention features are obtained through cross-attention processing. For each network layer's style attention branch, style features are projected to obtain style keys and value vectors, which are then processed through cross-attention to obtain style attention features.

[0015] Based on the above-described art style transfer system based on frequency domain modulation and attention guidance, in the attention-driven guidance module, the step of backpropagating the gradient of style guidance energy back to the current latent variable to be optimized to generate standard latent variables specifically involves: The style-guided energy is backpropagated relative to the gradient of the current latent variable to be optimized to obtain the gradient update direction and magnitude of the current latent variable to be optimized. The current latent variable to be optimized is adaptively updated along the gradient descent direction to complete the gradient calibration of the current latent variable to be optimized.

[0016] Beneficial effects: To address the difficulty of effectively separating content structure and style features in existing methods, which can easily lead to structural damage or insufficient stylization in the generated results, this invention firstly achieves effective decoupling of content and style in the frequency domain based on a structure-aware frequency domain modulation initialization strategy; it then constructs a decoupled cross-attention mechanism, which processes text, content, and style features separately through parallel channels, thereby achieving precise control over the generation process while avoiding feature interference; finally, it utilizes an attention-driven guidance mechanism to model style transfer as an iterative optimization problem in the latent space, and effectively alleviates the feature dilution problem in deep networks by dynamically calibrating the feature distribution through gradient guidance. Attached Figure Description

[0017] Figure 1 The present invention provides style transfer results for an art style transfer method based on frequency domain modulation and attention guidance in different scenarios.

[0018] Figure 2 The results show a qualitative comparison of different art style transfer methods, where a is the image to be processed, b is the style reference image, c is the method of this application, d is Attention Distillation, e is AttenST, f is CSGO, g is DiffuseST, h is InstantStyle, i is StyleID, and j is Styleshot. Detailed Implementation

[0019] The following examples are intended to illustrate the present invention, and not to further limit the invention.

[0020] Example 1

[0021] This embodiment provides an art style transfer method based on frequency domain modulation and attention guidance, including: S1: The image to be processed is compressed and mapped to a low-dimensional latent space. The initial noise-free latent variables are then subjected to DDIM inversion to obtain the initial latent variables.

[0022] S2: Using a two-dimensional fast Fourier transform, the initial latent variables are transformed from the spatial domain to the frequency domain, separating the high-frequency structure component and the low-frequency style component; the low-frequency style component is suppressed according to a set ratio, while the high-frequency structure component is retained, and then transformed back to the spatial domain through an inverse Fourier transform to obtain the initially purified latent variables. Based on the initially purified latent variables, iterative denoising is performed to generate the current latent variables to be optimized.

[0023] Existing research indicates that the latent space of diffusion models exhibits significant frequency bias, with low-frequency components primarily encoding image hue and carrying strong source style priors during inversion. If the latent variables from standard inversion are directly used as the starting point for generation, the residual low-frequency information from the source image will fiercely compete with the target style, leading to artifacts and aliasing.

[0024] To address this, this invention proposes a structure-aware frequency-domain modulation method aimed at purifying the generation process from its source. The core strategy is to suppress low-frequency style interference in the source image while preserving high-frequency structural cues.

[0025] To achieve decoupled frequency control in the latent space, a frequency domain processing framework was constructed using two-dimensional fast Fourier transform and its inverse transform.

[0026] In the specific implementation process, the initial latent variables are first mapped to the frequency domain to obtain the spectrum. Then a low-pass filter was used. and its complementary high-pass filter The spectrum is precisely separated into two frequency bands: high frequency and low frequency, namely the low frequency style component. High-frequency structural components To quantitatively control the information intensity of different frequency bands, this invention also defines frequency domain modulation coefficients. , To construct an asymmetric modulation strategy.

[0027] The low-frequency suppression strategy is achieved through the formula: To achieve this.

[0028] High-frequency suppression strategy is achieved through the formula: To achieve this.

[0029] pass The element-wise multiplication operation linearly attenuates the low-frequency components while maintaining the integrity of the high-frequency components.

[0030] This mechanism allows for adjustments The size of the value quantitatively controls the degree to which information in a specific frequency band is retained.

[0031] To maintain the statistical distribution of the latent space and introduce generated diversity, preferably, after obtaining the initially purified latent variables, the following formula is also used: To inject complementary Gaussian noise; In the formula, , These represent the latent variables after the injection of complementary Gaussian noise and the latent variables after preliminary purification, respectively. Represents the frequency domain modulation coefficients, used to control the degree of low-frequency information retention; This represents Gaussian noise with a mean of 0 and a variance of σ².

[0032] This invention employs a structure-aware frequency-domain modulation method, utilizing Fast Fourier Transform (FFT) to asymmetrically decouple the inversion latent variables in the frequency domain. This significantly suppresses low-frequency style components of the original image while precisely preserving its high-frequency structural information, generating a new starting point. The low-frequency style prior is effectively suppressed, while the high-frequency structural information is retained, eliminating the need for subsequent denoising steps to counteract the source image texture. This significantly improves style transfer quality, effectively removes potential interference from low-frequency tones in the source image, ensures the purity of visual expression, and simultaneously enhances the efficiency of style texture injection and color representation.

[0033] S3: Based on the content image to be processed and the style reference image, extract text features, content features and style features. For each network layer, construct three parallel cross-attention branches. After independently injecting the text features, content features and style features, respectively, the calculation results of the three branches are weighted and fused to obtain the current attention features.

[0034] In latent diffusion models, such as the standard Stable Diffusion XL (SDXL), image generation is primarily achieved through cross-attention between image queries and text features (as keys and values) within the denoising network UNet. This single text-guided approach relies solely on text, cannot directly incorporate visual image features, and struggles to precisely control the content structure and texture during style transfer.

[0035] To better balance content fidelity and stylization effects in multimodal scenarios, this invention proposes a decoupled cross-attention mechanism.

[0036] In the specific implementation process, firstly, in order to enhance visual control, the present invention utilizes a pre-trained CLIP image encoder to extract content features and style features respectively. These features are highly aligned semantically and can effectively represent content structure information and style texture elements.

[0037] Based on this, each network layer of the present invention constructs three parallel cross-attention branches, which independently inject text features, content features, and style features, respectively, as follows: For the text attention branch of each network layer, the text features are projected to obtain text key and value vectors, and then cross-attention processing is performed to obtain text attention features. For each network layer's content attention branch, after projecting the content features to obtain content key and value vectors, the content attention features are obtained through cross-attention processing. For each network layer's style attention branch, style features are projected to obtain style keys and value vectors, which are then processed through cross-attention to obtain style attention features.

[0038] In each layer of UNet, these three decoupled attention paths are weighted and fused to obtain the current attention features that ultimately guide denoising.

[0039] This invention employs parallel attention branches to process features extracted from text, style, and content separately, effectively preventing interference between features while ensuring precise control of the generation process.

[0040] S4: Construct style anchors using content query vectors and style key-value vectors; calculate the difference between the current attention features and style anchors to obtain style guidance energy; backpropagate the gradient of style guidance energy back to the current latent variable to be optimized to generate standard latent variables.

[0041] Traditional key-value (KV) injection methods achieve training-free style transfer by replacing the key and value of the self-attention layer with features from a style reference image. However, this simple replacement strategy is prone to failure in deep networks. Because the backbone of a diffusion model extensively employs residual connections, in this structure, the original content features are directly passed through the identity mapping and dominate; while the injected style features are only superimposed on the residual branches. Therefore, as features propagate forward through ResNet blocks, the injected features are often diluted by the original content features, resulting in insufficient texture representation.

[0042] To address the feature dilution problem caused by residual structure, this invention proposes an active attention-driven guidance mechanism.

[0043] Unlike passive feature replacement, this invention models the stylization process as an iterative optimization problem in the latent space. By utilizing gradient information to dynamically calibrate features, the feature dilution problem is alleviated, thereby achieving effective fusion of style and content.

[0044] The core of this mechanism lies in constructing a style-guided energy function that can measure the difference between the current generated state and the ideal stylized state.

[0045] In the specific implementation process, based on the UNet architecture of the pre-trained SDXL model, a style anchor with a consistent structure is first constructed. Preferably, the style anchor is constructed using content query vectors and style key-value vectors, which is achieved through the formula... To achieve this.

[0046] In the formula, , , , , These represent style anchors, query matrices, style key vectors, style value vectors, and self-attention functions, respectively.

[0047] The query matrix is ​​obtained by linear projection of the current UNet intermediate layer features, and originates from the current state of the image to be processed during the denoising process.

[0048] The style key vector is obtained by linear projection of the style reference image extracted by the CLIP image encoder, and is used for style feature matching.

[0049] The style value vector is obtained by linear projection of the style reference image after extraction by the CLIP image encoder, and it carries the texture details of the style image.

[0050] Style anchors represent idealized stylistic features while maintaining the semantics of the content, serving as a target reference to guide the energy function.

[0051] This style anchor represents the feature state of incorporating reference style textures while maintaining the semantic framework of the target content.

[0052] Based on this, the difference between the current attention features and the style anchor point is calculated to obtain the style guidance energy, which is obtained through the formula: To achieve this.

[0053] In the formula, This represents style-guided energy, a scalar value used to quantify the difference between the current attentional features and the style anchor point; Indicates the current attention characteristics; Indicates style anchor points; This represents the L1 norm.

[0054] The style-guided energy function quantifies the degree to which the current latent variables deviate from the target style, and the gradient it generates indicates the optimal direction of stylization evolution, thus providing clear optimization guidance for the denoising process.

[0055] To mitigate the instability caused by direct interference from high-frequency noise during inference, this invention employs a more robust pre-sampling, post-calibration decoupling strategy. Within the DDIM sampling framework, this invention treats the approximate value of the original latent variable predicted by the denoising network at each time step as an optimizable parameter surrogate. After obtaining this intermediate state, instead of immediately proceeding to the next sampling step, an attention-driven guidance mechanism is applied to perform instantaneous gradient calibration.

[0056] Preferably, the step of backpropagating the gradient of the style-guided energy back to the current latent variable to be optimized to generate standard latent variables specifically involves: The style-guided energy is backpropagated relative to the gradient of the current latent variable to be optimized to obtain the gradient update direction and magnitude of the current latent variable to be optimized. The current latent variable to be optimized is adaptively updated along the gradient descent direction to complete the gradient calibration of the current latent variable to be optimized.

[0057] This invention utilizes the Adam optimizer to adaptively estimate the first and second moments of the gradient to automatically manage the update magnitude, replacing the fixed gradient step size. The generated standard latent variables participate in the subsequent DDIM denoising step, ensuring that the gradient signal can be directly propagated back to the backbone distribution. This effectively solves the feature dilution problem in forward propagation and achieves progressive and accurate style alignment.

[0058] In addition, existing research indicates that different layers of UNet have a clear semantic division of labor: the downsampling path is mainly responsible for extracting the abstract structural layout and semantic skeleton, while the upsampling path focuses on restoring high-resolution textures.

[0059] Based on this characteristic, the present invention also includes a hierarchical asymmetric feature injection operation. In order to adopt a hierarchical injection strategy, only content features are injected in the downsampling layer to stabilize the generated spatial structure; and only style features are injected in the upsampling layer, thereby achieving effective separation of content structure and style features and completing high-quality style transfer.

[0060] Although the aforementioned structure-aware frequency domain modulation effectively decouples the source image texture from the target style at the feature level, providing a clean feature starting point for the generation process, slight semantic drift may still occur in the generation trajectory during high-intensity stylization denoising.

[0061] To enhance the robustness of structural alignment, this invention introduces Tile ControlNet as an explicit spatial hard constraint. It directly utilizes the image of the content to be processed as a dense conditional input, applying a local semantic consistency constraint through multi-scale convolutional feature resampling. This mechanism allows the model to maintain strong alignment with the source content in both overall layout and local semantics, while ignoring high-frequency texture noise.

[0062] To address the difficulty of effectively separating content structure and style features in existing methods, which can easily lead to structural damage or insufficient stylization in the generated results, this invention firstly implements a structure-aware frequency domain modulation initialization strategy to effectively decouple content and style in the frequency domain. It then constructs a decoupled cross-attention mechanism, processing text, content, and style features separately through parallel channels, thereby achieving precise control over the generation process while avoiding feature interference. Finally, it utilizes an attention-driven guidance mechanism to model style transfer as an iterative optimization problem within the latent space, dynamically calibrating feature distribution through gradient guidance, effectively alleviating the feature dilution problem in deep networks.

[0063] Example 2 Based on Embodiment 1, the present invention also provides an art style transfer system based on frequency domain modulation and attention guidance, comprising: DDIM Inversion Module: The image to be processed is compressed and mapped to a low-dimensional latent space. The resulting initial noise-free latent variables are then subjected to DDIM inversion to obtain the initial latent variables. Frequency domain modulation module: Using two-dimensional fast Fourier transform, the initial latent variables are transformed from the spatial domain to the frequency domain, separating the high-frequency structure component and the low-frequency style component; the low-frequency style component is suppressed according to a set ratio, the high-frequency structure component is retained, and then transformed back to the spatial domain through inverse Fourier transform to obtain the initially purified latent variables; based on the initially purified latent variables, iterative noise reduction is performed to generate the current latent variables to be optimized. Decoupled cross-attention module: Based on the content image to be processed and the style reference image, extract text features, content features and style features. Each network layer constructs three parallel cross-attention branches. After independently injecting the text features, content features and style features, the calculation results of the three branches are weighted and fused to obtain the current attention features. Attention-driven guidance module: Constructs style anchors using content query vectors and style key-value vectors; calculates the difference between the current attention features and style anchors to obtain style guidance energy; backpropagates the gradient of style guidance energy back to the current latent variable to be optimized to generate standard latent variables.

[0064] Based on the above-described art style transfer system based on frequency domain modulation and attention guidance, in the decoupled cross-attention module, each network layer constructs three parallel cross-attention branches, which independently inject text features, content features, and style features, respectively. Specifically: For the text attention branch of each network layer, the text features are projected to obtain text key and value vectors, and then cross-attention processing is performed to obtain text attention features. For each network layer's content attention branch, after projecting the content features to obtain content key and value vectors, the content attention features are obtained through cross-attention processing. For each network layer's style attention branch, style features are projected to obtain style keys and value vectors, which are then processed through cross-attention to obtain style attention features.

[0065] Based on the above-described art style transfer system based on frequency domain modulation and attention guidance, in the attention-driven guidance module, the step of backpropagating the gradient of style guidance energy back to the current latent variable to be optimized to generate standard latent variables specifically involves: The style-guided energy is backpropagated relative to the gradient of the current latent variable to be optimized to obtain the gradient update direction and magnitude of the current latent variable to be optimized. The current latent variable to be optimized is adaptively updated along the gradient descent direction to complete the gradient calibration of the current latent variable to be optimized.

[0066] experiment 1. Experimental setup The method provided in this invention is based on a pre-trained Stable Diffusion XL (SDXL) model. For the attention-driven guidance module, the number of optimization iterations is set to 1, and the learning rate is set to 0.02. In the frequency domain modulation module, a Gaussian low-pass filter is used to define the frequency mask, and the frequency domain modulation coefficients... The default setting is 0.7. All experiments were conducted on a single vGPU-32GB (32GB) GPU.

[0067] For the datasets, content images were sampled from the MS-COCO dataset, while style images were selected from the WikiArt dataset. For quantitative comparison, 800 stylized images were generated by randomly selecting content and style images from each dataset.

[0068] 2. Qualitative Results Depend on Figure 1 As can be seen, the method in this application demonstrates good generalization performance for images with complex content in different scenarios. Experimental results show that the method in this application can not only achieve subtle artistic style transfer, but also maintain the structural layout of the source image during the high-stylization process, generating high-quality results with semantic consistency and visual harmony.

[0069] 3. Comparison with existing methods This application's method is comprehensively compared with state-of-the-art diffusion-based style transfer methods. Based on how style information is integrated into the generation process, these baseline methods are categorized into two types: self-attention-based methods (StyleID, DiffuseST, AttenST, Attention Distillation), which primarily align content and style by modulating internal spatial features; and cross-attention-based methods (InstantStyle, Styleshot, CSGO), which typically inject style through contextual interaction or decoupled adapters. The baseline methods employ their recommended configurations.

[0070] The following is a brief introduction to the existing methods described above.

[0071] StyleID: Training-free, achieving style transfer through self-attention key replacement and latent space AdaIN. Compared to the method in this application, StyleID has no frequency domain modulation, no attention-driven guidance, only static feature replacement, is susceptible to feature dilution due to residual structure, and has style coupling during initialization.

[0072] DiffuseST: Training-free, it integrates text embeddings and spatial features, injecting separate content and style features, and extracting intermediate features through DDIM inversion. Compared to the method in this application, DiffuseST lacks a frequency domain modulation purification generation starting point and attention-driven guidance, relying on static decoupling injection, and style features in deep networks are still suppressed.

[0073] AttenST: Training-free, it aligns visual features of content and style through self-attention key-value swapping. Compared to the method in this application, AttenST lacks frequency domain modulation, decoupled cross-attention, and attention-driven guidance, relying solely on static attention modulation, resulting in insufficient stylization depth.

[0074] Attention Distillation: Constructs an attention distillation loss, optimizes it through backpropagation in the latent space, and aligns it with the ideal and current stylization feature distribution. Compared to the method in this application, Attention Distillation has no frequency modulation domain, no decoupled cross attention, and only distills static attention features, resulting in shallow style texture and weak structural robustness.

[0075] InstantStyle: CLIP feature subtraction decouples style, injecting only style features into the UNet upsampled style-specific layer. Compared to the method in this application, InstantStyle lacks a frequency modulation domain, which eliminates low-frequency suppression to remove source style interference. It also lacks attention-driven guidance, and hard injection is dominated by residual connections, resulting in significant stylization deficiencies.

[0076] Styleshot trains a dedicated style-aware encoder and a content fusion encoder, injecting style features through parallel cross-attention. Compared to the method in this application, Styleshot lacks a frequency domain modulation sanitization initialization starting point and attention-driven guidance, relies on a pre-trained encoder, and has insufficient depth in style and content fusion.

[0077] CSGO explicitly decouples content and style using independent feature channels, combines ControlNet for spatial constraints, and performs end-to-end training and optimization. Compared to the method in this application, CSGO lacks a frequency modulation domain and attention-driven guidance, relying solely on static feature separation, which cannot alleviate the feature dilution problem caused by the residual network.

[0078] 3.1 Quantitative comparison In the quantitative evaluation, LPIPS, FID, and ArtFID were selected as core evaluation metrics to quantify model performance from different dimensions. Specifically, LPIPS was used to measure the degree of content structure preservation between the stylized image and its corresponding content image; FID was used to evaluate the fidelity of style features between the stylized image and its corresponding style image. Furthermore, to comprehensively consider both performance metrics, ArtFID was also introduced, defined as... This metric considers both content and style retention simultaneously and aligns closely with human judgment. In assessing style similarity, the CSD metric is used; a higher value indicates better style transfer. Additionally, a user rating (User) is generated through a survey of 100 people's subjective preferences; higher scores indicate stronger user preferences.

[0079] Table 1 presents the results of each quantitative assessment.

[0080] Table 1 Quantitative Assessment Results

[0081] As shown in Table 1, compared with other methods, the method of this application achieves a better balance between content fidelity and style alignment. Specifically, the model corresponding to the method of this invention achieved the lowest FID score, indicating that the generated texture is closest to the target style in terms of distribution; at the same time, the LPIPS of the model corresponding to the method of this invention also reached the optimal level, indicating that the content structure was effectively preserved. Therefore, the method of this application obtained the best ArtFID comprehensive score, verifying its superior balance between content preservation and style transfer.

[0082] 3.2 Qualitative Comparison Depend on Figure 2 As shown, most existing methods struggle to balance structure preservation and style transfer intensity. For example, attention-guided methods (such as Attention Distillation and AttenST) often suffer from insufficient stylization and content leakage, producing results that are typically just simple color transformations, failing to fully capture artistic textures. Conversely, strong injection methods based on cross-attention (such as StyleShot and InstantStyle), while offering rich texture representation, are prone to structural collapse and semantic loss due to insufficient spatial constraints. In contrast, the method in this application effectively solves this core challenge, achieving deep style and semantic transfer while accurately preserving the structural layout of the source image. The generated results possess both realistic artistic brushstrokes and clear visual semantics.

[0083] 4. Ablation test To verify the effectiveness of each core module, ablation experiments were conducted, and the results are shown in Table 2. First, when the frequency domain modulation module was removed, the model degenerated to initialization using standard DDIM inversion. Due to the failure to decouple the high-frequency and low-frequency information of the source image, the generated image exhibited significant style conflicts and high-frequency artifacts, confirming the module's crucial role in mitigating structure-style entanglement.

[0084] If the attention-driven guidance module is removed, the generated result is merely a simple color transfer, with insufficiently distinct brushstrokes. In contrast, this invention extracts texture features, achieving a natural integration of style and content.

[0085] The complete model of this application achieved the lowest ArtFID score (48.12), indicating that the modules work together to better balance content structure and style texture.

[0086] Table 2 Ablation Experiment Results

[0087] The above experiments show that the method in this application outperforms existing methods in terms of LPIPS, FID and ArtFID, and can achieve high-quality output with only a single iteration, effectively balancing structure preservation and stylization intensity.

Claims

1. A method for art style transfer based on frequency domain modulation and attention guidance, characterized in that, include: S1: The image to be processed is compressed and mapped to a low-dimensional latent space. The initial noise-free latent variables are then subjected to DDIM inversion operation to obtain the initial latent variables. S2: Using a two-dimensional fast Fourier transform, the initial latent variables are transformed from the spatial domain to the frequency domain, separating the high-frequency structure component and the low-frequency style component; the low-frequency style component is suppressed according to a set ratio, while the high-frequency structure component is retained, and then transformed back to the spatial domain through an inverse Fourier transform to obtain the initially purified latent variables. Based on the initially purified latent variables, iterative denoising is performed to generate the current latent variables to be optimized. S3: Based on the content image to be processed and the style reference image, extract text features, content features and style features. For each network layer, construct three parallel cross-attention branches. After independently injecting the text features, content features and style features, respectively, the calculation results of the three branches are weighted and fused to obtain the current attention features. Each network layer constructs three parallel cross-attention branches, which independently inject text features, content features, and style features, respectively. Specifically: For the text attention branch of each network layer, the text features are projected to obtain text key and value vectors, and then cross-attention processing is performed to obtain text attention features. For each network layer's content attention branch, after projecting the content features to obtain content key and value vectors, the content attention features are obtained through cross-attention processing. For each network layer's style attention branch, style features are projected to obtain style keys and value vectors, and then cross-attention processing is used to obtain style attention features. S4: Construct style anchors using content query vectors and style key-value vectors; Calculate the difference between the current attention feature and the style anchor point to obtain the style-guided energy; backpropagate the gradient of the style-guided energy back to the current latent variable to be optimized to generate standard latent variables, specifically: The style-guided energy is backpropagated relative to the gradient of the current latent variable to be optimized to obtain the gradient update direction and magnitude of the current latent variable to be optimized. The current latent variable to be optimized is adaptively updated along the gradient descent direction to complete the gradient calibration of the current latent variable to be optimized.

2. The art style transfer method based on frequency domain modulation and attention guidance according to claim 1, characterized in that, S4 describes constructing style anchors using content query vectors and style key-value vectors, which are then used to formulate... To achieve; In the formula, , , , , These represent style anchors, query matrices, style key vectors, style value vectors, and self-attention functions, respectively.

3. The art style transfer method based on frequency domain modulation and attention guidance according to claim 1, characterized in that, S4 describes calculating the difference between the current attention feature and the style anchor point to obtain the style guidance energy, which is obtained through the formula: To achieve; In the formula, Style indicates that energy is guided; Indicates the current attention characteristics; Indicates style anchor points; This represents the L1 norm.

4. The art style transfer method based on frequency domain modulation and attention guidance according to claim 1, characterized in that, After obtaining the preliminary purified latent variables, S2 also uses the formula: To inject complementary Gaussian noise; In the formula, , These represent the latent variables after the injection of complementary Gaussian noise and the latent variables after preliminary purification, respectively. Represents the frequency domain modulation coefficients; This represents Gaussian noise with a mean of 0 and a variance of σ².

5. The art style transfer method based on frequency domain modulation and attention guidance according to claim 1, characterized in that, S4 is followed by a hierarchical asymmetric feature injection operation, which employs a hierarchical injection strategy, injecting only content features in the downsampling layer and only style features in the upsampling layer.

6. An art style transfer system based on frequency domain modulation and attention guidance, characterized in that, include: DDIM Inversion Module: The image to be processed is compressed and mapped to a low-dimensional latent space. The resulting initial noise-free latent variables are then subjected to DDIM inversion to obtain the initial latent variables. Frequency domain modulation module: Using two-dimensional fast Fourier transform, the initial latent variables are transformed from the spatial domain to the frequency domain, separating the high-frequency structural components and the low-frequency style components; The low-frequency style components are suppressed according to a set ratio, while the high-frequency structure components are retained. Then, the components are transformed back to the spatial domain through inverse Fourier transform to obtain the preliminary purified latent variables. Based on the preliminary purified latent variables, the noise is iteratively removed to generate the current latent variables to be optimized. Decoupled cross-attention module: Based on the content image to be processed and the style reference image, extract text features, content features and style features. Each network layer constructs three parallel cross-attention branches. After independently injecting the text features, content features and style features, the calculation results of the three branches are weighted and fused to obtain the current attention features. Each network layer constructs three parallel cross-attention branches, which independently inject text features, content features, and style features, respectively. Specifically: For the text attention branch of each network layer, the text features are projected to obtain text key and value vectors, and then cross-attention processing is performed to obtain text attention features. For each network layer's content attention branch, after projecting the content features to obtain content key and value vectors, the content attention features are obtained through cross-attention processing. For each network layer's style attention branch, style features are projected to obtain style keys and value vectors, and then cross-attention processing is used to obtain style attention features. Attention-driven guidance module: Constructs style anchors using content query vectors and style key-value vectors; Calculate the difference between the current attention feature and the style anchor point to obtain the style-guided energy; backpropagate the gradient of the style-guided energy back to the current latent variable to be optimized to generate standard latent variables, specifically: The style-guided energy is backpropagated relative to the gradient of the current latent variable to be optimized to obtain the gradient update direction and magnitude of the current latent variable to be optimized. The current latent variable to be optimized is adaptively updated along the gradient descent direction to complete the gradient calibration of the current latent variable to be optimized.

Images