A zero-shot image style transfer method based on joint frequency perception and spatial control

Abstract

The application discloses a zero-shot image style transfer method based on joint frequency perception and spatial control of a latent diffusion model, and relates to the technical field of image processing and computer vision.The application processes low-frequency components and high-frequency components respectively in a latent space, so that content structure information and style texture information are differentially controlled in a fusion process, which helps to improve the content drift problem in the generated result and improve the expression ability of style texture.The application encodes a structure edge map through VAE, constructs a spatial control mask matched with a latent representation space, which helps to reduce the spatial misplacement problem caused by pixel space mask scaling, and improves the style leakage, edge blur and halo artifact phenomenon in the local area.The application realizes zero-shot image style transfer based on a pre-trained latent diffusion model, does not need additional training for the target style, and has high inference efficiency and good deployment convenience.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision technology, and specifically to a zero-shot image style transfer method based on a latent diffusion model that combines frequency perception and spatial control. Background Technology

[0002] Image style transfer aims to transfer style features from a reference image to a target image while preserving the semantic structure of the content image, thereby generating an image result that combines the original content with the target style. This technology has high application value in fields such as digital content generation, artistic creation, image editing, and multimedia processing.

[0003] Early image style transfer methods mainly included optimization-based methods and feedforward neural network-based methods, such as Adaptive Instance Normalization (AdaIN). These methods can achieve style feature transfer to a certain extent, but in complex scenes, they often suffer from insufficient texture detail representation and limited ability to preserve structure, making it difficult to simultaneously achieve style representation and content consistency.

[0004] In recent years, denoising diffusion probabilistic models (DDPM) and latent diffusion models (LDM) have demonstrated good generation quality and controllability in image generation tasks. Style transfer methods based on diffusion models can leverage the generation priors of pre-trained models to improve the generation quality and style representation of the resulting images, and are gradually becoming an important technical route for image style transfer. Meanwhile, to reduce computational costs and deployment complexity, zero-shot style transfer schemes requiring little or no training are also receiving increasing attention.

[0005] However, when applying latent diffusion models directly to style transfer of zero-shot images, existing techniques still face the challenge of balancing style representation and structural preservation. On the one hand, diffusion models exhibit a degree of randomness during the reverse denoising process, which can easily cause semantic structural shifts in the content image, leading to issues such as changes in the main geometric structure, distortion of key semantic features, or unstable local layouts in the generated results. On the other hand, employing strong spatial constraints to maintain content structure may inhibit the full transfer of style and texture, resulting in insufficient expression of brushstrokes, textures, and details in the generated results.

[0006] Furthermore, most existing spatial control methods directly perform mask scaling or region blending based on pixel space, failing to fully consider the representational differences between latent representations and pixel features in the latent diffusion model, thus easily leading to the following problems:

[0007] One issue is content drift. Because variational autoencoders (VAEs) have limited ability to preserve high-frequency information during the encoding process, high-frequency details in the latent representation are easily weakened, resulting in insufficient texture details in the generated image and potentially affecting the consistency of the main structure and local semantics.

[0008] Second, there is the issue of style leakage. Existing methods typically employ simple downsampling of the pixel space mask. This mask is difficult to maintain spatial consistency with the underlying representation after VAE encoding, which may cause style information to seep into areas where content should be preserved, resulting in phenomena such as blurred edges, local halos, or regional contamination.

[0009] Therefore, how to balance content structure preservation and style texture expression during zero-sample image style transfer, and achieve spatial control that is compatible with the latent representation of the latent diffusion model, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0010] The purpose of this invention is to provide a zero-sample image style transfer method based on joint frequency sensing and spatial control, so as to solve the problems of content drift, style leakage, and difficulty in balancing style expression and structure preservation in existing diffusion model image style transfer methods.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] A zero-shot image style transfer method based on joint frequency awareness and spatial control includes the following steps:

[0013] Step 1: Receive image data including content image, style reference image and text prompt, extract image edge map, and use pre-trained VAE encoder and inversion technology to map image data into inversion latent representation;

[0014] Step 2: Perform discrete wavelet transform on the inverted latent representation to decouple content and style in the frequency domain, perform adaptive instance normalization alignment on the low-frequency structural components, perform energy enhancement and weighted fusion on the high-frequency style details, and output the fused initial latent representation after inverse discrete wavelet transform.

[0015] Step 3: Expand the image edge map into a pseudo image in the channel dimension, encode the pseudo image using a pre-trained VAE encoder, and then construct a multi-scale spatial control mask cache set corresponding to different resolution levels through pooling operations.

[0016] Step 4: Introduce the IP-Adapter component to decouple text conditions and image conditions; the IP-Adapter component includes a text condition branch, an image condition branch, and a decoupled cross-attention module. Text prompts are input to the text condition branch to provide semantic constraints, and style reference images are input to the image condition branch to provide style conditions. The decoupled cross-attention module models the text conditions and image conditions respectively; the image edge map is input to ControlNet to inject structure-guided features; simultaneously, the style signal injection region in the IP-Adapter image condition branch is modulated using the multi-scale spatial control mask, and denoising operations are iteratively performed to finally output the zero-sample image style transfer result.

[0017] Furthermore, step 1 is detailed as follows:

[0018] Get a content image and a style reference image and text prompts Use edge detection algorithms to extract content images Transform into a structural edge graph And Canny edge map ; to display content images and style reference images The initial latent representations are obtained by inputting them into the pre-trained variational autoencoder. and Then, the DDIM inversion technique based on null-text optimization is used to obtain the latent representation of content inversion. and style inversion potential representation .

[0019] Furthermore, step 2 is detailed as follows:

[0020] Step 2.1: Perform discrete wavelet transform on each channel of the content inversion latent representation and the style inversion latent representation respectively to obtain the high-frequency components and low-frequency approximate components of the content inversion latent representation and the high-frequency components and low-frequency approximate components of the style inversion latent representation.

[0021] Step 2.2: Perform adaptive instance normalization on the low-frequency approximation components of the content inversion latent representation and the style inversion latent representation to obtain the fused low-frequency components;

[0022] Step 2.3: Multiply the high-frequency components of the style inversion latent representation by the enhancement factor, and perform weighted fusion with the high-frequency components of the content inversion latent representation to obtain the fused high-frequency components;

[0023] Step 2.4: Reconstruct the modulation through inverse discrete wavelet transform to obtain the initial latent representation.

[0024] Furthermore, step 3 is detailed as follows:

[0025] First, the single-channel structure edge map is expanded into a three-channel pseudo-image in the channel dimension. Then, the pseudo-image is encoded using a pre-trained VAE encoder to obtain a mask latent representation corresponding to the latent spatial distribution.

[0026] The original spatial control mask is obtained by aggregating the channel features of the latent representation of the mask using mean pooling. ;

[0027] Finally, adaptive average pooling is used to construct a hierarchical mask pyramid cache set. :

[0028]

[0029] in, This represents a multi-scale spatial control mask cache set; This represents the target resolution set, such as the feature map resolutions corresponding to different layers of U-Net; This indicates a specific resolution; This indicates that the original spatial control mask is adaptively pooled to the resolution. .

[0030] Furthermore, step 4 is detailed as follows:

[0031] Step 4.1: Import the IP-Adapter component, and receive text prompts via text conditional branches. The image conditional branch includes an image encoder and an image conditional projection module, wherein the image encoder is used to extract the style reference image. The image conditional projection module is used to map the visual features to a conditional feature space that matches the U-Net cross-attention layer; the decoupled cross-attention module models the text conditions and image conditions respectively;

[0032] Step 4.2: Input the Canny edge map into the pre-trained ControlNet architecture, extract the multi-scale geometric control residuals, and sum and inject them into the corresponding decoder layer of U-Net.

[0033] Step 4.3: In the iterative denoising step, for a specific network layer of U-Net, from... Search for the mask corresponding to the resolution in the middle. For any target resolution Let the corresponding multi-scale spatial control mask be . ;

[0034] Step 4.4: Utilize the multi-scale spatial control mask to output cross-attention for the IP-Adapter image branches. Modulation:

[0035]

[0036] in Represents the Hadama product; This represents the style conditional features output by the IP-Adapter image branch at the cross-attention layer. This represents the branch attention output of the image after spatial mask modulation;

[0037] Step 4.5: Perform reverse denoising using the DDIM scheduler, taking the fused initial latent representation as the starting point for generation; during each step of reverse denoising, the text conditional branch of the IP-Adapter is based on text prompts. The IP-Adapter provides semantic constraints, its image conditional branch provides style conditions based on the style reference image, and the ControlNet provides structural constraints based on the Canny edge map. These three types of conditions jointly guide the U-Net's noise prediction and latent representation update. The final output is the style transfer result of the decoded zero-sample image.

[0038] The zero-shot image style transfer method takes a content image, a style reference image, and a text prompt as input. The content image is used to provide the main structure and semantic layout, the style reference image is used to provide the target style features, and the text prompt is used as input to the text conditional branch of the IP-Adapter and provides semantic generation constraints during the reverse denoising process.

[0039] By adopting the above technical solution, the present invention has the following beneficial effects:

[0040] (1) By processing the low-frequency and high-frequency components separately in the latent space, the present invention enables differentiated control of content structure information and style texture information during the fusion process, which helps to improve the content drift problem in the generated results and enhance the expressive power of style texture.

[0041] (2) By encoding the structural edge map with VAE, the present invention constructs a spatial control mask that matches the potential representation space, which helps to reduce the spatial misalignment problem caused by pixel spatial mask scaling and improves style leakage, edge blurring and halo artifacts in local areas.

[0042] (3) This invention achieves zero-sample image style transfer based on a pre-trained latent diffusion model, without the need for additional training for the target style, and has high inference efficiency and good deployment convenience. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the overall process of the zero-sample image style transfer method of the present invention;

[0044] Figure 2 This is a schematic diagram of the dynamic wavelet latent fusion process in the method of the present invention;

[0045] Figure 3 This is a schematic diagram of the multi-scale spatial control process of the VAE compression mask in the method of the present invention;

[0046] Figure 4 This is a flowchart illustrating the reasoning process of the zero-sample style transfer method of this invention. Detailed Implementation

[0047] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments. It should be understood that the following embodiments are only used to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0048] like Figure 1 As shown, this embodiment provides a zero-shot image style transfer method based on joint frequency awareness and spatial control. The method is based on a pre-trained latent diffusion model, preferably using the Stable DiffusionXL (SDXL) model, and combines an IP-Adapter (image cue adapter) and a ControlNet (conditional control network) to implement conditional guidance and structural constraints. The IP-Adapter is used to decouple the text conditions corresponding to the text cue and the image conditions corresponding to the style reference image, while the ControlNet provides structural constraints based on the Canny edge map of the content image. The method mainly includes the following processes: preprocessing and dual-source inversion, dynamic wavelet latent fusion, spatial control based on a VAE compression masking strategy, and cross-attention modulation and inverse denoising generation.

[0049] During preprocessing and dual-source inversion, the system receives a content image, a style reference image, and text prompts. It extracts the structural edge map and Canny edge map of the content image and uses a pre-trained variational autoencoder (VAE) and inversion technique (DDIM inversion) to map the content image and style reference image to the corresponding inversion latent representations. The system outputs the content inversion latent representation, the style inversion latent representation, the single-channel structural edge map of the content image, and the Canny edge map. The text prompts are retained for use in the subsequent text conditional branch of the IP-Adapter.

[0050] The inverted latent representation is received during the dynamic wavelet latent fusion process. Content and style are decoupled in the frequency domain. Adaptive instance normalization (AdaIN) alignment is performed on the low-frequency structural components. Energy enhancement and weighted fusion are performed on the high-frequency style details. The fused initial latent representation is then output through inverse discrete wavelet transform (IDWT).

[0051] In the spatial control process based on the VAE compression masking strategy, the single-channel structure edge map is received, expanded into a pseudo-image in the channel dimension, and input into the pre-trained VAE encoder for feature mapping to obtain a mask latent representation that matches the latent representation space. Then, a multi-scale spatial control mask set corresponding to different resolution levels is constructed through pooling operations.

[0052] During the cross-attention modulation and inverse denoising generation process, the fused potential initial representation, text cue, Canny edge map, and multi-scale spatial control mask set are received. In the inverse generation process, the text cue is input into the text conditional branch of the IP-Adapter to provide semantic constraints, the style reference image is input into the image conditional branch of the IP-Adapter to provide style conditions, and the Canny edge map is input into ControlNet to inject structure-guided features. Simultaneously, the style signal injection region in the image conditional branch of the IP-Adapter is modulated using the spatial mask, and denoising operations are iteratively performed, ultimately outputting the zero-sample image style transfer result.

[0053] Combination Figure 1 The overall processing flow is shown below. The specific steps of the zero-sample image style transfer method of the present invention will be described in detail below:

[0054] Step 1: Data acquisition and inversion preprocessing.

[0055] Step 1.1: Given a content image and a style reference image and text prompts Content images provide the semantic layout and main structure to be preserved, style reference images provide the artistic textures and visual style to be transferred, and text prompts... This is used as input to the text conditional branch of the IP-Adapter to provide semantic constraints during the generation process. The goal of this invention is to synthesize a single output image. The output image retains the content image. Semantic layout, while transferring style reference images The artistic texture, and meets the text prompts The semantic constraints described.

[0056] Step 1.2: Use the PidiNet (Pixel Difference Network) edge detection algorithm to detect edges from the content image. Extract the structural edge map The Canny edge detection algorithm is used to analyze the content image. Convert to Canny edge map The structural edge map is used for subsequent spatial control mask construction, and the Canny edge map is used for structural guidance input of ControlNet.

[0057] Step 1.3: Transfer the content image and style reference images Input pre-trained variational autoencoder In the process, both are encoded into the latent space to obtain initial latent representations. and .

[0058] Step 1.4: Employ DDIM inversion technology based on null-text optimization (a denoising diffusion implicit model inversion technology based on null-text optimization). The process involves first using DDIM inversion to gradually map the image's latent representation from a low-noise state to a high-noise state. Then, during the inversion process, optimize the null-text conditional embedding to make the inversion trajectory closer to the original image's generation trajectory, thereby improving the content structure preservation capability during subsequent denoising generation. This technology is then used to map the initial latent representation back to its initial noise state, obtaining the content inversion latent representation. and style inversion potential representation In this embodiment, a 30-step inversion is preferably used to enhance the ability to preserve the structural information of the content image during subsequent generation processes.

[0059] Step 2: Forward frequency decoupling and dynamic wavelet latent fusion. See [link / document name] for its structure and process. Figure 2 In obtaining the inversion potential representation and Then, dynamic wavelet latent fusion is performed to decouple and fuse content and style information in the frequency domain. The specific implementation process is as follows:

[0060] Step 2.1, for each channel of the latent representation Perform Discrete Wavelet Transform (DWT):

[0061]

[0062]

[0063] in, They represent the latent representation of content inversion in the first... Low-frequency approximation components, horizontal high-frequency detail components, vertical high-frequency detail components, and diagonal high-frequency detail components on each channel; They respectively represent the latent representation of style inversion in the th... The corresponding wavelet components on each channel. Where, the subscript... Indicates content image, subscript Image representing style, This represents the time step after DDIM inversion. This represents the potential channel index. In this embodiment, the db4 wavelet basis of the Daubechies family is preferred as the DWT transform function.

[0064] Step 2.2: Perform adaptive instance normalization on the low-frequency approximation components to achieve global statistical feature alignment:

[0065]

[0066] Here, AdaIN represents Adaptive Instance Normalization, which achieves the fusion of low-frequency content structure and low-frequency style statistical information by adjusting the mean and standard deviation of low-frequency content components to the mean and standard deviation of low-frequency style components. Indicates the first The fused low-frequency components processed by AdaIN on each channel are used to transfer the global tonal and distribution features of the style image while preserving the overall structure of the content. The low-frequency components mainly characterize the overall structure, layout, and tonal information of the image.

[0067] Step 2.3: Perform style injection and energy enhancement on the high-frequency components. Specifically, perform style inversion on the high-frequency components of the latent representation. (High-frequency components here) It is a collective term for the three high-frequency sub-bands, namely .therefore, The content image is represented in the first... The set of three high-frequency detail components on each channel The style image is represented in the first (A set of three high-frequency detail components on each channel) multiplied by an enhancement factor Subsequently, the high-frequency components of the latent representation of content inversion Perform weighted fusion:

[0068]

[0069] in, For the first Fusion of high-frequency components on each channel Represents the fusion coefficient. This represents the enhancement factor. In this embodiment, it is preferred to set... , .

[0070] Step 2.4: Reconstruct the modulated latent representation using inverse discrete wavelet transform (IDWT) :

[0071]

[0072] The As the initial potential representation for the subsequent inverse denoising process.

[0073] Step 3: VAE compression mask and multi-scale spatial control. For its structure and process, please refer to [link / reference]. Figure 3 To address the "style leakage" defect caused by global style injection, this embodiment employs a VAE compression mask strategy to construct a spatial control mask that matches the potential representation space.

[0074] Step 3.1: Extract the single-channel binary structure edge map Repeated along the channel dimension, it expands into a three-channel pseudo-image tensor. .

[0075] Step 3.2: Utilize the same pre-trained VAE encoder as in Step 1.3 Encoding pseudo-images:

[0076]

[0077] This operation maps the edge map to a continuous latent representation space to obtain a masked latent representation corresponding to the latent space distribution.

[0078] Step 3.3: Aggregate channel features using mean pooling to obtain the original spatial control mask. :

[0079]

[0080] in, This represents the original spatial control mask obtained by aggregating the latent representations of the mask; The mask's latent representation is in the first... Spatial feature map of each channel; This represents the number of channels. The formula represents averaging the latent representation of the mask along the channel dimension to obtain a single-channel spatial control weight map.

[0081] Step 3.4: Given that SDXL's U-Net architecture operates across multiple resolution scales, an adaptive average pooling method is used to construct a multi-scale spatial control mask cache set. :

[0082]

[0083] in, This represents a multi-scale spatial control mask cache set; This represents the target resolution set, such as the feature map resolutions corresponding to different layers of U-Net; This indicates a specific resolution. This indicates that the original spatial control mask is adaptively pooled to the resolution. This is done to match the feature map size of the corresponding U-Net layer. The hierarchical mask set is used to match the feature map scale of different resolution levels of U-Net.

[0084] Step 4: Decouple cross-attention and reverse denoising generation. See the flowchart for details. Figure 4 In this step, the fused initial latent representation serves as the starting point for inverse denoising, and the text prompt... Style reference image as input to the IP-Adapter text conditional branch As input to the image conditional branch of the IP-Adapter, the Canny edge map serves as the structural control input of ControlNet, together guiding the generation of the target style image.

[0085] Step 4.1: Introduce an IP-Adapter component to decouple text and image conditions. The IP-Adapter component includes a text prompt processing section, an image condition processing section, and a decoupled cross-attention module. The text prompt processing section is used to receive text prompts. The text is then converted into text conditional features to participate in U-Net cross-attention computation via text conditional branches. The image conditional processing part includes an image encoder and an image conditional projection module, wherein the image encoder is used to extract style reference images. The image conditional projection module maps the visual features to a conditional feature space that matches the U-Net cross-attention layer, allowing them to participate in U-Net cross-attention computation through image conditional branches. The decoupled cross-attention module models both text and image conditions separately, providing semantic direction constraints for text prompts and style reference images for style texture, color distribution, and visual appearance constraints. Both jointly participate in the inverse denoising generation process.

[0086] Step 4.2: Introduce ControlNet to extract spatial structure constraints. Input the Canny edge map obtained in Step 1.2 into the pre-trained ControlNet architecture. ControlNet extracts multi-scale geometric control residuals through network blocks consistent with the Stable Diffusion U-Net encoder, and sums and injects them into the corresponding decoder layer of U-Net to constrain the main structure and spatial layout.

[0087] Step 4.3: In the iterative denoising step, for a specific network layer of U-Net, from... Search for the mask corresponding to the resolution in the middle. For any target resolution Let the corresponding multi-scale spatial control mask be . ,Right now .all Together they form a multi-scale mask cache set .

[0088] Step 4.4: Output cross-attention to the IP-Adapter image branches using the spatial mask. Modulation:

[0089]

[0090] in This represents the Hadama product. This represents the style conditional features output by the IP-Adapter image branch at the cross-attention layer. This represents the spatial control mask that matches the current U-Net layer resolution. This represents the image branch attention output after spatial mask modulation. This modulation operation allows for spatial control of the style injection region and injection intensity in the image conditional branch while preserving the semantic constraints of the text conditional branch.

[0091] Step 4.5: Perform reverse denoising using the DDIM scheduler, taking the fused initial latent representation as the starting point for generation. During each step of reverse denoising, the IP-Adapter's text conditional branch is based on text prompts. Semantic constraints are provided. The image conditional branch of the IP-Adapter provides style conditions based on the style reference image, and ControlNet provides structural constraints based on the Canny edge map. These three types of conditions jointly guide the noise prediction and latent representation update of U-Net. In this embodiment, a 50-step inverse denoising method is preferably used, and the classifier-free guided (CFG) scale is set to 3.5. The final output is the style transfer result of the decoded zero-sample image.

[0092] Step 5: Conduct experimental verification and performance evaluation of the zero-sample image style transfer method.

[0093] Step 5.1: Construct the test dataset. Select 40 artworks from the StyleRank dataset as style reference images, and 32 content images from PIE-Bench. Pair each content image with a style reference image to obtain 1280 content-style reference image pairs. For each content-style reference image pair, construct a corresponding text prompt. This is used to describe the main content and stylization requirements of the target generated image, such as preserving the main subject of the content image and generating an image with the artistic style of the reference image. The test samples can be divided into two main categories: natural images and generated images, and further include four types of scenes: animals, people, indoor scenes, and outdoor scenes, to verify the applicability of the method of the present invention in real-world and synthetic scenes.

[0094] Step 5.2: Set evaluation metrics. To evaluate the performance of the method of this invention in style transfer tasks, text alignment is used. Consistency with style The semantic relevance and stylistic expressiveness of the generated results are evaluated. Since this embodiment incorporates text prompts during the reverse denoising process... As input to the text conditional branch of the IP-Adapter, text alignment is therefore used. Regarding the generated results and text prompts The semantic consistency between features is evaluated using the DINO score (self-supervised visual feature similarity). Structural similarity (SSIM) and learned perceptual patch similarity (LPIPS) are used to evaluate the structural preservation ability and perceptual similarity of the generated results; ImageReward and harmonic mean are used. The Style-Content Trade-off Ratio (SCR) is used as a comprehensive evaluation metric to measure the balance between stylization and structural preservation. The harmonic mean (SCR) is used to... The specific calculation formula is as follows:

[0095]

[0096] The specific formula for calculating the Style-Content Trade-off (SCR) is as follows:

[0097]

[0098] In the formula, This is a minimal constant used to maintain numerical stability and prevent the denominator from being zero (in this embodiment, it is taken as...). ).

[0099] Step 5.3: Comparative and Ablation Experiments. To verify the effectiveness of the proposed dynamic wavelet latent fusion module, VAE compression mask space control strategy, and zero-shot architecture, comparative and ablation experiments were conducted on the test set containing 1280 content-style reference image pairs. The quantitative comparison results are shown in Table 1.

[0100] Table 1. Quantitative comparison experiment between existing methods and the method of the present invention.

[0101]

[0102] Step 5.4: Experimental Results Analysis. The method of this invention is compared with existing methods such as InstantStyle (instant stylization method), StyleShot (few-shot / fast style transfer method), StyleID (style transfer method based on identity / style decoupling), CSGO (content style composition method in text-to-image generation), and StyleSSP (sampling start point enhancement method based on training-free diffusion style transfer). Experimental results show that the method of this invention achieves better performance in terms of structure preservation metrics. It performed well on SSIM and LPIPS; and showed good performance on style consistency metrics. The method of this invention can maintain a high level; in terms of comprehensive evaluation indicators ImageReward, Compared with SCR, the method of this invention can better balance style expression and content preservation.

[0103] From a visual perspective, in cross-domain style transfer scenarios, the images generated by the method of this invention can effectively integrate the reference style while maintaining the main geometric layout and key semantic features of the content image. Particularly in complex lighting and portrait stylization scenarios, the method of this invention demonstrates good performance in improving content drift, texture smoothing, and local structural distortion.

[0104] Furthermore, in terms of frequency decoupling, employing the db4 wavelet basis from the Daubechies family helps to balance high-frequency artifact suppression with structural detail preservation. Regarding spatial control, replacing direct bilinear scaling of the pixel space with a VAE-compressed mask helps improve local halo and edge blurring issues caused by inconsistencies between the mask and the latent representation space. Based on a zero-shot inference framework using a pre-trained latent diffusion model, the method of this invention can achieve zero-shot image style transfer without additional fine-tuning for the target style.

[0105] Although the present invention has been described herein with reference to embodiments thereof, the above embodiments are merely preferred embodiments of the present invention, and the implementation of the present invention is not limited to the above embodiments. It should be understood that those skilled in the art can devise many other modifications and implementations, which will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. A zero-shot image style transfer method based on joint frequency sensing and spatial control, characterized in that, The method is based on a pre-trained latent diffusion model and includes the following steps: Step 1: Receive image data including content image, style reference image and text prompt, extract image edge map, and use pre-trained VAE encoder and inversion technology to map image data into inversion latent representation; Step 2: Perform discrete wavelet transform on the inverted latent representation to decouple content and style in the frequency domain, perform adaptive instance normalization alignment on the low-frequency structural components, perform energy enhancement and weighted fusion on the high-frequency style details, and output the fused initial latent representation after inverse discrete wavelet transform. Step 3: Expand the image edge map into a pseudo image in the channel dimension, encode the pseudo image using a pre-trained VAE encoder, and then construct a multi-scale spatial control mask cache set corresponding to different resolution levels through pooling operations. Step 4: Introduce the IP-Adapter component to decouple text conditions and image conditions; the IP-Adapter component includes a text condition branch, an image condition branch, and a decoupled cross-attention module. Text prompts are input to the text condition branch to provide semantic constraints, and style reference images are input to the image condition branch to provide style conditions. The decoupled cross-attention module models the text conditions and image conditions respectively; the image edge map is input to ControlNet to inject structure-guided features; simultaneously, the style signal injection region in the IP-Adapter image condition branch is modulated using the multi-scale spatial control mask, and denoising operations are iteratively performed to finally output the zero-sample image style transfer result.

2. The zero-shot image style transfer method based on joint frequency sensing and spatial control according to claim 1, characterized in that, Step 1 is described in detail as follows: Obtain a content image, a style reference image, and a text prompt; use an edge detection algorithm to convert the content image into a structure edge map. And Canny edge map The content image and style reference image are input into a pre-trained variational autoencoder to obtain initial latent representations, respectively. and ; Then, the DDIM inversion technique based on null-text optimization is used to obtain the latent representation of content inversion. and style inversion potential representation .

3. The zero-shot image style transfer method based on joint frequency sensing and spatial control according to claim 2, characterized in that, Step 2 is described in detail below: Step 2.1: Perform discrete wavelet transform on each channel of the content inversion latent representation and the style inversion latent representation respectively to obtain the high-frequency components and low-frequency approximate components of the content inversion latent representation and the high-frequency components and low-frequency approximate components of the style inversion latent representation. Step 2.2: Perform adaptive instance normalization on the low-frequency approximation components of the content inversion latent representation and the style inversion latent representation to obtain the fused low-frequency components; Step 2.3: Multiply the high-frequency components of the style inversion latent representation by the enhancement factor, and perform weighted fusion with the high-frequency components of the content inversion latent representation to obtain the fused high-frequency components; Step 2.4: Reconstruct the modulation through inverse discrete wavelet transform to obtain the initial latent representation.

4. The zero-shot image style transfer method based on joint frequency sensing and spatial control according to claim 3, characterized in that, Step 3 is described in detail below: First, the structural edge map of the single channel is... The pseudo-image is expanded into a three-channel pseudo-image in the channel dimension, and then the pseudo-image is encoded using a pre-trained VAE encoder to obtain a mask latent representation corresponding to the latent spatial distribution. The original spatial control mask is obtained by aggregating the channel features of the latent representation of the mask using mean pooling. ; Finally, adaptive average pooling is used to construct a hierarchical mask pyramid cache set. : in, This represents a multi-scale spatial control mask cache set; This represents the target resolution set, such as the feature map resolutions corresponding to different layers of U-Net; This indicates a specific resolution; This indicates that the original spatial control mask is adaptively pooled to the resolution. .

5. The zero-shot image style transfer method based on joint frequency sensing and spatial control according to claim 4, characterized in that, Step 4 is described in detail below: Step 4.1: Import the IP-Adapter component, and receive text prompts via text conditional branches. The text conditional branch is then converted into text conditional features; the image conditional branch includes an image encoder and an image conditional projection module, the image encoder being used to extract style reference images. The image conditional projection module is used to map the visual features to a conditional feature space that matches the U-Net cross-attention layer; the decoupled cross-attention module models the text condition and the image condition respectively; Step 4.2: Input the Canny edge map into the pre-trained ControlNet architecture, extract the multi-scale geometric control residuals, and sum and inject them into the corresponding decoder layer of U-Net. Step 4.3: In the iterative denoising step, for a specific network layer of U-Net, from... Search for the mask corresponding to the resolution in the middle. For any target resolution Let the corresponding multi-scale spatial control mask be . ; Step 4.4: Utilize the multi-scale spatial control mask to output cross-attention for the IP-Adapter image branches. Modulation: in Represents the Hadama product; This represents the style conditional features output by the IP-Adapter image branch at the cross-attention layer. This represents the branch attention output of the image after spatial mask modulation; Step 4.5: Perform reverse denoising using the DDIM scheduler, taking the fused initial latent representation as the starting point for generation; during each step of reverse denoising, the text conditional branch of the IP-Adapter is based on text prompts. The IP-Adapter provides semantic constraints, its image conditional branch provides style conditions based on the style reference image, and the ControlNet provides structural constraints based on the Canny edge map. These three types of conditions jointly guide the U-Net's noise prediction and latent representation update. The final output is the style transfer result of the decoded zero-sample image.

6. The zero-shot image style transfer method based on joint frequency sensing and spatial control according to claim 5, characterized in that, The DDIM inversion technique based on null-text optimization is as follows: First, DDIM inversion is used to gradually map the inversion potential representation of the image from a low-noise state to a high-noise state. Then, the null text conditional embedding is optimized during the inversion process to make the inversion trajectory closer to the original image generation trajectory. This technique maps the initial potential representation back to its initial noise state to obtain the content inversion potential representation and the style inversion potential representation.

7. The zero-shot image style transfer method based on joint frequency sensing and spatial control according to claim 6, characterized in that, The discrete wavelet transform uses the db4 wavelet basis of the Daubechies family.

8. The zero-shot image style transfer method based on joint frequency sensing and spatial control according to claim 7, characterized in that, The specific formula for obtaining the fused high-frequency components is as follows: in, For the first Fusion of high-frequency components on each channel Represents the fusion coefficient. Indicates the enhancing factor. The latent representation of style inversion is shown in the first... High-frequency components on each channel The content inversion potential representation is in the first... High-frequency components on each channel.

9. A zero-shot image style transfer method based on joint frequency sensing and spatial control according to claim 8, characterized in that, The channel features latently represented by the mean pooling aggregation mask are specifically expressed by the following formula: in, The mask's latent representation is in the first... Spatial feature map of each channel; Indicates the number of channels.

Images