Non-training image style migration method and device based on diffusion model noise recombination, equipment, medium and product

Abstract

The invention discloses a non-training image style migration method and device based on diffusion model noise recombination, equipment, a medium and a product, and relates to the field of image style migration. The method comprises the following steps: encoding information data, and performing circular processing of feature extraction on obtained submerged space data by adopting a diffusion model to obtain an intermediate data set; carrying out feature fusion based on the intermediate data set, and carrying out weighted combination on recombination noise obtained through fusion by adopting a moving average algorithm to obtain final recombination noise; performing feature extraction on the final recombination noise by adopting a diffusion model, and determining a new intermediate data set; taking the new intermediate data set as an intermediate data set, and returning to carry out the feature fusion step until the circular processing of feature extraction is stopped; and carrying out decoding processing on the final recombined noise to obtain a stylized image so as to realize image style migration. According to the method, efficient style migration can be realized, and generalization and practicability are improved.

Description

Technical Field

[0001] This application relates to the field of image style transfer, and in particular to a training-free image style transfer method, apparatus, device, medium, and product based on diffusion model noise reconstruction. Background Technology

[0002] The technical background of image style transfer is rooted in the long-term exploration of visual content decoupling and reconstruction in the fields of computer vision and machine learning. Its core goal is to separate and recombine the content structure and style features (such as color, texture, brushstrokes, artistic style, etc.) of an image to generate a new image that combines the semantics of both. The development of this technology is closely related to breakthroughs in computer graphics and deep learning. Early examples include Neural Style Transfer (NST) proposed by Gatys et al. in 2015, which uses Convolutional Neural Networks (CNNs) to minimize the feature differences between content and style images at different layers of the network through optimization algorithms. It uses high-level features to capture content structure and low-level features to model style correlation using Gram matrices. However, it has limitations such as time-consuming iterative optimization and insufficient representation of high-frequency details. Subsequently, Generative Adversarial Networks (GANs) have driven paradigm innovation. For example, CycleGAN and StarGAN achieve cross-domain transfer of unpaired data through cycle consistency loss or multi-domain classifiers. The StyleGAN series decouples the latent space to achieve fine-grained style control, but it relies on large-scale training data and requires retraining to generalize to new styles. In recent years, diffusion models (such as Latent Diffusion Models, LDMs) have emerged, generating high-quality images in the latent space through an inverse diffusion process, especially excelling in the diversity and coherence of texture details. Traditional diffusion model methods learn style feature maps by training networks, but they suffer from limitations such as reliance on large amounts of paired content-style image datasets for training, resulting in high data acquisition costs and limited generalization ability. With the increasing prevalence of multimodal data applications, users have a growing demand for real-time and flexible image style transfer, especially in scenarios where data is scarce or where rapid adaptation to new styles is required (such as artistic creation and real-time filters). Training-free style transfer methods have become a research hotspot. However, existing training-free methods face technical bottlenecks in areas such as noise feature processing and cross-domain feature alignment, making it difficult to achieve high-quality style transfer while maintaining computational efficiency. Furthermore, in the long generation process, the inter-domain differences between content and style information accumulate in ordinary training-free methods, leading to artifacts, blurring, or content distortion in the generated images, particularly in terms of preserving and enhancing high-frequency details. Summary of the Invention

[0003] The purpose of this application is to provide a training-free image style transfer method, apparatus, device, medium, and product based on diffusion model noise reconstruction, which can achieve efficient style transfer and improve generalization and practicality.

[0004] To achieve the above objectives, this application provides the following solution:

[0005] Firstly, this application provides a training-free image style transfer method based on diffusion model noise reconstruction, including:

[0006] Acquire information data; the information data includes: content images and style images;

[0007] The information data is encoded, and a diffusion model is used to perform a cyclical feature extraction process on the obtained latent space data to obtain an intermediate dataset; the feature extraction includes: adding noise and denoising; the intermediate dataset includes: noisy intermediate data and denoised intermediate data;

[0008] After the loop processing of any feature extraction:

[0009] Feature fusion is performed based on the intermediate dataset, and the reconstructed noise obtained by fusion is weighted and combined using a moving average algorithm to obtain the final reconstructed noise; wherein, the feature fusion is performed using the covariance alignment method of Cholesky decomposition and a noise reconstruction module guided by frequency domain features.

[0010] The diffusion model is used to extract features from the final recombined noise, and the denoised intermediate data in the corresponding intermediate dataset is used as the content image denoising intermediate data contained in the intermediate dataset to obtain a new intermediate dataset.

[0011] The new intermediate dataset is used as the intermediate dataset, and the process returns to the step of "performing feature fusion based on the intermediate dataset and using a moving average algorithm to weight and combine the fused recombined noise to obtain the final recombined noise" until the feature extraction loop stops.

[0012] The final reconstructed noise is decoded to obtain a stylized image, thereby achieving image style transfer.

[0013] Secondly, this application provides a training-free image style transfer device based on diffusion model noise reconstruction, comprising:

[0014] An information data acquisition module is used to acquire information data; the information data includes: content images and style images;

[0015] The feature extraction module is used to encode the information data and perform a cyclical feature extraction process on the obtained latent space data using a diffusion model to obtain an intermediate dataset; the feature extraction includes: adding noise and denoising; the intermediate dataset includes: noisy intermediate data and denoised intermediate data;

[0016] The feature fusion module is used to perform feature fusion based on the intermediate dataset after any feature extraction loop, and to use a moving average algorithm to weight and combine the fused reconstructed noise to obtain the final reconstructed noise; wherein, the feature fusion is performed using the covariance alignment method of Cholesky decomposition and a noise reconstruction module guided by frequency domain features.

[0017] The replacement module is used to extract features from the final recombined noise using a diffusion model, and to obtain the denoised intermediate data in the corresponding intermediate dataset as the content image denoising intermediate data contained in the denoised intermediate data in the intermediate dataset, so as to obtain a new intermediate dataset.

[0018] The return module is used to take the new intermediate dataset as an intermediate dataset and return to the "feature fusion module" until the loop processing of feature extraction stops;

[0019] The decoding module is used to decode the final reconstructed noise to obtain a stylized image, thereby achieving image style transfer.

[0020] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method for training-free image style transfer based on diffusion model noise reconstruction.

[0021] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for training-free image style transfer based on diffusion model noise reconstruction.

[0022] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for training-free image style transfer based on diffusion model noise reconstruction.

[0023] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0024] This application provides a training-free image style transfer method, apparatus, device, medium, and product based on diffusion model noise reconstruction. The method involves encoding information data and performing a cyclical process of feature extraction from the latent space data using a diffusion model to obtain an intermediate dataset. Feature fusion is then performed on the intermediate dataset, and a moving average algorithm is used to weight and combine the fused reconstructed noise to obtain the final reconstructed noise. Features are extracted from the final reconstructed noise using a diffusion model to determine a new intermediate dataset. This new intermediate dataset is then used as the final reconstructed noise dataset, and the feature fusion step is repeated until the cyclical feature extraction process stops. Finally, the final reconstructed noise is decoded to obtain a stylized image, thereby achieving image style transfer. This application's covariance alignment method based on Cholesky decomposition addresses the inability to capture multi-channel dependencies by modeling inter-channel correlations. Simultaneously, it ensures numerical stability through matrix positive definite constraints, avoiding the complexity and instability of traditional covariance calculations. Furthermore, the noise reconstruction module guided by frequency domain features separates the frequency components of noise through wavelet decomposition, replacing the high-frequency parts of content noise with the high-frequency components of style noise. This avoids artifacts caused by diffusion models at longer time steps while enhancing image texture details and style consistency. Finally, through cyclic reconstruction and feature fusion of the diffusion model, efficient style transfer can be achieved without additional training, making it suitable for scenarios with limited data or real-time style transformation, thus improving the method's generalization and practicality. Therefore, this application achieves efficient style transfer and enhances generalization and practicality. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 The flowchart shows a training-free image style transfer method based on diffusion model noise reconstruction.

[0027] Figure 2 The flowchart shows the framework of a training-free image style transfer method based on diffusion model noise reconstruction.

[0028] Figure 3 A flowchart corresponding to the covariance alignment method for Cholesky decomposition;

[0029] Figure 4 This is a schematic diagram of the network structure of a noise recombination module guided by frequency domain features.

[0030] Figure 5This is a structural diagram of a training-free image style transfer device based on diffusion model noise reconstruction.

[0031] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0033] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] To address the challenges of traditional training methods requiring model training on specific style data, and the lack of efficient alignment mechanisms in noise processing and feature fusion for training-free style transfer methods, which struggle to balance computational efficiency and transfer quality, this paper proposes a training-free image style transfer method and system based on diffusion model noise reconstruction. This application utilizes a covariance alignment method based on Cholesky decomposition to model inter-channel correlations, overcoming the inability of methods like AdaIN to capture multi-channel dependencies. Simultaneously, positive definite matrix constraints ensure numerical stability, avoiding the complexity and instability of traditional covariance calculations. The frequency-domain feature-guided noise reconstruction module separates the frequency components of noise through wavelet decomposition, replacing the high-frequency components of content noise with the high-frequency components of style noise. This avoids artifacts caused by long-term diffusion while enhancing image texture details and style consistency. Finally, through cyclic reconstruction of intermediate noise and feature fusion within the diffusion model, this application achieves efficient style transfer without additional training, making it suitable for scenarios with limited data or real-time style transformation, thus improving the method's generalization and practicality.

[0035] A Covariance Alignment (Transfer) Method Based on Cholesky Decomposition: In noise processing of diffusion models, the Adaptive Instance Normalization (AdaIN) method modifies style features through mean and variance adjustments. AdaIN's limitation is that it only captures the independent statistical characteristics of a single channel, without considering channel dependencies. The covariance matrix effectively encapsulates data relationships, especially dependencies between different image channels. However, its computational complexity reduces processing efficiency. Therefore, a covariance matching transformation module is designed. This module constructs a transformation matrix by decomposing the covariance matrix using the Cholesky decomposition method, and uses the transformation matrix to align noise. This method mitigates numerical instability by ensuring positive definiteness, improving image stylization quality while maintaining numerical stability and computational efficiency in style transfer.

[0036] A frequency-domain feature-guided noise reconstruction module: For style transfer tasks, it is necessary to ensure the personalized texture and brushstrokes of the generated image. Therefore, how to feed the style information of the style image into the network for learning is a problem worth considering in depth. In existing solutions, a relatively long diffusion model generation time step is generally set to allow the diffusion model to generate more image details. For style transfer tasks, although this can improve stylization performance, it leads to the continuous accumulation of inter-domain differences between content and style information, resulting in artifacts, blurring, and content distortion in the final generated image. Therefore, this application designs a frequency-domain reconstruction module for diffusion model noise. This module first performs wavelet decomposition on the noise of the diffusion model to extract its low-frequency and high-frequency components. The low-frequency components are further converted to Fourier domain modulation to enhance their color features; while the high-frequency components are replaced with high-frequency style components generated by the covariance alignment of style noise from the diffusion model noise addition process and content noise from the denoising process. Through the above reconstruction, the style texture details of the image can be effectively enhanced without accumulating inter-domain differences, thereby improving the overall stylization quality of the image.

[0037] In one exemplary embodiment, such as Figure 1 As shown, a training-free image style transfer method based on diffusion model noise reconstruction is provided, including:

[0038] Step 100: Obtain information data; information data includes: content images and style images.

[0039] Step 200: Encode the information data and use a diffusion model to perform a cyclical feature extraction process on the obtained latent space data to obtain an intermediate dataset. Feature extraction includes adding noise and denoising; the intermediate dataset includes noisy intermediate data and denoised intermediate data.

[0040] After the loop processing of any feature extraction:

[0041] Step 300: Feature fusion is performed based on the intermediate dataset, and the fused reconstructed noise is weighted and combined using a moving average algorithm to obtain the final reconstructed noise. Feature fusion is performed using the Cholesky covariance alignment method and a noise reconstruction module guided by frequency domain features.

[0042] Step 400: Use a diffusion model to extract features from the final recombined noise, and use the denoised intermediate data from the corresponding intermediate dataset as the content image denoising intermediate data contained in the intermediate dataset to obtain a new intermediate dataset.

[0043] Step 500: Use the new intermediate dataset as the intermediate dataset and return to the step "Fuse features based on the intermediate dataset and use the moving average algorithm to weight and combine the fused recombined noise to obtain the final recombined noise" until the feature extraction loop stops.

[0044] Step 600: Decode the final reconstructed noise to obtain a stylized image, thereby achieving image style transfer.

[0045] In one embodiment, the information data is encoded, and a diffusion model is used to perform a cyclical feature extraction process on the obtained latent space data to obtain an intermediate dataset, specifically including:

[0046] The information data is input into the encoder to obtain the latent space data.

[0047] A diffusion model is used to perform cyclic noise addition on the latent space data to obtain intermediate noise data. The stopping condition for the cyclic noise addition is that the noise is Gaussian noise. The intermediate noise data includes intermediate noise addition data of the content image and intermediate noise addition data of the style image obtained from each denoising process.

[0048] The Gaussian noise that has been stopped in the noise addition loop is subjected to a denoising loop to obtain denoised intermediate data. The stopping condition for the denoising loop is that the image is denoised. The denoising intermediate data includes the content image denoising intermediate data and the style image denoising intermediate data obtained from each denoising process.

[0049] The noisy intermediate data and the denoised intermediate data are defined as the intermediate dataset.

[0050] Specifically, feature fusion is performed based on the intermediate dataset, and a moving average algorithm is used to weight and combine the fused recombined noise to obtain the final recombined noise, which includes:

[0051] The covariance alignment method based on Cholesky decomposition is used to perform covariance alignment on the intermediate dataset to obtain aligned tensors.

[0052] A noise reconstruction module guided by frequency domain features is adopted to reconstruct noise from the aligned tensor. The noise reconstruction module includes a pooling module and an inverse pooling module guided by wavelet features. During the reconstruction process, the pooling module decomposes the aligned tensor into multiple frequency components, and the inverse pooling module reconstructs the multiple frequency components.

[0053] The weighted combination of the fused recombined noise is obtained by using a moving average algorithm.

[0054] Covariance alignment methods for Cholesky decomposition specifically include:

[0055] Based on the content tensor and style tensor, mean removal is performed separately to obtain processed data; the content tensor is the intermediate data of the content image with added noise; the style tensor is the intermediate data of the style image with added noise.

[0056] The covariance matrix is ​​solved based on the processed data, and the Cholesky decomposition method is used to decompose the covariance matrix into the product of two lower triangular matrices; the transformation matrix is ​​determined based on the product of the lower triangular matrices; the stylization features are determined based on the transformation matrix, and the aligned tensor is obtained.

[0057] The expression for a lower triangular matrix is:

[0058] L c =Cholesky(∑ c +∈I).

[0059] L s =Cholesky(∑ s +∈I).

[0060] The expression for the transformation matrix is:

[0061] M = L s ·L s -1 .

[0062] Among them, L c L is the lower triangular matrix obtained by Cholesky decomposition of the content tensor; s The lower triangular matrix is ​​obtained by Cholesky decomposition of the style tensor; Cholesky is the decomposition function corresponding to the Cholesky decomposition method; ∑ c Let be the covariance matrix corresponding to the content tensor; ∈ is the regularization constant; I is the identity matrix; ∑ s L is the covariance matrix corresponding to the style tensor;s -1 For L s The inverse matrix of ; M is the transformation matrix.

[0063] As an optional implementation, the noise reconstruction module guided by frequency domain features uses a Haar wavelet filter for wavelet transform and inverse wavelet transform. Specifically, the pooling module uses wavelet transform to decompose the aligned tensor into multiple frequency components, and Fourier adjustment is used to enhance the global structure and color of the included low-frequency components. The high-frequency components are decomposed, and the inverse pooling module uses inverse wavelet transform to reconstruct the enhanced low-frequency components and the decomposed high-frequency components, resulting in reconstructed noise. The reconstructed noise is a reconstructed stylized tensor.

[0064] This application includes two parts: feature extraction and feature fusion training. For example... Figure 2 As shown, in feature extraction, the content image and style image are first fed into the decoder. Then, a diffusion model is used to add and remove noise from the content and style images, extracting their intermediate noise data. Feature fusion is divided into two stages: Stage 1, using the intermediate data N' from the diffusion model denoising process. c and N′ respectively s N s After covariance alignment, the data is fed into the frequency domain reconstruction module. After multiple reconstructive iterations, the reconstructed noise is combined using a moving average connection module. In stage two, the reconstructed noise from stage one is fed into the diffusion model for feature extraction, utilizing newly acquired intermediate data N′. cs Again with N′ s N s The noise is then fused and reconstructed, and the resulting reconstructed noise is fed into the decoder to obtain a high-quality stylized image.

[0065] Feature extraction steps:

[0066] Step 1: Obtain content images and style images from the original dataset.

[0067] Step 2: Input the content image and style image into the encoder to obtain its latent space data.

[0068] Step 3: Add noise to the latent space data using a diffusion model. Each noise addition will produce an intermediate data point until the final noise is pure Gaussian noise.

[0069] Step 4: Denoise the pure Gaussian noise. Each denoising step will produce an intermediate data point until the final denoised image is obtained.

[0070] Feature fusion steps:

[0071] Step 1: Obtain intermediate data N′ respectively c and N′ s N s .

[0072] Step 2: Obtain N′ c and N′ s N s Perform covariance alignment separately to obtain two aligned tensors.

[0073] Step 3: Input the two tensors obtained in Step 2 into the frequency domain reconstruction module to obtain the reconstructed noise. This frequency domain reconstruction module mainly includes a wavelet feature-guided pooling module and an inverse pooling module. During the reconstruction process, pooling is first performed to decompose the noise into multiple frequency components, and then inverse pooling is performed to reconstruct the frequency components.

[0074] Step 4: Use the moving average algorithm to weight and combine the recombined noise from each step to obtain the final recombined noise. The moving average algorithm uses an existing data smoothing technique, which will not be elaborated on here.

[0075] Step 5: Use a diffusion model to extract features from the recombined noise to prepare for the next feature fusion.

[0076] Step 6: Utilize the intermediate data features N′ obtained from the recombination noise cs As the new N′ c Repeat steps 2, 3, and 4.

[0077] Step 7: Decode the final reconstructed noise to obtain a stylized image.

[0078] In this application's training-free image style transfer method based on diffusion model noise reconstruction, there are two key modules: a covariance alignment method based on Cholesky decomposition and a noise reconstruction module guided by frequency domain features. The usage methods and steps of these two modules in this application are described below:

[0079] (1) Covariance Alignment Method Based on Cholesky Decomposition: In style transfer research using diffusion models, a problem exists that commonly used adaptive normalization can only achieve color transfer, while other methods suffer from instability. Therefore, this application combines the covariance matrix with Cholesky decomposition to address this issue. The purpose of introducing the covariance matrix is ​​to improve style details in addition to color alignment, while Cholesky decomposition ensures the stability of the alignment process. The process of the covariance alignment method based on Cholesky decomposition is as follows: Figure 3As shown. The diffusion model is an algorithmic framework, typically implemented using various deep learning frameworks such as PyTorch and TensorFlow.

[0080] Step 1: Given a content tensor and a style tensor with C channels, H height and W width, focus them by subtracting the mean of the content and style noise respectively.

[0081]

[0082] Where, N c and N s For content tensor and style tensor, μ c and μ s These are their means. The processing data corresponding to the content tensor; This refers to the processing data corresponding to the style tensor.

[0083] Step 2: Apply the formula (1) to obtain and Solve for the corresponding covariance matrix respectively:

[0084]

[0085] Where, Σ c and Σ s Let the covariance matrices be the corresponding content tensor and style tensor. and for and The transpose of .

[0086] Step 3: Subsequently, the covariance matrix is ​​decomposed into the product of two lower triangular matrices using Cholesky decomposition, as shown in Equation (3):

[0087] L c =Cholesky(∑ c +∈I),L s =Cholesky(∑ s +∈I) (3)

[0088] Step 4: Calculate the transformation matrix by multiplying the lower triangular matrices to transform the features of the content image:

[0089] M = L s ·L s -1 (4)

[0090] Among them, L s -1 For L s The inverse matrix.

[0091] Step 5: Subsequently, stylized features are obtained by applying the transformation matrix to the features of the content image. During the adjustment process, the mean μ of the style tensor is re-added. s This makes the mean of the transformation features consistent with the style tensor:

[0092]

[0093] A Noise Reconstruction Module Guided by Frequency Domain Features: Existing research has demonstrated the crucial role of noise in diffusion models, and noise optimization can improve image generation. However, the denoising process prioritizes the generation of low-frequency signals, resulting in insufficient generation of high-frequency signals. Therefore, to enhance the high-frequency detail performance of the model, this application designs a noise reconstruction module guided by frequency domain features. This module aims to strengthen various frequency bands of the image to improve the perceptual quality of the generated image. The input of this module incorporates the covariance alignment method of Cholesky decomposition, and its network structure is as follows: Figure 3 As shown.

[0094] This module supplements the covariance alignment method of Cholesky decomposition with the fusion of frequency domain information. First, the low-frequency myopia component from style tensor one is Fourier-adjusted, and its low-frequency and high-frequency components are further enhanced in the Fourier domain. Then, the horizontal, vertical, and diagonal detail components obtained from the decomposition of style component two are recombined and merged. In this way, the final generated image incorporates high-frequency features from another tensor, improving the detail quality of the generated image without hindering network convergence during the generation process.

[0095] Step 1: The Haar wavelet filter is used as the basis for wavelet transform, which includes two processes: wavelet transform and inverse wavelet transform. The former transforms the image from the spatial domain to the frequency domain, while the latter reconstructs the frequency domain components back into the spatial domain. The Haar wavelet transform includes four kernels, namely LL... T LH T HL T and HH T .

[0096]

[0097] Where L and H represent low-pass and high-pass filters, respectively. T To convert the column vector of parameters of low-pass filter L into a row vector format; H T This is to convert the column vector of parameters of the high-pass filter H into a row vector format.

[0098] Low-pass filters primarily focus on low-frequency signals, such as contour and structural information. In contrast, high-pass filters emphasize high-frequency signals, capturing finer-grained details, such as subtle edges. Figure 4 The diagram shows the frequency components of a given image obtained through Haar wavelet transform, where LL represents the low-frequency component, and LH, HL, and HH represent the high-frequency components in the vertical, horizontal, and diagonal directions, respectively.

[0099] Wavelet transform decomposes the eigenvalues ​​into LL... T LH T HL T HH T Four frequency components. Low-frequency component LL T The main focus is on the macroscopic structure and global features of the image, including its layout and smooth color transitions, while the high-frequency component LH... T HL T HH T Capture fine details, such as edges and complex textures. Therefore, Fourier adjustment is used to enhance the low-frequency component (LL). T The global structure and color. Its enhanced low-frequency components can be represented as:

[0100] F(LL T ) = FFT(LL T (7)

[0101] F′(LL T )=F(LL T )⊙β (8)

[0102] LL T′ =IFFT(F′(LL) T (9)

[0103] Where FFT stands for Fourier Transform, β is the modulation coefficient, ⊙ represents element-wise multiplication, and IFFT stands for Inverse Fourier Transform. F(LL) T The data represents the low-frequency components after Fourier transform and entry into the Fourier domain. In the Fourier domain, β-multiplication modulation is performed, and the result is expressed as F′(LL). T ) indicates. LL T′ To perform an inverse Fourier transform on the data after modulation in the Fourier domain, so that it returns to the original domain data representation.

[0104] Step 2: For high-frequency details, decompose the stylization tensor 2, as it has a higher energy distribution, thus containing more stylistic details in its high-frequency components. Use its high-frequency component LH. T HL T HH T The enhanced low-frequency component LL obtained in step 1 T Performing an inverse wavelet transform yields the reconstructed stylized tensor. It can be expressed as:

[0105]

[0106] Where IDWT is the inverse wavelet transform. It is the reorganized style tensor.

[0107] Ultimately, this module can combine high-frequency detail information from another tensor during the image generation process, enhancing the detailed features of the generated image, thereby gradually outputting higher-quality features in the decoder.

[0108] This method primarily explores key issues in style transfer within image generation research. Currently, style transfer utilizes diffusion models to guide image generation because these models rely on large amounts of data, resulting in high-quality generated images. However, current style transfer tasks based on diffusion models face challenges such as insufficient style detail, image quality degradation, and overly complex modulation models. Numerous researchers have conducted in-depth studies on these issues from the perspectives of noise adjustment, model structure, and feature distillation. This method combines current mainstream noise adjustment techniques with relevant research from the perspective of frequency domain signal processing.

[0109] (1) Traditional AdaIN only considers single-channel statistics (mean, variance), while the covariance matrix captures cross-channel dependencies (such as correlations between color channels), enabling style transfer to extend beyond color transfer to include joint features between channels (such as texture structure). Cholesky decomposition avoids numerical instability issues (such as decomposition failure due to matrix singularity) by ensuring the positive definiteness of the covariance matrix, and improves computational efficiency while maintaining stability through efficient lower triangular matrix operations. Furthermore, we add a small regularization constant ∈ to the covariance matrix to force it to remain positive definite, avoiding the potential failure due to near-singularity in traditional covariance matrix decomposition. Moreover, the covariance alignment method based on Cholesky decomposition does not require complex iterative optimization; it directly constructs the transformation matrix by decomposing the lower triangular matrix (Equations 3 and 4), reducing computational redundancy.

[0110] (2) The signal analysis capability in the frequency domain decouples low-frequency and high-frequency features, thereby improving the quality of image generation. Therefore, from the perspective of the frequency domain, the multi-scale analysis capability of wavelet transform is used to decouple low-frequency and high-frequency features. Fourier modulation enhances the global features of low frequencies, while the direct recombination of stylized high-frequency components avoids the accumulation of differences between content and style domains caused by traditional long-step diffusion, thereby improving detail quality and suppressing artifacts.

[0111] In the noise addition and denoising process of the diffusion model, multi-stage intermediate data is extracted. This data naturally contains content and style features at different noise scales, providing rich material for fine-grained feature alignment (such as covariance alignment) and recombination. Through multiple covariance alignments and frequency domain recombinations, style features are gradually accumulated, avoiding insufficient style transfer caused by a single transformation. At the same time, by using a moving average connection module to weight and combine the noise from multiple rounds of recombination, the feature distribution is smoothed, reducing generation fluctuations introduced by random noise, and allowing style features to be more stably integrated into the content structure.

[0112] Furthermore, we can further explore the internal characteristics of the diffusion model, such as using attention mechanisms (self-attention, cross-attention) to replace the covariance alignment module, and achieving feature interaction across channels and spatial locations by calculating the attention weights of content and style noise.

[0113] In one exemplary embodiment, such as Figure 5 As shown, a training-free image style transfer device based on diffusion model noise reconstruction is provided, comprising:

[0114] The information data acquisition module is used to acquire information data, which includes content images and style images.

[0115] The feature extraction module is used to encode information data and perform cyclic processing of feature extraction on the obtained latent space data using a diffusion model to obtain an intermediate dataset. Feature extraction includes noise addition and noise removal. The intermediate dataset includes noisy intermediate data and denoised intermediate data.

[0116] The feature fusion module is used to perform feature fusion based on the intermediate dataset after any feature extraction loop. The fused reconstructed noise is weighted and combined using a moving average algorithm to obtain the final reconstructed noise. The feature fusion is performed using the covariance alignment method of Cholesky decomposition and a noise reconstruction module guided by frequency domain features.

[0117] The replacement module is used to extract features from the final recombined noise using a diffusion model, and to obtain the denoised intermediate data from the corresponding intermediate dataset, which is then used as the content image denoising intermediate data contained in the denoised intermediate data of the intermediate dataset, thus obtaining a new intermediate dataset.

[0118] The return module is used to take the new intermediate dataset as an intermediate dataset and return to the "feature fusion module" until the loop processing of feature extraction stops.

[0119] The decoding module is used to decode the final reconstructed noise to obtain a stylized image, thereby achieving image style transfer.

[0120] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores training-free image style transfer data based on diffusion model noise reconstruction. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a training-free image style transfer method based on diffusion model noise reconstruction.

[0121] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0122] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0123] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0124] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0125] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0126] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0127] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0128] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A training-free image style transfer method based on diffusion model noise reconstruction, characterized in that, include: Acquire information and data; The information data includes: content images and style images; The information data is encoded, and a diffusion model is used to perform a cyclical feature extraction process on the obtained latent space data to obtain an intermediate dataset; the feature extraction includes: adding noise and denoising; the intermediate dataset includes: noisy intermediate data and denoised intermediate data; After the loop processing of any feature extraction: Feature fusion is performed based on the intermediate dataset, and the reconstructed noise obtained by fusion is weighted and combined using a moving average algorithm to obtain the final reconstructed noise; wherein, the feature fusion is performed using the covariance alignment method of Cholesky decomposition and a noise reconstruction module guided by frequency domain features. The diffusion model is used to extract features from the final recombined noise, and the denoised intermediate data in the corresponding intermediate dataset is used as the content image denoising intermediate data contained in the intermediate dataset to obtain a new intermediate dataset. The new intermediate dataset is used as the intermediate dataset, and the process returns to the step of "performing feature fusion based on the intermediate dataset and using a moving average algorithm to weight and combine the fused recombined noise to obtain the final recombined noise" until the feature extraction loop stops. The final reconstructed noise is decoded to obtain a stylized image, thereby achieving image style transfer.

2. The training-free image style transfer method based on diffusion model noise reconstruction according to claim 1, characterized in that, The information data is encoded, and a diffusion model is used to perform cyclical feature extraction on the obtained latent space data to obtain an intermediate dataset, specifically including: The information data is input into the encoder to obtain latent space data; A diffusion model is used to perform a noisy loop processing on the latent space data to obtain noisy intermediate data; the stopping condition for the noisy loop processing is that the noise is Gaussian noise; the noisy intermediate data includes noisy intermediate data of the content image and noisy intermediate data of the style image obtained from each noisy processing; A denoising loop is performed on the Gaussian noise that has stopped the denoising loop to obtain intermediate denoised data; the stopping condition of the denoising loop is that the denoised image is obtained; the intermediate denoised data includes intermediate content image denoising data and intermediate style image denoising data obtained from each denoising process. The noisy intermediate data and the denoised intermediate data are determined as the intermediate dataset.

3. The training-free image style transfer method based on diffusion model noise reconstruction according to claim 2, characterized in that, Feature fusion is performed based on the intermediate dataset, and the recombined noise obtained by the fusion is weighted and combined using a moving average algorithm to obtain the final recombined noise, specifically including: The intermediate dataset is covariance aligned using the Cholesky decomposition method to obtain aligned tensors. A noise reconstruction module guided by frequency domain features is used to reconstruct noise from the aligned tensor to obtain reconstructed noise. The noise reconstruction module includes a pooling module and an inverse pooling module guided by wavelet features. During the reconstruction process, the pooling module decomposes the aligned tensor into multiple frequency components, and the inverse pooling module reconstructs the multiple frequency components. The weighted combination of the fused recombined noise is obtained by using a moving average algorithm.

4. The training-free image style transfer method based on diffusion model noise reconstruction according to claim 3, characterized in that, Covariance alignment methods for Cholesky decomposition specifically include: Based on the content tensor and style tensor, mean removal processing is performed separately to obtain processed data; the content tensor is the intermediate data of the content image with added noise; the style tensor is the intermediate data of the style image with added noise. The covariance matrix is ​​solved based on the processed data, and the Cholesky decomposition method is used to decompose the covariance matrix into the product of two lower triangular matrices. The transformation matrix is ​​determined based on the product of the lower triangular matrices; The stylization features are determined based on the transformation matrix, and the aligned tensor is obtained.

5. The training-free image style transfer method based on diffusion model noise reconstruction according to claim 4, characterized in that, The expression for a lower triangular matrix is: L c =Cholesky(∑ c +∈I); L s =Cholesky(∑ s +∈I); The expression for the transformation matrix is: M=L s ·L s -1 ; Among them, L c L is the lower triangular matrix obtained by Cholesky decomposition of the content tensor; s The lower triangular matrix is ​​obtained by Cholesky decomposition of the style tensor; Cholesky is the decomposition function corresponding to the Cholesky decomposition method; ∑ c Let be the covariance matrix corresponding to the content tensor; ∈ is the regularization constant; I is the identity matrix; ∑ s L is the covariance matrix corresponding to the style tensor; s -1 For L s The inverse matrix of ; M is the transformation matrix.

6. The training-free image style transfer method based on diffusion model noise reconstruction according to claim 3, characterized in that, The noise reconstruction module guided by frequency domain features uses Haar wavelet filters for wavelet transform and inverse wavelet transform; Among them, the alignment tensor is decomposed into multiple frequency components by wavelet transform based on the pooling module, and the low-frequency components are enhanced globally in terms of structure and color by Fourier adjustment. The high-frequency components contained in the frequency components are decomposed, and the low-frequency components that have been enhanced and the decomposed high-frequency components are recombined using the inverse wavelet transform method based on the inverse pooling module to obtain recombined noise; the recombined noise is a recombined stylized tensor.

7. A training-free image style transfer device based on diffusion model noise reconstruction, characterized in that, include: Information data acquisition module, used to acquire information data; The information data includes: content images and style images; The feature extraction module is used to encode the information data and perform a cyclical feature extraction process on the obtained latent space data using a diffusion model to obtain an intermediate dataset; the feature extraction includes: adding noise and denoising; the intermediate dataset includes: noisy intermediate data and denoised intermediate data; The feature fusion module is used to perform feature fusion based on the intermediate dataset after any feature extraction loop, and to use a moving average algorithm to weight and combine the fused reconstructed noise to obtain the final reconstructed noise; wherein, the feature fusion is performed using the covariance alignment method of Cholesky decomposition and a noise reconstruction module guided by frequency domain features. The replacement module is used to extract features from the final recombined noise using a diffusion model, and to obtain the denoised intermediate data in the corresponding intermediate dataset as the content image denoising intermediate data contained in the denoised intermediate data in the intermediate dataset, so as to obtain a new intermediate dataset. The return module is used to take the new intermediate dataset as an intermediate dataset and return to the "feature fusion module" until the loop processing of feature extraction stops; The decoding module is used to decode the final reconstructed noise to obtain a stylized image, thereby achieving image style transfer.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the trainingless image style transfer method based on diffusion model noise reconstruction as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the training-free image style transfer method based on diffusion model noise reconstruction as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the training-free image style transfer method based on diffusion model noise reconstruction as described in any one of claims 1-6.

Images