An image style transfer method and device based on a self-learning diffusion model and a storage medium
By employing a self-learning style encoder based on a self-learning diffusion model and decoupled cross-attention technology, the efficiency and quality issues of image style transfer in existing technologies are resolved. This achieves efficient and accurate decoupling and fusion of style and content, generating high-fidelity style-transferred images.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING NORMAL UNIVERSITY
- Filing Date
- 2026-03-02
- Publication Date
- 2026-06-16
AI Technical Summary
Existing diffusion-based image style transfer methods require large-scale paired datasets and cumbersome fine-tuning, making it difficult to achieve efficient and accurate style and content decoupling, and the generated images are prone to visual artifacts and content deviations.
A self-learning diffusion model is adopted, which extracts multi-dimensional content features and pure style features through a self-learning style encoder. Combined with decoupled cross attention and ControlNet branches, a seamless fusion of style and content is achieved to generate high-fidelity style transfer images.
It achieves high style fidelity and high content structure integrity in image style transfer without the need for large-scale datasets and fine-tuning, avoiding content leakage and visual artifacts, and improving generation quality.
Smart Images
Figure CN122222804A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and deep learning technology, and in particular to an image style transfer method, device and storage medium based on a self-learning diffusion model. Background Technology
[0002] Image style transfer is a core task in computer vision and graphics, aiming to apply the artistic style of a reference image to a target content image while preserving the semantic structure of the content image. With the development of deep learning technology, the need for image stylization has demonstrated enormous application value in fields such as digital art creation, film and television post-processing, and personalized content generation.
[0003] Early style transfer methods were primarily based on convolutional neural networks, achieving this by matching the statistical distributions of content features with style features. Subsequently, generative adversarial networks (GANs) improved the visual quality of generated images by learning the distributions of specific style domains. However, these traditional methods often face challenges when dealing with complex and nuanced artistic styles, such as impure style extraction, distorted content structure, and unstable generation quality, making it difficult to achieve an ideal balance between maintaining content fidelity and enhancing stylization intensity.
[0004] In recent years, diffusion models have opened up new avenues for high-quality image generation due to their superior generative capabilities and versatility. Nevertheless, applying diffusion models to image style transfer still faces technical bottlenecks. Existing diffusion-based methods typically require fine-tuning of the model or rely heavily on textual descriptions, often resulting in extracted style features mixed with content information from the original image, or generated content deviating from the original structure. Many advanced methods require long-term training with multiple images for a single style, making it difficult to achieve efficient and accurate transfer using only a single reference image. A major challenge in the diffusion process is how to seamlessly integrate highly decoupled content features with pure style features without producing visual artifacts.
[0005] Therefore, developing a style transfer method that can achieve deep decoupling of style and content without the need for large-scale pairwise datasets and cumbersome fine-tuning, and that can generate style transfer methods with both high style fidelity and high content structure integrity, has become an urgent need in the field of image stylization. Summary of the Invention
[0006] Purpose of the invention: This invention provides an image style transfer method, device, and storage medium based on a self-learning diffusion model. It achieves deep decoupling of styles through a self-learning style encoder and accurately integrates structural and style features during the diffusion generation process, thereby achieving high-fidelity artistic style transfer.
[0007] Technical solution: The image style transfer method based on a self-learning diffusion model described in this invention includes the following steps:
[0008] Step 1: Obtain the image to be processed and perform preprocessing. The image to be processed includes the content image and the style image.
[0009] Step 2: Construct a content preservation module to extract multi-dimensional content features;
[0010] Step 3: Construct a self-learning style encoder based on residual blocks and Transformer layer coding architecture, and extract style features;
[0011] Step 4: Construct a diffusion model generation network based on decoupled cross-attention to process content features and style features;
[0012] Step 5: Generate the final stylized image and evaluate the performance of the generated image in terms of style fidelity and content preservation.
[0013] Furthermore, in step 2, the multidimensional content features include structural features and semantic features. Adaptive Canny edge detection processing is performed on the content image. First, the content image undergoes grayscale preprocessing. Then, the pixel grayscale distribution pattern of the image is statistically analyzed using grayscale histograms. Based on the peaks, valleys, and pixel proportions of each grayscale interval, a threshold suitable for the image's own features is calculated. Then, gradient calculation, non-maximum suppression, and threshold detection are sequentially performed to generate a high-precision image edge map, which is the extracted structural feature. This process completely excludes the original color information of the content image, retaining only the spatial contour and skeletal structure of the image.
[0014] Furthermore, semantic features of the content image are extracted and input into a pre-trained BLIP-2 multimodal model. The prompt "A photo of" triggers the model's zero-shot inference capability. The model then accurately represents the core content of the image using natural language and generates corresponding content description text, which constitutes the extracted semantic features. This feature ensures that the generation model accurately captures and understands the core semantic objects of the image, effectively avoiding the interference of style noise on semantic understanding.
[0015] Furthermore, in step 3, the self-learning style encoder includes a low-level feature extraction sub-network of four cascaded residual blocks (ResBlock), a feature dimension adaptation layer, a learnable query, a high-level style encoding sub-network of two Transformer Encoders, and a style feature normalization output layer connected in sequence.
[0016] Furthermore, a random masking operation is performed on the input style image to generate a mask image with a specified masking ratio, constructing a pseudo-training sample pair between the original style image and the mask image. This sample pair is then sequentially input into the four cascaded residual blocks (ResBlocks) of the self-learning encoder to obtain a low-level feature map of a specified dimension. The low-level feature map is then adapted to the Transformer Encoder layer through a feature dimension adaptation layer. Finally, it is concatenated with a learnable query and input into the Transformer Encoder layer and the style feature normalization output layer. Ultimately, this enables the encoder to extract content information and pure, fine-grained style features from the style image.
[0017] Furthermore, a multi-self-supervised loss function is constructed to calculate the feature consistency loss. Used to constrain mask image features Features of the original image Cosine similarity; introducing mean alignment loss Alignment loss with variance The details are as follows:
[0018]
[0019]
[0020]
[0021] in, and Let the mean and variance of the feature be respectively represented by the channel dimension; the resulting total loss function is: .
[0022] Furthermore, in step 4, a diffusion model generation network based on decoupled cross-attention is constructed to process content features and style features. The semantic features in the content features and the style embedding vector output by the self-learning style encoder are input into the decoupled cross-attention module, and independent key-value pairs are constructed for the semantic features and style embedding vectors respectively. They are modeled separately through parallel attention branches, and then the outputs of the two branches are weighted and fused by adjustable style control coefficients to ensure the dominance of content semantics while achieving interference-free injection of style features. The structural features in the content features and the style embedding vector channels are concatenated and input into the ControlNet branch. The generated residual signal is injected into each scale level of U-Net to strongly constrain the integrity of the content structure. The style information is continuously effective at multiple scales during the denoising process through downsampling, intermediate layers, and upsampling at all scale levels of U-Net. Finally, random Gaussian noise is used as the initial input to complete multiple steps of denoising iteration to generate the final stylized image.
[0023] Furthermore, in step 5, the final stylized image is generated. The performance of the generated image in terms of style fidelity and content preservation is evaluated from three dimensions: style similarity, content preservation, and quality of generated results. In terms of style similarity, the CLIP-S style similarity index is used. The feature vectors of the generated image and the reference style image are extracted by the pre-trained CLIP model, and the cosine similarity between the two is calculated and the average value is taken. The higher the value, the higher the matching degree of the style features of the generated image with the reference style. In terms of content preservation, the SSIM structural similarity index is used. SSIM calculates the similarity between the generated image and the original content image from three core dimensions: brightness, contrast, and structure. The higher the value, the more complete the spatial structure and geometric contour are preserved, and the less the content structure distortion is, thus comprehensively measuring the content preservation effect. In terms of quality of generated results, the Aesthetic aesthetic rating index is used. Aesthetic evaluates the aesthetic attributes of the image from dimensions such as color harmony, texture naturalness, and artistic expression. The higher the value, the better the generation quality.
[0024] Accordingly, an image style transfer device based on a self-learning diffusion model includes: one or more processors;
[0025] Storage device for storing one or more programs or user data;
[0026] When the one or more programs are executed by one or more processors, the one or more processors implement an image style transfer method based on a self-learning diffusion model.
[0027] Correspondingly, an image style transfer storage medium based on a self-learning diffusion model is provided, on which a computer program is stored, which, when executed by a processor, implements an image style transfer method based on a self-learning diffusion model.
[0028] Beneficial Effects: Compared with the prior art, the present invention has the following significant advantages: The present invention greatly alleviates the content leakage problem in style transfer through the synergistic effect of a self-learning style encoder, edge detection, and a diffusion generation network; the self-learning style encoder purifies style features and removes content information from style images; edge detection extracts pure structural features from content images, eliminates style interference, and constrains content integrity; the diffusion generation network, through decoupling cross-attention and ControlNet branch design, allows pure style and pure content features to be accurately matched and do not interfere with each other, ensuring that style transfer only acts on the style expression level, without destroying the content structure or leaking redundant content of the style image, achieving the dual effect of style fidelity and content preservation. Attached Figure Description
[0029] Figure 1 This is a schematic diagram of the method flow of the present invention.
[0030] Figure 2 This is a structural and design layout diagram of the model used in this invention. Detailed Implementation
[0031] like Figure 1 As shown, an image style transfer method based on a self-learning diffusion model includes the following steps:
[0032] Step 1: Obtain the image to be processed and perform preprocessing. The image to be processed includes the content image and the style image.
[0033] Step 2: Construct a content preservation module to extract multi-dimensional content features;
[0034] Step 3: Construct a self-learning style encoder based on residual blocks and Transformer layer coding architecture, and extract style features;
[0035] Step 4: Construct a diffusion model generation network based on decoupled cross-attention to process content features and style features;
[0036] Step 5: Generate the final stylized image and evaluate the performance of the generated image in terms of style fidelity and content preservation.
[0037] In step 2, the multidimensional content features include structural and semantic features. Adaptive Canny edge detection is performed on the content image. First, the content image undergoes grayscale preprocessing. Then, the pixel grayscale distribution pattern is statistically analyzed using a grayscale histogram. Based on the peaks, valleys, and pixel proportions of each grayscale interval, a threshold suitable for the image's own features is calculated. Next, gradient calculation, non-maximum suppression, and threshold detection are sequentially performed to generate a high-precision image edge map, which represents the extracted structural features. This process completely excludes the original color information of the content image, retaining only the spatial contour and skeletal structure of the image.
[0038] Semantic features are extracted from the content image. This content image is then input into a pre-trained BLIP-2 multimodal model. The prompt "A photo of" triggers the model's zero-shot inference capability. The model accurately represents the core content of the image using natural language and generates corresponding content description text, which constitutes the extracted semantic features. This feature ensures that the generation model accurately captures and understands the core semantic objects of the image, effectively avoiding the interference of style noise on semantic understanding.
[0039] In step 3, the self-learning style encoder consists of a low-level feature extraction sub-network of four cascaded residual blocks (ResBlock), a feature dimension adaptation layer, a learnable query, a high-level style encoding sub-network of two Transformer Encoders, and a style feature normalization output layer, which are connected in sequence.
[0040] A random masking operation is performed on the input style image to generate a mask image with a specified mask ratio. A pseudo-training sample pair between the original style image and the mask image is constructed. Then, this sample pair is sequentially input into the four cascaded residual blocks ResBlock of the self-learning encoder to obtain the low-level feature map of the specified dimension. The low-level feature map is then adapted to the dimension of the Transformer Encoder layer through a feature dimension adaptation layer. Finally, it is concatenated with the learnable query and input into the Transformer Encoder layer and the style feature normalization output layer. Ultimately, the encoder is able to extract pure, fine-grained style features from the style image.
[0041] Construct a multi-self-supervised loss function and calculate the feature consistency loss. Used to constrain mask image features Features of the original image Cosine similarity; introducing mean alignment loss Alignment loss with variance The details are as follows:
[0042]
[0043]
[0044]
[0045] in, and Let the mean and variance of the feature be respectively represented by the channel dimension; the resulting total loss function is: .
[0046] In step 4, a diffusion model generative network based on decoupled cross-attention is constructed to process content features and style features. The semantic features in the content features and the style embedding vector output by the self-learning style encoder are input into the decoupled cross-attention module to construct independent key-value pairs for the semantic features and style embedding vectors respectively. These are modeled separately through parallel attention branches, and then the outputs of the two branches are weighted and fused by adjustable style control coefficients to ensure the dominance of content semantics while achieving interference-free injection of style features. The structural features in the content features and the style embedding vector channels are concatenated and input into the ControlNet branch. The generated residual signal is injected into each scale layer of U-Net to strongly constrain the integrity of the content structure. Style information is continuously effective at multiple scales during the denoising process through downsampling, intermediate layers, and upsampling at all scale levels of U-Net. Finally, random Gaussian noise is used as the initial input to complete multiple steps of denoising iteration to generate the final stylized image.
[0047] In step 5, the final stylized image is generated. The performance of the generated image in terms of style fidelity and content preservation is evaluated from three dimensions: style similarity, content preservation, and quality of generated result. For style similarity, the CLIP-S style similarity index is used. The feature vectors of the generated image and the reference style image are extracted using a pre-trained CLIP model. The cosine similarity between the two is calculated and averaged. A higher value indicates a higher match between the style features of the generated image and the reference style. For content preservation, the SSIM structural similarity index is used. SSIM calculates the similarity between the generated image and the original content image from three core dimensions: brightness, contrast, and structure. A higher value indicates more complete preservation of spatial structure and geometric contours, and less distortion of content structure, comprehensively measuring the content preservation effect. For quality of generated result, the Aesthetic aesthetic rating index is used. Aesthetic evaluates the aesthetic attributes of the image from dimensions such as color harmony, texture naturalness, and artistic expression. A higher value indicates better generation quality.
[0048] The model described above uses pre-trained IP-Adapter parameters and Stable Diffusion V1.5 parameters, and the validation of each module is based on this set of parameters. The experimental input image size was set to 512×512×3, and the random mask block size was specified as 72×72, while ensuring that there were no overlapping areas between the mask blocks; after modifying 40% of the pixel values in the image to (0, 0, 0), the mask image was input into the SLS-Encoder network.
[0049] To achieve rapid model convergence, the style-aware encoder provided by the Hugging Face platform was used as the initial parameters for the self-learning style encoder. This leverages prior knowledge of the parameters to avoid parameter oscillations during training from scratch. Subsequent adjustments were made only to the innovative layer parameters of the self-learning style encoder, such as the feature dimension adaptation layer and learnable queries, significantly improving convergence speed without compromising the effectiveness of the core innovative design. The frozen parameters were used to initialize the residual block layer and the Transformer encoder layer. Model training was performed using two A40-48G graphics cards, with a batch size of 16 per card. The AdamW optimizer was selected, with a learning rate of 0.0001, and a total of 6000 training iterations.
[0050] Finally, this invention is comprehensively compared with mainstream advanced methods from three dimensions: style similarity, content preservation, and the quality of generated results. AdaIN is a traditional convolutional method, StyTr2 is a Transformer encoding method, while DreamBooth, InS, and StyleID are all diffusion model-based methods. To ensure fairness, all methods were tested on the same hardware device using the model parameters provided by the official documentation. Quantitative comparison results are detailed in Table 1. Experimental results show that this invention achieves an optimal balance between style similarity and content preservation while maintaining the quality of diffusion model generation, and its overall style transfer performance is significantly better than the compared methods.
[0051] Table 1 Test Result Comparison Table
[0052]
Claims
1. An image style transfer method based on a self-learning diffusion model, characterized in that, Includes the following steps: Step 1: Obtain the image to be processed and perform preprocessing. The image to be processed includes the content image and the style image. Step 2: Construct a content preservation module to extract multi-dimensional content features; Step 3: Construct a self-learning style encoder based on residual blocks and Transformer layer coding architecture, and extract style features; Step 4: Construct a diffusion model generation network based on decoupled cross-attention to process content features and style features; Step 5: Generate the final stylized image and evaluate the performance of the generated image in terms of style fidelity and content preservation.
2. The image style transfer method based on a self-learning diffusion model as described in claim 1, characterized in that, In step 2, the multidimensional content features include structural features and semantic features; Adaptive Canny edge detection processing is performed on the content image. First, the content image is preprocessed by grayscale conversion. Then, the pixel grayscale distribution pattern of the image is statistically analyzed by grayscale histogram. Based on the peak value, valley value and pixel ratio of each grayscale interval, a threshold that adapts to the image's own features is calculated. Then, the gradient calculation, non-maximum suppression and threshold detection of edge detection are performed in sequence to generate a high-precision image edge map, which is the extracted structural feature.
3. The image style transfer method based on a self-learning diffusion model as described in claim 2, characterized in that, Semantic features of the content image are extracted and input into the pre-trained BLIP-2 multimodal model. The zero-shot reasoning ability of the model is triggered by the prompt word "A photo of". The model performs accurate natural language representation of the core content of the image and generates corresponding content description text, which is the extracted text semantic features.
4. The image style transfer method based on a self-learning diffusion model as described in claim 1, characterized in that, In step 3, the self-learning style encoder consists of a low-level feature extraction sub-network of four cascaded residual blocks (ResBlock), a feature dimension adaptation layer, a learnable query, a high-level style encoding sub-network of two Transformer Encoders, and a style feature normalization output layer, which are connected in sequence.
5. The image style transfer method based on a self-learning diffusion model as described in claim 4, characterized in that, A random masking operation is performed on the input style image to generate a mask image with a specified mask ratio. A pseudo-training sample pair between the original style image and the mask image is constructed. Then, this sample pair is sequentially input into the four cascaded residual blocks ResBlock of the self-learning encoder to obtain the low-level feature map of the specified dimension. The low-level feature map is then adapted to the dimension of the TransformerEncoder layer through a feature dimension adaptation layer. Finally, it is concatenated with the learnable query and input into the TransformerEncoder layer and the style feature normalization output layer. Ultimately, the encoder is able to extract pure, fine-grained style features from the style image.
6. The image style transfer method based on a self-learning diffusion model as described in claim 5, characterized in that, Construct a multi-self-supervised loss function and calculate the feature consistency loss. Used to constrain mask image features Features of the original image Cosine similarity; introducing mean alignment loss Alignment loss with variance The details are as follows: in, and Let the mean and variance of the feature be respectively represented by the channel dimension; the resulting total loss function is: .
7. The image style transfer method based on a self-learning diffusion model as described in claim 1, characterized in that, In step 4, a diffusion model generative network based on decoupled cross-attention is constructed to process content features and style features. The semantic features in the content features and the style embedding vector output by the self-learning style encoder are input into the decoupled cross-attention module to construct independent key-value pairs for the semantic features and style embedding vectors respectively. These are modeled separately through parallel attention branches, and then the outputs of the two branches are weighted and fused by adjustable style control coefficients to ensure the dominance of content semantics while achieving interference-free injection of style features. The structural features in the content features and the style embedding vector channels are concatenated and input into the ControlNet branch. The generated residual signal is injected into each scale layer of U-Net to strongly constrain the integrity of the content structure. Style information is continuously effective at multiple scales during the denoising process through downsampling, intermediate layers, and upsampling at all scale levels of U-Net. Finally, random Gaussian noise is used as the initial input to complete multiple steps of denoising iteration to generate the final stylized image.
8. The image style transfer method based on a self-learning diffusion model as described in claim 1, characterized in that, In step 5, the final stylized image is generated. The performance of the generated image in terms of style fidelity and content preservation is evaluated from three dimensions: style similarity, content preservation, and quality of generated result. For style similarity, the CLIP-S style similarity index is used. The feature vectors of the generated image and the reference style image are extracted using a pre-trained CLIP model. The cosine similarity between the two is calculated and averaged. A higher value indicates a higher match between the style features of the generated image and the reference style. For content preservation, the SSIM structural similarity index is used. SSIM calculates the similarity between the generated image and the original content image from three core dimensions: brightness, contrast, and structure. A higher value indicates more complete preservation of spatial structure and geometric contours, and less distortion of content structure, comprehensively measuring the content preservation effect. For quality of generated result, the Aesthetic aesthetic rating index is used. Aesthetic evaluates the aesthetic attributes of the image from dimensions such as color harmony, texture naturalness, and artistic expression. A higher value indicates better generation quality.
9. An image style transfer device based on a self-learning diffusion model, comprising: One or more processors; Storage device for storing one or more programs or user data; When the one or more programs are executed by one or more processors, the one or more processors implement the image style transfer method based on a self-learning diffusion model as described in any one of claims 1 to 8.
10. An image style transfer storage medium based on a self-learning diffusion model, wherein a computer program is stored thereon, which, when executed by a processor, implements the image style transfer method based on a self-learning diffusion model as described in any one of claims 1 to 8.