An image style transfer method based on diffusion model structure regularization

Abstract

This invention discloses an image style transfer method based on diffusion model structure regularization, comprising: 1. acquiring and preprocessing a modern face image from the source domain and a Terracotta Warrior image from the target domain to construct an unpaired cross-domain training dataset including images from the source and target domains; 2. constructing an image cross-domain transfer network based on a pre-trained CycleGAN-Turbo diffusion model, including a generator core integrating an attention reweighting module, a hierarchical text prompting module, and a multi-object loss calculation module, wherein the generator core includes a VAE encoder, a U-Net denoising network, a cross-attention layer, an attention reweighting module, and a VAE decoder; 3. training the image cross-domain transfer network based on the pre-trained CycleGAN-Turbo diffusion model; 4. inputting the face image to be converted into the image cross-domain transfer network trained in step 3, and outputting a style-transferred image from face to Terracotta Warrior. This invention suppresses global structural collapse and texture mismatch in style transfer from face to Terracotta Warrior, improving the quality of the generated image.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically an image style transfer method based on diffusion model structure regularization. Background Technology

[0002] The Terracotta Army is a brilliant achievement of archaeological work in New China. Its unique artistic style and morphological features have become a symbol of Chinese history and culture, attracting worldwide attention. As an important element of cultural dissemination and artistic innovation, the Terracotta Army holds an irreplaceable position and value in the study of ancient Chinese military, historical, and cultural fields. Combining modern human faces with the style of the Terracotta Army is an innovative attempt to integrate traditional culture with modern technology, providing new ideas for bridging the gap between people and cultural relics and for the digital protection of cultural heritage.

[0003] With the development of computer vision, deep learning technology, especially generative adversarial networks and diffusion models, has made significant progress in image generation and style transfer, greatly improving the accuracy and expressiveness of image style transfer. However, existing image translation methods in the field of style transfer mainly focus on simple stylization processing (e.g., converting photographs into painting styles), and style transfer research is mainly concentrated on areas such as animal poses and landscapes, lacking specificity for capturing and transferring facial features. When dealing with complex facial structures, problems such as structural collapse, texture mismatch, and memory overflow caused by introducing external priors exist, making it difficult for the generated images to meet the requirements of high-quality transfer.

[0004] The complex task of combining face and Terracotta Warrior styles involves intricate feature extraction and style reconstruction, placing higher demands on the training data and algorithm design of existing style transfer models. Current research on face-to-Terracotta Warrior style transfer is still in its early stages, lacking style transfer methods that are deeply adapted and optimized for the specific cultural and artistic style of the Terracotta Warriors. There is no mature technology that can achieve high-quality, high-similarity style transfer between faces and Terracotta Warriors. In addition, most of the existing face and Terracotta Warrior style transfer research is based on publicly available face datasets from abroad, which are mainly Western faces and lack coverage of Asian facial features. Moreover, the face-to-Terracotta Warrior style transfer process is quite complex and cannot accurately capture the fusion features of face and Terracotta Warrior styles.

[0005] Therefore, it is necessary to propose a style transfer method that can adapt to style transfer tasks with huge semantic gaps and local structural deformations, so as to overcome the problems of structural collapse, texture mismatch and memory overflow caused by the introduction of external priors when dealing with complex facial structures in existing style transfer methods. This method simplifies the style transfer process from human face to Terracotta Warrior and improves the similarity and style consistency of generated images. This has important academic value and social significance for promoting the application of technology in the field of culture and art. Summary of the Invention

[0006] The purpose of this invention is to provide an image style transfer method based on diffusion model structure regularization, which not only eliminates the need for external geometric priors, but also successfully suppresses global structural collapse and texture mismatch in the style transfer process from human face to Terracotta Warrior, thereby improving the quality of the generated image.

[0007] This invention is achieved through the following technical solution: An image style transfer method based on diffusion model structure regularization includes the following steps: Step 1: Obtain modern human face images from the source domain and terracotta warrior images from the target domain, and perform preprocessing to construct an unpaired cross-domain training dataset that includes images from the source and target domains. Step 2: Construct an image cross-domain transfer network based on the pre-trained CycleGAN-Turbo diffusion model, including a generator core integrating an attention reweighting module, a hierarchical text prompting module, and a multi-objective loss calculation module, wherein: The scheduler of the hierarchical text prompting module generates conditional texts with different emphases based on the current training progress. These conditional texts, as prompt words, are converted into conditional embedding feature vectors by the CLIP text encoder and used as key-value pairs. Injected into the cross-attention layer of the U-Net network; The generator core includes a VAE encoder, a U-Net denoising network, a cross-attention layer, an attention reweighting module, and a VAE decoder. Specifically: the VAE encoder maps the source domain image to the latent space to obtain latent variables; the U-Net denoising network extracts multiple features from the latent variables to obtain the original latent features; during the forward denoising process of U-Net, the original latent features, as query vectors, interact with the externally injected text conditional embedding key matrix at the cross-attention layer, calculating the original cross-attention map through matrix multiplication; the attention reweighting module captures the original cross-attention map and calculates its spatial information entropy, dynamically generating a reweighting mask to adaptively redistribute attention weights, outputting an enhanced attention map; the VAE decoder upsamples the denoised latent features output by the U-Net denoising network, restoring them to a pixel-level Terracotta Warriors-style image. The multi-target loss calculation module calculates the training loss of the image cross-domain transfer network, updates the network parameters through the error backpropagation mechanism, and forces the network to converge to a concentrated weight distribution. Step 3: Train an image cross-domain transfer network based on a pre-trained CycleGAN-Turbo diffusion model; Step 4: Input the face image to be converted into the image cross-domain transfer network trained in Step 3, and output the style transfer image from face to Terracotta Warrior.

[0008] Furthermore, the process of step 1 is as follows: Step 1.1: Obtain the high-definition open-source face dataset SeePrettyFace, and use a pre-trained multi-task convolutional neural network to perform attribute analysis on the original face images, and select face images that meet the gender and age characteristics as source domain images. Step 1.2: Obtain the original image of the real Terracotta Warriors. Use the Real-ESRGAN model to perform depth denoising and feature sharpening on the original image of the real Terracotta Warriors. Then, use the contrast-limited adaptive histogram equalization algorithm to perform brightness equalization on the image to obtain the target domain image. Step 1.3: Using a salient object detection network based on the U-Net architecture, automatically strip the foreground and background of the source and target domain images respectively. For difficult samples with incomplete segmentation, manual pixel-level fine-tuning is used to retain edge details, including hair and armor, and the background is uniformly replaced with a pure white standard background to obtain the source and target domain images with purified background. Step 1.4: Using the target region as the geometric center, expand the purified source region image and target region image outward by 20% and crop them to a fixed size. Then, use bilinear interpolation algorithm to uniformly scale the image resolution to 256×256 pixels. After enhancing the contrast in Lab color space, normalize the RGB pixel values ​​to the range of [-1, 1] to obtain the preprocessed source region image and target region image respectively. Then, the preprocessed target domain image is augmented offline to expand the target domain image. The expanded target domain image and the preprocessed source domain image are used together as samples in the training dataset to construct a cross-domain training dataset.

[0009] Furthermore, the offline amplification in step 1.4 includes random geometric flipping, HSV color space perturbation, and random rectangular occlusion.

[0010] Furthermore, step 3 is as follows: Step 3.1: The hierarchical text prompting module generates conditional texts with different emphases based on the training progress and injects them into the cross-attention layer of the U-Net network, providing explicit external semantic constraints for the attention reweighting module. The specific process is as follows: Step 3.1.1, Semantic Hierarchical Deconstruction: The dynamic prompt scheduler of the hierarchical text prompt module deconstructs the target domain image style into a set of semantic prompts with three independent dimensions according to the training progress; Step 3.1.2, Dynamic prompting scheduling strategy: Define the current global training step percentage as... The scheduler of the hierarchical text prompt module is based on The threshold-based, phased, dynamic combination of semantic prompts serves as the input prompt for itself. This generates conditional text with different emphases; Step 3.1.3: The conditional text is input as a cue word into the CLIP text encoder and is transformed into a conditional embedding feature vector. and as key-value pairs Injected into the cross-attention layer of the U-Net network; Step 3.2: Map the source domain image to the latent space using a pre-trained VAE encoder to obtain latent variables. Then The data is fed into the multi-scale module of the core U-Net denoising network for multi-scale feature extraction to obtain the original latent features. primal latent features The text conditional embedding key matrix interacts with the cross-attention layer, and the original cross-attention map is calculated through matrix multiplication. The attention reweighting module calculates the original cross-attention map. Spatial information entropy is used to dynamically generate an enhanced attention map that redistributes spatial weights. ; The enhanced attention map Returning to the cross-attention layer, the weighted multiplication operation with the text conditional value matrix is ​​performed, and the attention-enhanced features are output. Attention enhancement features The residual connections are returned to the U-Net denoising network and compared with the original latent features in the U-Net backbone data stream. Perform element-by-element addition and fusion; Repeat step 3.2. After a full time-step denoising process, the U-Net backbone outputs the final denoised latent features. The VAE decoder side displays the latent features after final denoising. Upsampling was performed to restore it to a pixel-level Terracotta Warriors-style image; Step 3.3: Calculate the training loss of the image cross-domain transfer network using the multi-objective loss calculation module. Update the network parameters through the error backpropagation mechanism to force the network to converge to a concentrated weight distribution to suppress global structural collapse, thus completing the model training. The specific process is as follows: Step 3.3.1: Calculate the basic adversarial loss of the CycleGAN-Turbo diffusion model. ; Step 3.3.2: Calculate the attention entropy regularization loss. First, use the Shannon entropy formula to calculate the spatial average entropy of a single-layer attention map. , is represented as: In the formula: Indicates the first Layer in time step Attention entropy; and These represent the height and width of a single-layer attention map, respectively. Represents the original attention map Each spatial location Cross different attention head dimensions Semantic uncertainty index; Then, by calculating the expected value of the average entropy across all time steps and the attention layer, the attention entropy regularization loss is obtained. , is represented as: In the formula: Indicates time step samples Calculate the expected value; This represents the total number of all cross-attention layers constrained by entropy regularization. Step 3.3.3: Construct a multi-objective joint optimization function , is represented as: In the formula: To balance the weighting coefficient, set it to 0.05; By minimizing This forces the network to converge to a concentrated weight distribution, thereby implicitly suppressing fuzzy and diffuse attention noise.

[0011] Furthermore, the semantic cue set of the three independent dimensions in step 3.1.1 includes structural layer cue. Material layer prompts and style layer tips ,in: The structural layer suggests Used to lay the geometric skeleton and topological foundation of the generated object, providing priority protection and generation area guidance for the attention reweighting module; The material layer prompts Used to define the surface physical visual properties of the generated object, guiding the specific direction of texture generation; The style layer prompt Used to inject historical and cultural context, thereby constraining higher-level style attributes.

[0012] Furthermore, the specific process of step 3.1.2 is as follows: 1) When At this time: The scheduler's input is only a structural layer hint. The generator is activated with the highest signal-to-noise ratio to conduct geometric priors on key features, including hair buns and facial features, to guide the construction of the underlying geometric contours. 2) When At time: the scheduler's input is and The model guides the model to accurately map the rough clay texture onto the established three-dimensional structure; 3) When At time: The scheduler's input includes 、 and The system includes full-style prompts, fine-tunes detailed features, and injects deep cultural semantics to achieve deep semantic rendering. In stages 1) to 3), the scheduler outputs conditional texts with different emphases.

[0013] Furthermore, in step 3.2, the attention reweighting module calculates the original cross-attention graph. Spatial information entropy is used to dynamically generate an enhanced attention map that redistributes spatial weights. The specific process is as follows: Step 3.2.1: Calculate the attention entropy map: Extract the original attention map output from the cross-attention layer. Calculate using Shannon's entropy formula Each spatial location Cross different attention head dimensions Semantic uncertainty index , is represented as: In the formula: For time step Downward attention map In spatial location And attention is on The weights; , Let B be the set of real numbers, and B be the batch size. and They are height and width, respectively. For the total number of attention heads; , is a smoothing term used to prevent numerical overflow; The higher the value, the more dispersed the attention distribution in spatial locations; Step 3.2.2: Generate an adaptive reweighted mask: Based on the difference between the spatial information entropy and the maximum theoretical entropy, generate a soft mask for adjusting the feature distribution using an inverse scaling function. , is represented as: In the formula: For the Sigmoid function, the value is mapped to interval; ,express The maximum theoretical entropy under a single attention head; This is a learnable scaling factor, initially set to 1.0; Step 3.2.3, Apply reweighting and renormalization: Apply the generated soft mask Acting on the original attention map And then renormalize along the spatial dimension, as follows: In the formula: This is a normalization operation for spatial dimensions; This indicates element-wise multiplication; To control the fixed hyperparameter of the reweighting intensity, it is set to... ; For time step Downward attention map In spatial location And attention is on The weights; Time step after normalization operation Downward attention map In spatial location And attention is on The weights, i.e., the weights of the redistribution space, are the weights of the enhanced attention map. .

[0014] Furthermore, the basic adversarial loss in step 3.3.1 Caused by the loss of resistance Cyclic consistency loss and identity loss Weighted average, represented as: In the formula: , and These are respectively adversarial loss, cycle consistency loss, and identity loss; , and Both represent hyperparameters, and , , ; The resistance loss Minimizing the distribution difference between the Terracotta Warriors style image and the target domain image using the PatchGAN discriminator is expressed as: In the formula: For discriminators; and Both represent fields. Indicates from domain To Domain Mapping; and All are random variables, representing the domain Heyu Samples sampled from the middle; Indicates to According to distribution Calculate the expected value; Indicates to According to distribution Calculate the expected value; The cycle consistency loss Represented as: In the formula: Indicates from domain To Domain Mapping; It is an L1 norm; The loss of identity Represented as: .

[0015] The present invention has the following beneficial technical effects: 1) Solving structural collapse and texture mismatch: By integrating a lightweight attention reweighting module into the denoising process of the diffusion model, attention weights are adaptively allocated, which successfully suppresses the global structural collapse in the style transfer process, realizes the generation of complex structures, including hair buns, from scratch, and corrects the facial features, thereby improving the quality of the generated image.

[0016] 2) No external geometric prior is required, and the engineering feasibility is high: By exploring the intrinsic potential of the cross-attention mechanism and information entropy regularization to guide structure generation, the huge memory overhead (e.g., OOM risk) caused by stacking external 3D models or keypoint detectors is avoided, and efficient and stable training can be achieved with limited hardware resources.

[0017] 3) Hierarchical semantic decoupling control: The innovative hierarchical text prompting system deconstructs abstract style into three dimensions and dynamically schedules them according to the training progress, providing the diffusion model with extremely clear spatial and semantic guidance, effectively avoiding the destruction of the underlying geometric structure by complex style noise.

[0018] 4) Significantly improved image quality: Quantitative experimental results show that the FID index of the images generated by the method of the present invention reaches an excellent 106.2194; the CLIP Score reaches 26.8732, which is significantly better than the CLIP Score of the existing CycleGAN, CUT and SDEdit methods; it can be seen that the method of the present invention achieves the best balance between feature fidelity and style transfer. Attached Figure Description

[0019] Figure 1 The overall architecture of the image style transfer method based on diffusion model structure regularization of the present invention; Figure 2 : A schematic diagram showing the internal details of the attention reweighting module (ARM) of this invention; Figure 3 A visual comparison of the cross-attention heatmaps of the network model and the baseline model of this invention; Figure 4: Evolution of attention entropy values ​​in the denoising time step of the network model and the baseline model of the present invention; Figure 5 The qualitative comparison experiment results of the network model of this invention and three baseline models on style transfer from human face to Terracotta Warriors; Figure 6 Qualitative analysis results of ablation experiments on the network model of this invention. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to specific embodiments. These descriptions are for explanation purposes only and are not intended to limit the scope of the invention.

[0021] refer to Figure 1 As shown, this invention provides an image style transfer method based on diffusion model structure regularization, comprising the following steps: Step 1: Obtain modern human face images from the source domain and terracotta warrior images from the target domain, and perform preprocessing to construct an unpaired cross-domain training dataset. The process is as follows: Step 1.1: Obtain the high-definition open-source face dataset SeePrettyFace, and use a pre-trained multi-task convolutional neural network to perform attribute analysis on the original face images. Automatically identify and remove female and young children samples that do not meet the geometric feature requirements, and select 3200 modern Asian male face images as source domain images. Step 1.2: Obtain 400 original images of real terracotta warriors. Use the Real-ESRGAN model to perform depth denoising and feature sharpening on the original images of real terracotta warriors. While maintaining the original spatial resolution of the image, restore and highlight the weathering particle texture and key edge details of the terracotta surface. Then, use the contrast-limited adaptive histogram equalization (CLAHE) algorithm to perform brightness equalization on the image, restore resolution and enhance illumination, suppress highlight clipping and brighten shadow details to obtain the target domain image. Step 1.3: Using a salient object detection network based on the U-Net architecture, automatically strip the foreground and background of the source and target domain images respectively. For difficult samples with incomplete segmentation, manual pixel-level fine-tuning is used to retain edge details, including hair and armor. The background is uniformly replaced with a pure white standard background to completely eliminate irrelevant environmental semantic noise and obtain the source and target domain images with purified background. Step 1.4: Using the target region as the geometric center, expand the purified source region image and target region image outward by 20% and crop them to a fixed size. Then, use bilinear interpolation algorithm to uniformly scale the image resolution to 256×256 pixels. After enhancing the contrast in Lab color space, normalize the RGB pixel values ​​to the range of [-1, 1] to obtain the preprocessed source region image and target region image respectively. To prevent subsequent model mode collapse, the preprocessed target domain images were augmented offline, including random geometric flipping, HSV color space perturbation, and random rectangular occlusion, expanding the target domain image samples to 1480 images. These 1480 target domain images and the preprocessed source domain images were used together as samples in the training dataset to construct a cross-domain training dataset. Step 2, construct as follows Figure 3 The image cross-domain transfer network shown is based on the pre-trained CycleGAN-Turbo diffusion model, including a generator core with an integrated attention reweighting module (ARM), a hierarchical prompt module, and a multi-objective loss calculation module, wherein: 1) Data processing of the hierarchical text prompt module: In order to effectively utilize the potential target domain structure prior of the pre-trained model, this embodiment designs a dynamic hierarchical text prompt module. The scheduler of the hierarchical text prompt module is based on the current training progress. The system generates conditional texts with different emphases, which are then used as prompts by the CLIP text encoder to convert them into conditional embedding feature vectors. and as key-value pairs The Cross-Attention Layer injected into the U-Net network provides explicit external semantic constraints for the Attention Reweighting Module (ARM). 2) The core data processing procedure of the generator: First, the source domain image is mapped to the latent space through a pre-trained variational autoencoder (VAE Encoder) to obtain latent variables. Then The data is fed into the multi-scale modules (Down / Mid / Up Blocks) of the core U-Net denoising network for multi-scale feature extraction to obtain the original latent features. In the forward denoising process of U-Net, the original latent features The query vector interacts with the externally injected text conditional embedding key matrix in the cross-attention layer, and the original cross-attention map is calculated through matrix multiplication. (Attention Map), and then the attention reweighting module (ARM) embedded in the cross-attention layer captures the original cross-attention map. By calculating its spatial information entropy, a reweighted soft mask is dynamically generated and applied to the original cross-attention map. The enhanced attention map outputs the spatial weights of the redistribution. Subsequently, attention enhancement maps Continue the weighted multiplication operation with the text value matrix within the cross-attention layer to output the attention-enhanced features. Attention enhancement features The residual connections are returned to the U-Net denoising network and compared with the original latent features in the U-Net backbone data stream. Element-wise addition is performed to implicitly enhance key geometric features, including the hairline and facial features, in the latent space. This interactive process is repeated, and after full time-step denoising, the U-Net backbone outputs the final denoised latent features. The variational autoencoder decoder (VAE decoder) processes the final denoised latent features. Upsampling was performed to restore it to a pixel-level Terracotta Warriors-style image; 3) Data processing of the Multi-objective Loss Calculation module: During the training phase, the generated Terracotta Warriors-style images and the target domain images in the training dataset are fed into the PatchGAN discriminator for adversarial training, using adversarial loss... Cyclic consistency loss Loss of identity and attention entropy regularization loss A multi-objective joint optimization function is constructed, and then the parameter update of the U-Net backbone and attention reweighting module (ARM) is guided by the error backpropagation mechanism, so that the model can achieve accurate geometric deformation and structural reshaping while generating high-fidelity clay texture images; Step 3: Train the image cross-domain transfer network based on the pre-trained CycleGAN-Turbo diffusion model. The process is as follows: Step 3.1: The hierarchical text prompting module generates conditional texts with different emphases based on the training progress and injects them into the cross-attention layer of the U-Net network, providing explicit external semantic constraints for the attention reweighting module (ARM). The specific process is as follows: Step 3.1.1, Semantic Hierarchical Deconstruction: The dynamic prompt scheduler of the hierarchical text prompt module deconstructs the target domain image style into a set of semantic prompts with three independent dimensions according to the training progress, including structural layer prompts. Material layer prompts and style layer tips ,in: 1) Structural layer hints It is a contour description that includes the hair bun and facial topology, used to lay the geometric skeleton and topological foundation of the generated object, providing priority protection and generation region guidance for the attention reweighting module. Its core description is: "a sculpture of a human face, with a hair bun on the head, clear and symmetrical facial features, rigid contour, frontal view". 2) Material layer prompts It is a description of physical properties, including terracotta and weathered unglazed clay, used to define the surface physical and visual properties of the generated object, guide the specific direction of texture generation, and prevent artifacts such as human flesh or plastic. Its core description is "texture of terracotta, unglazed clay, weathered surface, mattefinish, subtle cracks, rough details". 3) Style layer hints Used to inject historical and cultural context, thereby constraining high-level stylistic attributes, preventing stylistic drift to other fields of sculptural art, and improving the accuracy of high-level stylistic attributes, its core description is: "ancient Chinese terracotta warrior from the Qin dynasty, historical artifact, archaeological discovery, museum lighting". Step 3.1.2, Dynamic prompting scheduling strategy: Define the current global training step percentage as... The scheduler of the hierarchical text prompt module is based on Threshold-based phased dynamic combination input prompts This generates conditional texts with different focuses, and the specific process is divided into three stages: 1) Phase 1 is the coarse structural calibration period ( The scheduler's input is only a structural layer hint. The generator is activated with the highest signal-to-noise ratio to conduct geometric priors on key features, including hair buns and facial features, to guide the construction of the underlying geometric contours, thereby constructing an accurate spatial skeleton. 2) Phase Two ( The input to the scheduler is and Based on a stable geometric topology, the model is guided to accurately map the rough clay texture onto the established three-dimensional structure. 3) Phase Three ( The scheduler's input includes 、 and The entire style is guided by the principle that after the structure and materials are formed, the details are finely adjusted and deep cultural semantics are injected to achieve the desired effect. The scheduler outputs conditional texts with different emphases in three stages; Step 3.1.3: The conditional text is input as a cue word into the CLIP text encoder and is transformed into a conditional embedding feature vector. and as key-value pairs Injected into the cross-attention layer of the U-Net network; Step 3.2: Map the source domain image to the latent space using a pre-trained variational autoencoder (VAE Encoder) to obtain the latent variables. Then The data is fed into the multi-scale module of the core U-Net denoising network for multi-scale feature extraction to obtain the original latent features. primal latent features The conditional embedding key matrix injected from the outside interacts with the cross-attention layer, and the original cross-attention map is calculated through matrix multiplication. ; like Figure 2 As shown, the attention reweighting module (ARM) computes the original cross-attention map. Spatial information entropy, for Online intervention is performed on the attention distribution, and an enhanced attention map with dynamically redistributed spatial weights is generated based on spatial information entropy. The specific process is as follows: Step 3.2.1: Calculate the attention entropy map: Extract the original attention map output from the cross-attention layer. Calculate using Shannon's entropy formula Each spatial location Cross different attention head dimensions Semantic uncertainty index , is represented as: In the formula: For time step Downward attention map In spatial location And attention is on The weights; , Let B be the set of real numbers, and B be the batch size. and They are height and width, respectively. For the total number of attention heads; , is a smoothing term used to prevent numerical overflow; The higher the value, the more dispersed the attention distribution in spatial locations; Step 3.2.2: Generate an adaptive reweighted mask: Based on the difference between the spatial information entropy and the maximum theoretical entropy, generate a soft mask for adjusting the feature distribution using an inverse scaling function. , is represented as: In the formula: For the Sigmoid function, the value is mapped to interval; ,express The maximum theoretical entropy under a single attention head; This is a learnable scaling factor, initially set to 1.0; Step 3.2.3, Apply reweighting and renormalization: Apply the generated soft mask Acting on the original attention map And then renormalize along the spatial dimension, as follows: In the formula: This is a normalization operation for spatial dimensions; This indicates element-wise multiplication; To control the fixed hyperparameter of the reweighting intensity, it is set to... ; For time step Downward attention map In spatial location And attention is on The weights; Time step after normalization operation Downward attention map In spatial location And attention is on The weights, i.e., the weights of the redistribution space, are the weights of the enhanced attention map. ; By renormalizing along the spatial dimension, we can adaptively enhance the feature saliency of low-entropy regions (i.e., foreground key semantics) while suppressing noise interference in high-entropy regions (background), resulting in an enhanced attention map with redistributed spatial weights. Enhance attention map Returning to the cross-attention layer, the weighted multiplication operation with the text conditional value matrix is ​​performed, and the attention-enhanced features are output. Attention enhancement features The residual connections are returned to the U-Net denoising network and compared with the original latent features in the U-Net backbone data stream. Perform element-by-element addition and fusion; Repeat step 3.2. After a full time-step denoising process, the U-Net backbone outputs the final denoised latent features. The variational autoencoder decoder (VAE decoder) processes the final denoised latent features. Upsampling was performed to restore it to a pixel-level Terracotta Warriors-style image; Step 3.3: Calculate the training loss of the image cross-domain transfer network using the multi-objective loss calculation module. Update the network parameters through the error backpropagation mechanism to force the network to converge to a concentrated weight distribution to suppress global structural collapse, thus completing the model training. The specific process is as follows: Step 3.3.1: Calculate the basic adversarial loss of the CycleGAN-Turbo diffusion model. Its contribution to combating losses Cyclic consistency loss and identity loss Weighted average, represented as: In the formula: , and These are respectively adversarial loss, cycle consistency loss, and identity loss; , and Both represent hyperparameters, and , , ; The resistance loss By minimizing the distribution difference between the Terracotta Warriors-style image and the target domain image using the PatchGAN discriminator, the network model learns the texture and style features of the target domain image. To Domain mapping and its discriminator To resist losses Defined as: In the formula: For discriminators; and Both represent fields. Indicates from domain To Domain Mapping; and All are random variables, representing the domain Heyu Samples sampled from the middle; Indicates to According to distribution Calculate the expected value; Indicates to According to distribution Calculate the expected value; The cycle consistency loss It is a key constraint function in unsupervised translation that preserves the semantic meaning of the content, ensuring that the image, after undergoing translation... The closed-loop transformation can approximately restore the original image, thus preserving the semantic content of the source image during style transfer. This can be represented as: In the formula: and Both represent fields. Indicates from domain To Domain The mapping, Indicates from domain To Domain Mapping; and All are random variables, representing the domain Heyu Samples sampled from the middle; It is an L1 norm; Indicates to According to distribution Calculate the expected value; Indicates to According to distribution Calculate the expected value; The loss of identity By introducing identity mapping constraints, the generator is forced to maintain style output invariance when inputting target domain images. In the early stages of training, this loss term effectively stabilizes the generator's weight distribution, maintains the robustness of hue and identity features during style transfer, and prevents facial structure collapse caused by the model's excessive pursuit of stylization. This can be expressed by the formula: In the formula: and Both represent fields. Indicates from domain To Domain The mapping, Indicates from domain To Domain Mapping; and All are random variables, representing the domain Heyu Samples sampled from the middle; It is an L1 norm; Indicates to According to distribution Calculate the expected value; Indicates to According to distribution Calculate the expected value; Step 3.3.2: Calculate the attention entropy regularization loss. First, use the Shannon entropy formula to calculate the spatial average entropy of a single-layer attention map. , is represented as: In the formula: Indicates the first Layer in time step Attention entropy; and These represent the height and width of a single-layer attention map, respectively. Represents the original attention map Each spatial location Cross different attention head dimensions Semantic uncertainty index; Then, by calculating the expected value of the average entropy across all time steps and the attention layer, the attention entropy regularization loss is obtained. , is represented as: In the formula: Indicates time step samples Calculate the expected value; This represents the total number of all cross-attention layers constrained by entropy regularization. Step 3.3.3: Construct a multi-objective joint optimization function , is represented as: In the formula: To balance the weights, set them to 0.05. Guide the distribution of attention; The U-Net backbone and ARM modules are guided by an error backpropagation mechanism. During the backpropagation process, the parameters are minimized. This forces the network to converge to a concentrated weight distribution, thereby implicitly suppressing blurred and diffuse attention noise and completing structural straightening correction. Step 4: Input the face image to be converted into the image cross-domain transfer network trained in Step 3, and predict the output style transfer image from face to Terracotta Warrior. The specific process is as follows: Freeze all weight parameters of the image cross-domain transfer network trained in Step 3, take a modern face image to be converted, preprocess it using the preprocessing method in Step 1, and input it into the trained image cross-domain transfer network. Under the guidance of the attention reweighting module (ARM) and the hierarchical text prompt module, forward denoising inference is performed in the U-Net backbone network to output a style transfer image from face to Terracotta Warrior with both high-fidelity terracotta texture and original identity features.

[0022] To verify the effectiveness of the image style transfer method based on diffusion model structure regularization proposed in this embodiment, the following experimental evaluation was conducted: 1) Experimental environment configuration: The network model training and inference in this embodiment are carried out in the Linux operating system environment. The deep learning framework used is PyTorch, the graphics card is NVIDIA GTX 4080, and it runs in the PyTorch 1.13.1 and CUDA 11.7 environment. The shortest training time for the network model training of 7000 steps in this embodiment is 31 hours. 2) Figure 3 Demonstrates different sampling time steps t Below, a comparison of the cross-attention weight distribution of the baseline model (CycleGAN-Turbo) and the image cross-domain transfer network of this embodiment shows that after attention entropy regularization loss, the attention weights exhibit stronger selectivity in the early stage of generation. As the denoising process progresses, the heatmap shows extremely high focus, accurately focusing on key geometric structural regions such as the face. This transformation from uncertainty to certainty is the intrinsic reason for the accurate generation of the hair bun. In contrast, the attention distribution of the baseline model is relatively diffuse and cannot determine the specific location of the hair bun generation. This indicates that the model of this embodiment can effectively induce the model to extract structured representations and reduce the interference of redundant information. 3) Figure 4 shows the evolution of attention entropy values ​​in the proposed image cross-domain transfer network model and the baseline model during the denoising time step. From the spatial decoupling capability shown in Figure 4(a), the model exhibits extremely precise regional differentiation control. Throughout the entire time step evolution, the mean entropy of the foreground region is consistently suppressed to a low level and remains stable, while the mean entropy of the background region is significantly higher. This further perfectly confirms the theoretical assumptions of the proposed method: for the foreground region of a face containing core identity features, the model spontaneously generates a low-entropy attention distribution, ensuring high fidelity of the facial features and the entire face region; while for the background and edge regions, the model adopts a high-entropy attention distribution. This dispersed attention can more broadly absorb the texture and color of the source domain, thus fully integrating the unique clay sculpture texture and historical weight of the Terracotta Warriors; from the activity of the global attention distribution shown in Figure 4(b) (Entropy vs... After introducing the entropy regularization mechanism (Timestep), the global mean entropy is generally higher than that of the baseline model, and both show a rapid increase in the early stage of generation followed by convergence. Although this embodiment applies low entropy induction to the foreground region, the slight increase in global mean entropy indicates that the mechanism does not cause the model to fall into attention collapse or over-smoothing. On the contrary, because the background region is given higher degrees of freedom to fit complex deformations and textures, the overall model maintains a stronger ability in style transfer. The baseline model, lacking such regional dynamic intervention, has a relatively rigid attention distribution, which makes the generated results prone to feature loss or artifact generation.

[0023] 4) Figure 5The paper presents comparative experimental results of the method (Ours) in this embodiment with three mainstream unpaired image translation methods: CycleGAN, CUT, and SDEdit. Visually, it is clear that traditional generative adversarial network-based methods (CycleGAN and CUT) perform poorly in handling tasks with significant geometric deformation spans. Although they simulate the rough texture of the Terracotta Warriors to some extent, facial features are almost completely lost, and unnatural artifacts and deformations are easily generated in areas with typical Terracotta Warrior features, such as the hair bun. The SDEdit method, based on a diffusion model, is trapped in preserving the content of the input image but generates almost no texture similar to the Terracotta Warriors, especially when a low denoising strength of 0 is used. At time 4, the model excessively preserved the color and lighting texture of the input image, failing to bridge the inter-domain gap and achieve the transformation to the style features of the Terracotta Warriors. When the denoising strength was increased to 0.75, although the image exhibited a texture similar to modern painted sculptures, it still deviated from the unique historical weight of the real Terracotta Warriors, and was also accompanied by the collapse of identity characteristics. In contrast, the method in this embodiment achieved a more perfect balance between feature fidelity and style transfer. It not only accurately generated the unique terracotta texture and three-dimensionality of the Terracotta Warriors, but also preserved the facial proportions and identity recognition of the input figure with high precision while completing deep style transfer. This excellent generation quality fully demonstrates the superior performance of the method in this embodiment in feature deconstruction and cross-attention control.

[0024] Table 1 summarizes the quantitative evaluation results of the method (Ours) in this embodiment and three mainstream unpaired image translation methods, CycleGAN, CUT, and SDEdit, based on three core metrics: FID, ID Score, and CLIP Score. It can be seen that: in terms of the FID metric, which measures the global image generation quality and target domain fit, the method in this embodiment achieves the best score of 106.2194, significantly outperforming traditional generative adversarial networks and diffusion models. This objectively proves that the method in this embodiment can generate a muddy material style that most closely resembles the distribution of real terracotta warriors. Regarding the ID Score, which evaluates facial feature preservation, SDEdit (strength=0.4) achieved a score of 0.3769 due to its low denoising intensity, essentially equivalent to adding only a light layer of color. While the initial score was inflated, its extremely poor FID (Film ID) of 246.9657 indicates that it did not achieve effective style transfer. In models that truly achieved deep texture transfer, the ID score of the method in this embodiment was comparable to SDEdit (0.75) and significantly better than CycleGAN and CUT, which caused severe distortion of facial features. On the CLIP Score, the method in this embodiment achieved 25.7504 points, successfully avoiding the topological collapse problem caused by traditional GAN ​​networks overfitting local rough textures to unilaterally improve style matching. In summary, the quantitative data shows that the method in this embodiment did not fall into extreme overfitting of a single metric, but successfully achieved an optimal comprehensive balance between the contradictory task of terracotta warrior texture and real human identity features.

[0025] Table 1. Quantitative evaluation results of each model on the test set. 5) Figure 6The ablation experiments demonstrate the impact of different components on the generated results. The baseline model exhibits significant limitations in handling cross-domain geometric deformations. Constrained by the rigid contours of the source image, the model not only fails to generate a hair bun structure in the top region but also suffers from severe distortion and disproportion of facial features. When the ARM module is removed, retaining only the hierarchical text prompt module significantly improves the distortion of facial features, resulting in a more natural physical texture for the sculpture. This indicates that multi-dimensional semantic constraints can effectively anchor target domain features and improve the stability of facial generation. However, due to the lack of an internal attention redistribution mechanism, the model remains constrained by the original spatial constraints and is completely unable to generate the novel hair bun structure, proving that purely external semantic guidance lacks the underlying motivation to change the geometric topology. When the hierarchical text prompt module is removed and only the ARM module is relied upon for driving, the model shows significant improvement. The model did indeed demonstrate the ability to break through the geometric constraints of the source domain, with obvious structural deformation appearing in the top of the head. Due to the loss of clear semantic navigation, this deformation exhibited extreme blindness and disorder, failing to form a standard bun shape. Its disordered attention expansion led to severe structural distortions in the facial contours and features. In contrast, the complete network model in this embodiment achieved deep coupling between "semantic definition" and "generation mechanism." Under the precise orientation of hierarchical prompts, the ARM module guided attention to the target area in an orderly manner, successfully generating a well-structured and spatially voluminous side bun. Furthermore, while maintaining the high stability of the facial features, it endowed the face with the authentic texture of a terracotta warrior sculpture. This demonstrates that in cross-domain geometric reshaping tasks, the "semantic orientation" provided by the hierarchical text prompt module and the "structural dynamics" provided by the ARM module are indispensable.

[0026] Table 2 shows the results of four key indicators of each module's contribution to generation quality and structural control. Regarding global generation quality, the complete network model (Ours) in this embodiment achieved the best performance with a core indicator FID of 106.67, a significant decrease compared to the baseline model (116.67). This indicates that the network model's generation results better match the real data distribution of the Terracotta Warriors. Conversely, the FID of the w / o ARM variant surged to 141.94, confirming that the lack of an attention reweighting module (ARM) in the underlying attention reassignment produces unnatural planar artifacts, resulting in indistinct details and blurred textures in semantic regions such as facial features. In terms of perceptual distance (LPIPS) and identity preservation (ID Score), the baseline model and the w / o ARM variant achieved the best values ​​in LPIPS and ID Score, respectively. This is because neither effectively altered the physical contours of the source image. This mechanical preservation of the original image structure leads to high scores in LPIPS and ID Score, thus masking the model's insufficient ability to handle complex deformations. Text data shows that if only the ARM module is introduced and the semantic guidance of the hierarchical text prompt module is removed, disordered attention expansion will lead to severe facial distortion, causing the ID Score to drop to 0.0893. In contrast, the complete network model in this embodiment, while successfully generating the hair bun and achieving large-scale geometric reconstruction, still maintains a high IDScore of 0.2223. In the cross-modal semantic fit evaluation, the complete network model in this embodiment achieved the highest CLIP Score (26.8732), significantly outperforming the baseline model and various variants, proving that the hierarchical text prompt mechanism has successfully injected the Terracotta Warrior style into the feature space. It can be seen that the quantitative indicators in Table 2 are consistent with... Figure 6 The high degree of fit between qualitative visual analysis and the system fully demonstrates the superior performance of the "semantic orientation" and "mechanism dynamics" coupled system in cross-domain style transfer tasks.

[0027] Table 2 Quantitative results of ablation experiments

Claims

1. An image style transfer method based on diffusion model structure regularization, characterized in that, Includes the following steps: Step 1: Obtain modern human face images from the source domain and terracotta warrior images from the target domain, and perform preprocessing to construct an unpaired cross-domain training dataset that includes images from the source and target domains. Step 2: Construct an image cross-domain transfer network based on the pre-trained CycleGAN-Turbo diffusion model, including a generator core integrating an attention reweighting module, a hierarchical text prompting module, and a multi-objective loss calculation module, wherein: The scheduler of the hierarchical text prompting module generates conditional texts with different emphases based on the current training progress. These conditional texts, as prompt words, are converted into conditional embedding feature vectors by the CLIP text encoder and used as key-value pairs. Injected into the cross-attention layer of the U-Net network; The generator core includes a VAE encoder, a U-Net denoising network, a cross-attention layer, an attention reweighting module, and a VAE decoder. Specifically: the VAE encoder maps the source domain image to the latent space to obtain latent variables; the U-Net denoising network extracts multiple features from the latent variables to obtain the original latent features; during the forward denoising process of U-Net, the original latent features, as query vectors, interact with the externally injected text conditional embedding key matrix at the cross-attention layer, calculating the original cross-attention map through matrix multiplication; the attention reweighting module captures the original cross-attention map and calculates its spatial information entropy, dynamically generating a reweighting mask to adaptively redistribute attention weights, outputting an enhanced attention map; the VAE decoder upsamples the denoised latent features output by the U-Net denoising network, restoring them to a pixel-level Terracotta Warriors-style image. The multi-target loss calculation module calculates the training loss of the image cross-domain transfer network, updates the network parameters through the error backpropagation mechanism, and forces the network to converge to a concentrated weight distribution. Step 3: Train an image cross-domain transfer network based on a pre-trained CycleGAN-Turbo diffusion model; Step 4: Input the face image to be converted into the image cross-domain transfer network trained in Step 3, and output the style transfer image from face to Terracotta Warrior.

2. The image style transfer method based on diffusion model structure regularization according to claim 1, characterized in that, The process of step 1 is as follows: Step 1.1: Obtain the high-definition open-source face dataset SeePrettyFace, and use a pre-trained multi-task convolutional neural network to perform attribute analysis on the original face images, and select face images that meet the gender and age characteristics as source domain images. Step 1.2: Obtain the original image of the real Terracotta Warriors. Use the Real-ESRGAN model to perform depth denoising and feature sharpening on the original image of the real Terracotta Warriors. Then, use the contrast-limited adaptive histogram equalization algorithm to perform brightness equalization on the image to obtain the target domain image. Step 1.3: Using a salient object detection network based on the U-Net architecture, automatically strip the foreground and background of the source and target domain images respectively. For difficult samples with incomplete segmentation, manual pixel-level fine-tuning is used to retain edge details, including hair and armor, and the background is uniformly replaced with a pure white standard background to obtain the source and target domain images with purified background. Step 1.4: Using the target region as the geometric center, expand the purified source region image and target region image outward by 20% and crop them to a fixed size. Then, use bilinear interpolation algorithm to uniformly scale the image resolution to 256×256 pixels. After enhancing the contrast in Lab color space, normalize the RGB pixel values ​​to the range of [-1, 1] to obtain the preprocessed source region image and target region image respectively. Then, the preprocessed target domain image is augmented offline to expand the target domain image. The expanded target domain image and the preprocessed source domain image are used together as samples in the training dataset to construct a cross-domain training dataset.

3. The image style transfer method based on diffusion model structure regularization according to claim 2, characterized in that, The offline augmentation in step 1.4 includes random geometric flipping, HSV color space perturbation, and random rectangular occlusion.

4. The image style transfer method based on diffusion model structure regularization according to claim 1, characterized in that, The process of step 3 is as follows: Step 3.1: The hierarchical text prompting module generates conditional texts with different emphases based on the training progress and injects them into the cross-attention layer of the U-Net network, providing explicit external semantic constraints for the attention reweighting module. The specific process is as follows: Step 3.1.1, Semantic Hierarchical Deconstruction: The dynamic prompt scheduler of the hierarchical text prompt module deconstructs the target domain image style into a set of semantic prompts with three independent dimensions according to the training progress; Step 3.1.2, Dynamic prompting scheduling strategy: Define the current global training step percentage as... The scheduler of the hierarchical text prompt module is based on The threshold-based, phased, dynamic combination of semantic prompts serves as the input prompt for itself. This generates conditional text with different emphases; Step 3.1.3: The conditional text is input as a cue word into the CLIP text encoder and is transformed into a conditional embedding feature vector. and as key-value pairs Injected into the cross-attention layer of the U-Net network; Step 3.2: Map the source domain image to the latent space using a pre-trained VAE encoder to obtain latent variables. Then The data is fed into the multi-scale module of the core U-Net denoising network for multi-scale feature extraction to obtain the original latent features. primal latent features The text conditional embedding key matrix interacts with the cross-attention layer, and the original cross-attention map is calculated through matrix multiplication. The attention reweighting module calculates the original cross-attention map. Spatial information entropy is used to dynamically generate an enhanced attention map that redistributes spatial weights. ; The enhanced attention map Returning to the cross-attention layer, the weighted multiplication operation with the text conditional value matrix is ​​performed, and the attention-enhanced features are output. Attention enhancement features The residual connections are returned to the U-Net denoising network and compared with the original latent features in the U-Net backbone data stream. Perform element-by-element addition and fusion; Repeat step 3.

2. After a full time-step denoising process, the U-Net backbone outputs the final denoised latent features. The VAE decoder side displays the latent features after final denoising. Upsampling was performed to restore it to a pixel-level Terracotta Warriors-style image; Step 3.3: Calculate the training loss of the image cross-domain transfer network using the multi-objective loss calculation module. Update the network parameters through the error backpropagation mechanism to force the network to converge to a concentrated weight distribution to suppress global structural collapse, thus completing the model training. The specific process is as follows: Step 3.3.1: Calculate the basic adversarial loss of the CycleGAN-Turbo diffusion model. ; Step 3.3.2: Calculate the attention entropy regularization loss. First, use the Shannon entropy formula to calculate the spatial average entropy of a single-layer attention map. , is represented as: In the formula: Indicates the first Layer in time step Attention entropy; and These represent the height and width of a single-layer attention map, respectively. Represents the original attention map Each spatial location Cross different attention head dimensions Semantic uncertainty index; Then, by calculating the expected value of the average entropy across all time steps and the attention layer, the attention entropy regularization loss is obtained. , is represented as: In the formula: Indicates time step samples Calculate the expected value; This represents the total number of all cross-attention layers constrained by entropy regularization. Step 3.3.3: Construct a multi-objective joint optimization function , is represented as: In the formula: To balance the weighting coefficient, set it to 0.05; By minimizing This forces the network to converge to a concentrated weight distribution, thereby implicitly suppressing fuzzy and diffuse attention noise.

5. The image style transfer method based on diffusion model structure regularization according to claim 4, characterized in that, The semantic cue set of the three independent dimensions in step 3.1.1 includes structural layer cue. Material layer prompts and style layer tips ,in: The structural layer suggests Used to lay the geometric skeleton and topological foundation of the generated object, providing priority protection and generation area guidance for the attention reweighting module; The material layer prompts Used to define the surface physical visual properties of the generated object, guiding the specific direction of texture generation; The style layer prompt Used to inject historical and cultural context, thereby constraining higher-level style attributes.

6. The image style transfer method based on diffusion model structure regularization according to claim 5, characterized in that, The specific process of step 3.1.2 is as follows: 1) When At this time: The scheduler's input is only a structural layer hint. The generator is activated with the highest signal-to-noise ratio to conduct geometric priors on key features, including hair buns and facial features, to guide the construction of the underlying geometric contours. 2) When At time: the scheduler's input is and The model guides the model to accurately map the rough clay texture onto the established three-dimensional structure; 3) When At time: The scheduler's input includes 、 and The system includes full-style prompts, fine-tunes detailed features, and injects deep cultural semantics to achieve deep semantic rendering. In stages 1) to 3), the scheduler outputs conditional texts with different emphases.

7. The image style transfer method based on diffusion model structure regularization according to claim 4, characterized in that, In step 3.2, the attention reweighting module calculates the original cross-attention graph. Spatial information entropy is used to dynamically generate an enhanced attention map that redistributes spatial weights. The specific process is as follows: Step 3.2.1: Calculate the attention entropy map: Extract the original attention map output from the cross-attention layer. Calculate using Shannon's entropy formula Each spatial location Cross different attention head dimensions Semantic uncertainty index , is represented as: In the formula: For time step Downward attention map In spatial location And attention is on The weights; , Let B be the set of real numbers, and B be the batch size. and They are height and width, respectively. For the total number of attention heads; , is a smoothing term used to prevent numerical overflow; The higher the value, the more dispersed the attention distribution in spatial locations; Step 3.2.2: Generate an adaptive reweighted mask: Based on the difference between the spatial information entropy and the maximum theoretical entropy, generate a soft mask for adjusting the feature distribution using an inverse scaling function. , is represented as: In the formula: For the Sigmoid function, the value is mapped to interval; ,express The maximum theoretical entropy under a single attention head; This is a learnable scaling factor, initially set to 1.0; Step 3.2.3, Apply reweighting and renormalization: Apply the generated soft mask Acting on the original attention map And then renormalize along the spatial dimension, as follows: In the formula: This is a normalization operation for spatial dimensions; This indicates element-wise multiplication; To control the fixed hyperparameter of the reweighting intensity, it is set to... ; For time step Downward attention map In spatial location And attention is on The weights; Time step after normalization operation Downward attention map In spatial location And attention is on The weights, i.e., the weights of the redistribution space, are the weights of the enhanced attention map. .

8. The image style transfer method based on diffusion model structure regularization according to claim 4, characterized in that, The basic adversarial loss in step 3.3.1 Caused by the loss of resistance Cyclic consistency loss and identity loss Weighted average, represented as: In the formula: , and These are respectively adversarial loss, cycle consistency loss, and identity loss; , and Both represent hyperparameters, and , , ; The resistance loss Minimizing the distribution difference between the Terracotta Warriors style image and the target domain image using the PatchGAN discriminator is expressed as: In the formula: For discriminators; and Both represent fields. Indicates from domain To Domain Mapping; and All are random variables, representing the domain Heyu Samples sampled from the middle; Indicates to According to distribution Calculate the expected value; Indicates to According to distribution Calculate the expected value; The cycle consistency loss Represented as: In the formula: Indicates from domain To Domain Mapping; It is an L1 norm; The loss of identity Represented as: 。

Images