An image style transfer method based on content and style parser

Abstract

A method for image style transfer based on a content and style parser includes: using a stable diffusion model to learn and encode image content features. A style parser is constructed, a visual deformer (ViT) is used to extract style features, and a style embedding is generated through a multi-layer perceptron. The content parser processes the content image in layers to generate a content embedding, which is integrated into the model using residual addition, so that the content features are preserved during style transfer. During the inference phase, the style embedding is injected into the upsampling block to enhance the style expression, while the content embedding is injected into the downsampling block to ensure the integrity of the content. This method effectively solves the problems of resource-intensive training and the entanglement of style and content in traditional diffusion models, reduces one-way bias, and comprehensively improves image quality and style similarity. In terms of visual experience, the generated images achieve a delicate balance between quality, content, and style similarity, significantly optimizing the user's visual experience.

Description

Technical Field

[0001] The present invention relates to artificial intelligence computer vision technology, and in particular to an image style conversion method based on content and style parser. Background Art

[0002] In the field of image style transfer, the current dominant approach is to utilize diffusion models. Current image style transfer often relies on diffusion models, which, while highly generative, require significant resources to train. Consequently, the industry has increasingly focused on fine-tuning the cross-attention mechanism and its weights to reduce costs. However, this process interweaves style and content during feature extraction and sampling, often leading to unilateral bias in results. Diffusion models are resource-intensive to train, and when fine-tuning the cross-attention mechanism and its weights, style and content become entangled during feature extraction and sampling, resulting in unilateral bias in results. For example, diffusion model methods are prone to mixing irrelevant content into stylized images, deviating from the target style. Traditional methods can struggle to capture complex styles or compromise content integrity, and poor style-content coupling can lead to suboptimal stylization results.

[0003] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0004] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide an image style conversion method based on content and style parser.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for image style conversion based on content and style parser, comprising the following steps:

[0007] S1. Building a basic model: Using the stable diffusion model as the basic architecture, the image content features are learned and encoded through the denoising and de-noising processes.

[0008] S2. Build a style parser and extract style features: Use the visual deformer ViT as a style parser to extract style features from the reference image and generate style embeddings through a multi-layer perceptron.

[0009] S3. Build a content parser and extract content features: Build a content parser that processes content images hierarchically to generate multiple levels of latent representations as content embeddings. This content embeddings are then integrated into the model via residual addition to stabilize the content architecture and preserve content features during style transfer.

[0010] S4. Build a content controller and inject content: Process the content image through the content controller to capture the basic structure and minimize the interference of style features;

[0011] S5. Injecting style and content: Injecting style embeddings into the upsampling block of the model to enhance style expression, while injecting content embeddings into the downsampling block of the model to ensure content preservation;

[0012] S6. Reasoning: In the reasoning stage, the constructed content parser and style parser are used to decouple style from content, capturing and reproducing complex styles while maintaining the integrity of the content.

[0013] Furthermore, the processing of the style analyzer specifically includes:

[0014] The image is divided into blocks and linearly embedded using the visual deformer ViT;

[0015] Calculate query, key, and value matrices through the self-attention mechanism, update the input feature representation, and generate style features;

[0016] By normalizing the output of the self-attention mechanism, the training process is stabilized and the model's generalization ability is improved;

[0017] Style embedding is further extracted from the normalized features through a multi-layer perceptron (MLP).

[0018] Furthermore, the training process of the style analyzer specifically includes:

[0019] During training, we use the Gram loss to ensure style alignment by comparing the Gram matrices of features of the generated and target images.

[0020] Keep the visual features of the generated image and the target image aligned through perceptual loss;

[0021] The original style loss, perceptual loss and Gram loss are weighted summed to obtain the total style loss;

[0022] Randomly drop image conditions during training, enabling classifier-free bootstrapping during inference.

[0023] Furthermore, the style parser includes the following style injection strategies:

[0024] Leveraging the hierarchical nature of convolutional neural networks, we identify and specifically use upsampling blocks to capture and inject style information.

[0025] A cross-attention mechanism is implemented to inject style information into the upsampling block guided by image conditions, thereby enhancing style expression.

[0026] Furthermore, the operations of the content controller specifically include:

[0027] A tiling control strategy is adopted to preserve content information without affecting the style transfer process;

[0028] Use ControlNet to manage spatial information within the U-Net architecture, capturing the basic structure of the content image and minimizing the interference of style features;

[0029] The content fusion encoder integrates the content to generate multiple levels of latent representations to form content embedding;

[0030] Integrate content embedding into U-Net via residual addition;

[0031] Through the ControlNet mechanism, content embedding is accurately injected into the upsampling block of the diffusion model to achieve the separation and conversion of content and style.

[0032] Furthermore, the training process of the content analyzer specifically includes:

[0033] Keep the parameters of the pre-trained diffusion model unchanged during the training phase;

[0034] Use image-text pairs for training to achieve effective correspondence between content and text descriptions;

[0035] Use mean squared error loss to measure and minimize the difference between the model's predicted noise and the actual noise;

[0036] Use adversarial loss to train the discriminator to enhance its ability to distinguish between real and generated images;

[0037] The content loss is formed by weighted combination of mean squared error loss and adversarial loss to balance the contribution of the two losses to content preservation quality.

[0038] Furthermore, the content embeddings derived from the content parser are injected into the downsampling block of the model.

[0039] Furthermore, in step S6, the reasoning includes a reversal operation for content preservation, specifically including:

[0040] During the sampling process, the input image is encoded into a latent space, and the ReNoise technique reverses the sampling process to generate a latent noise representation directly from the real image, thus eliminating the need for additional noise injection;

[0041] An iterative approach is used to update the noise, enhancing the approximation of the expected position in the forward diffusion progression by averaging the predictions.

[0042] Furthermore, step S6 specifically includes:

[0043] Leveraging the dissociative nature of the crisscross attention mechanism, we independently adjust the image conditional weights for content and style conditions during inference.

[0044] Construct a latent representation of the final image by weightedly combining the attention results of content and style conditions;

[0045] Set the text condition weight factor to zero;

[0046] When the weight factors of both content and style conditions are zero, the model reverts to the original text-to-image diffusion model.

[0047] A computer program product includes a computer program, wherein when the computer program is executed by a processor, the image style conversion method based on the content and style parser is implemented.

[0048] The present invention has the following beneficial effects:

[0049] This paper proposes an image style transfer method based on a content-based style parser. This method accurately decouples style and content, effectively avoiding the resource-intensive training requirements of existing diffusion models. It also addresses the one-way bias caused by the intertwining of style and content during feature extraction and sampling. Through a specially constructed style and content parser, the method accurately injects style and content information into different blocks of a U-Net, enabling flexible and diverse style transfer without compromising content integrity. For example, in a qualitative comparison, compared to methods such as CS:GO, the method accurately transfers style while fully preserving structures and details such as buildings and characters. Furthermore, by collaboratively training the style parser with multiple losses and finely optimizing the content parser with multiple losses, the method comprehensively improves image quality and style similarity. The method achieves excellent performance in quantitative evaluation using the CSD and CAS metrics, demonstrating its high efficiency in style and content control. User studies also demonstrate that the method leads in selecting style and content scores and overall quality. The generated images achieve a delicate balance between quality, content similarity, and style similarity, significantly enhancing the user visual experience.

[0050] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is an overall flow chart of the image style conversion method based on content and style parser according to an embodiment of the present invention.

[0052] Figure 2 This is an algorithm architecture diagram of the image style conversion method based on content and style parser according to an embodiment of the present invention.

[0053] Figure 3The stylized result images generated by the embodiment of the present invention under different style conditions. The first image is used as the content image, and the illustration represents the style image.

[0054] Figure 4 This is a qualitative comparison of the embodiment of the present invention and other advanced methods. Other methods produce unilateral expression bias: diffusion model-based methods ((b)-(h)) tend to express style, while traditional style transfer methods ((i)-(k)) tend to express content.

[0055] Figure 5 These are the results of extensive qualitative comparison experiments between the embodiments of the present invention and the diffusion model-based method (the ah column) and the traditional style transfer method (the ik column).

[0056] Figure 6 The percentage result of users' preference for the overall effect.

[0057] Figure 7 Ablation experiments in which the content parser of an embodiment of the present invention is embedded in different U-Net blocks.

[0058] Figure 8 Ablation experiments in which the style parser of an embodiment of the present invention is embedded in different U-Net blocks.

[0059] Figure 9 This is the influence of the style analyzer strength on the results according to the embodiment of the present invention.

[0060] Figure 10 Ablation experiments of (a) stylized representation for style detail control and (b) ReNoise inversion for content enhancement control according to embodiments of the present invention.

[0061] Figure 11 This is an ablation study of an embodiment of the present invention using Canny edge detection as a control condition to replace the content controller.

[0062] Figure 12 This is the influence of the content controller strength on the results according to the embodiment of the present invention.

[0063] Figure 13 This is the impact of the content parser strength on the results of the embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0065] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0066] See Figure 1 and Figure 2 , an embodiment of the present invention provides an image style conversion method based on content and style parser, comprising the following steps:

[0067] Step S1. Building a basic model: Using a stable diffusion model as the basic architecture, learning and encoding image content features through the process of adding and denoising;

[0068] In a preferred embodiment, the diffusion model uses U-Net to implement the denoising process and is trained through mean square error loss.

[0069] Step S2. Construct a style parser and extract style features: Use the visual deformer ViT as a style parser to extract style features from the reference image and generate style embedding through a multi-layer perceptron.

[0070] In a preferred embodiment, the processing of the style parser specifically includes: performing block division and linear embedding processing on the image through the visual deformer ViT; calculating the query, key and value matrices through the self-attention mechanism, updating the input feature representation, and generating style features; stabilizing the training process and improving the model generalization ability by normalizing the output of the self-attention mechanism; and further extracting style embedding from the normalized features through the multi-layer perceptron MLP.

[0071] In a preferred embodiment, the training process of the style parser specifically includes: during training, using Gram loss by comparing the Gram matrices of the features of the generated image and the target image to ensure style alignment; maintaining the visual feature alignment of the generated image and the target image through perceptual loss to enhance model performance; the total style loss is a weighted sum of the original style loss, perceptual loss and Gram loss to achieve accurate expression of style and content; randomly discarding image conditions during training to achieve classifier-free guidance in inference and improve the generalization ability of the model.

[0072] In a preferred embodiment, the style parser includes the following style injection strategy: utilizing the hierarchical characteristics of convolutional neural networks, identifying and specifically using upsampling blocks to capture and inject style information, as distinguished from the content information captured by downsampling blocks; implementing a cross-attention mechanism, guided by image conditions, to inject style information into upsampling blocks, thereby enhancing style expression; through a targeted injection strategy, avoiding injecting style information into all network blocks, preventing confusion between style and content features, while ensuring the effective transmission and expression of style information.

[0073] Step S3. Build a content parser and extract content features: Build a content parser, process the content image hierarchically, generate multiple levels of latent representation as content embeddings, and integrate the content embeddings into the model through residual addition to stabilize the content architecture and ensure that the content features are preserved during the style transfer process.

[0074] In a preferred embodiment, the training process of the content parser specifically includes: keeping the parameters of the pre-trained diffusion model unchanged during the training phase; using image-text pairs for training to achieve effective correspondence between content and text descriptions; using mean square error loss to measure and minimize the difference between model prediction noise and actual noise to improve the quality of generated images; using adversarial loss to train the discriminator to enhance its ability to distinguish between real and generated images, thereby improving the authenticity of generated images; forming content loss by weighted combination of mean square error loss and adversarial loss to balance the contribution of the two losses to content retention quality.

[0075] In a preferred embodiment, the content embeddings derived from the content parser are injected into the downsampling block of the model, ensuring that low-level aspects of the content image, such as shapes and edges, are effectively captured and preserved.

[0076] Step S4. Construct content controller and inject content: Process the content image through the content controller to capture the basic structure and minimize the interference of style features.

[0077] In a preferred embodiment, the operations of the content controller specifically include: adopting a tiling control strategy to retain content information while not affecting the style transfer process; using ControlNet to manage spatial information within the U-Net architecture to effectively capture the basic structure of the content image and minimize the interference of style features; integrating content through a content fusion encoder to generate multiple levels of latent representations to form content embeddings; integrating content embeddings into the U-Net through residual addition to ensure the stability and integrity of content features during the style transfer process; and accurately injecting content embeddings into the upsampling blocks of the diffusion model through the ControlNet mechanism to achieve effective separation and conversion of content and style.

[0078] Step S5. Injecting style and content: injecting style embedding into the upsampling block of the model to enhance style expression, while injecting content embedding into the downsampling block of the model to ensure content preservation;

[0079] Step S6. Reasoning: In the reasoning stage, the constructed content parser and style parser are used to decouple style from content, allowing complex styles to be captured and reproduced while maintaining the integrity of the content.

[0080] In a preferred embodiment, in step S6, the inference includes an inversion operation for content preservation, specifically including: encoding the input image into a latent space during the sampling process, and reversing the sampling process through the ReNoise technique to generate a latent noise representation directly from the real image, thereby eliminating the need for additional noise injection; updating the noise using an iterative method, enhancing the approximation of the expected position in the forward diffusion progress through average prediction, improving the accuracy of content reconstruction, while minimizing the computational cost; improving the quality of image content reconstruction through an iterative process while maintaining the distribution of the noise latent representation.

[0081] In a preferred embodiment, during the inference process, the separation characteristics of the cross-attention mechanism are utilized to independently adjust the image condition weights of the content and style conditions during the inference stage; the potential representation of the final image is constructed by weightedly combining the attention results of the content and style conditions; the text condition weight factor is set to zero; when the weight factors of the content and style conditions are both zero, the model reverts to the original text-to-image diffusion model.

[0082] This paper proposes an innovative image style transfer method that achieves precise decoupling of style and content by constructing content and style parsers. This method uses a stable diffusion model as its foundational architecture and learns image content features through a denoising and denoising process. The style parser, based on the visual warper ViT, extracts style features and generates a style embedding. The content parser processes the content image hierarchically, generating a content embedding that is integrated into the model via residual addition. During the inference phase, the style embedding is injected into the upsampling block, and the content embedding is injected into the downsampling block, ensuring both style expression and content preservation. This method effectively overcomes the resource-intensive training issues of traditional diffusion models, avoids the entanglement of style and content during the transfer process, reduces the one-way bias between style and content, and improves image quality and style similarity. This method overcomes the dilemma of balancing accurate style expression with proper content preservation in traditional methods, thereby enhancing the decoupling of style and content. User studies have shown that this method is overwhelmingly favored in terms of visual experience, generating images that strike a delicate balance between quality, content, and style similarity, significantly improving the user visual experience.

[0083] The following further describes an algorithm example and experimental verification of a specific embodiment of the present invention.

[0084] Basic Model

[0085] Stable diffusion consists of two main stages: a noise addition stage, in which Gaussian noise ∈ is gradually introduced into the starting data x0 through a Markov chain; and a denoising stage.

[0086] In the denoising stage, the system uses Gaussian distribution N(0,1) to extract the noise x t Generate samples. This is done through a trainable denoising model ∈ θ (x t ,t,c), the model is parameterized by θ.

[0087] Denoising model ∈ θ (·) Based on the U-Net architecture, it is optimized using the mean squared error loss function. This loss function is derived from a simplified version of the variational bound and is expressed as:

[0088]

[0089] Where c represents an optional condition variable.

[0090] Style Parser

[0091] In the present invention, the style parser is very important in extracting and injecting style features. To this end, the present invention makes improvements in three aspects, including style representation, style injection block and training scheme.

[0092] Stylized Representation

[0093] The visual deformer ViT is used to extract style features. The image is divided into blocks and linearly embedded:

[0094] X={x1,x2,…,x N}.

[0095] These embeddings are then processed through the ViT layer. In each attention layer, the self-attention weights are calculated as follows:

[0096]

[0097] Where Q, K, V are the query, key, and value matrices computed as follows:

[0098] Q=XW Q ,K=XW K ,V=XW V .

[0099] Each ViT layer updates the input feature representation to produce the final style feature:

[0100] Z=Norm(X+Attn(Q,K,V)),

[0101] Norm represents the normalization operation, which is used to stabilize the training process and improve the generalization ability of the model.

[0102] The style embedding is extracted as the output of a specific layer:

[0103] E s =MLP(Norm(Z)),

[0104] Among them E s represents style embedding, MLP represents multi-layer perceptron, which is used to further extract style features from the normalized feature Z.

[0105] Training program

[0106] During the training process of the style parser, specific weights are assigned to different loss functions to balance their contributions to the total loss. The training objectives of the style parser include:

[0107]

[0108] To enhance model performance, a perceptual loss is introduced to maintain perceptual fidelity and align the visual features of generated and target images:

[0109] L p =|VGG(g)-VGG(t)| 2 ,

[0110] Where g and t represent the generated image and target image respectively.

[0111] We also use the Gram loss by comparing the Gram matrices of the features of the generated and target images to ensure style alignment:

[0112] L g =|Gram(g)-Gram(t)| 2 .

[0113] The style loss is a weighted sum of all components:

[0114] L style =α s ·L s +β s ·L p +γ s ·L g ,

[0115] The weight α s =0.2,β s =0.4, and γ s =0.4, respectively.

[0116] Additionally, image conditions are randomly dropped during training to enable classifier-free bootstrapping during inference:

[0117]

[0118] If the image condition is discarded, the image embedding will simply be zeroed out.

[0119] Style injection block

[0120] It is generally understood that in convolutional neural networks, lower convolutional layers learn low-level aspects such as shape and color, while deeper convolutional layers focus on semantic information.

[0121] Like the text condition, the image condition is generated by injecting guidance through the cross-attention layer. In addition, the inventors observed in the experiment that multiple blocks capture style and content information in different ways, such as Figure 8 shown.

[0122] Specifically, the downsampling block tends to capture content information, the upsampling block can capture style information such as color, material, and texture, while the middle block has no single preference.

[0123] If all blocks are injected, the result will be the generation of wrong content and style features, which is common in this type of methods.

[0124] Therefore, the style information is injected into the upsampling block, as Figure 2 shown.

[0125] This targeted injection reduces the need for extensive parameter tuning and enhances stylistic expression.

[0126] Content Parser

[0127] Previous methods usually directly process content image embeddings for style transfer, but this may retain inherent style information and affect the performance of complex tasks. Therefore, a content parser is constructed to extract specific structural information for control.

[0128] Content Controller

[0129] Inspired by direct processing methods, tiling control is adopted to ensure that content information is preserved without affecting style. The model processes the content image, effectively capturing its basic structure while minimizing style features. Leveraging ControlNet's ability to manage spatial information within the U-Net architecture, the content fusion encoder integrates content and produces multiple levels of latent representations for content embedding. c : in Involving intermediate sample blocks, arrive Contains downsampling blocks, L represents the total number of layers. The content is embedded into f by residual addition c Integration into U-Net:

[0130]

[0131] f 0 represents the latent features of the middle block of the content controller U-Net, and f 1 to f L represents the representation in the upsampling block. Figure 2 As shown in Figure 3, the final injection method follows the standard ControlNet mechanism and is injected into the upsampling block of the diffusion model.

[0132] Training program

[0133] During the training phase, we only focus on optimizing the parser without changing the parameters of the pre-trained diffusion model. The training process uses image-text pairs and integrates the following losses.

[0134] In order to measure the difference between the predicted noise and the actual noise, the mean square error loss is introduced, and the formula is as follows:

[0135]

[0136] where p i is the model’s prediction of the noise, a i is the actual noise and n is the number of samples. This loss helps the model predict the noise more accurately, thereby improving the quality of the generated images.

[0137] To improve the realism of generated images, an adversarial loss is used as follows:

[0138] L a =-[D(g)log(D(g))+(1-D(g))log(1-D(g))],

[0139] where D is the probability that the discriminator predicts that the generated image g is real. The adversarial loss trains the discriminator to distinguish between real and generated images, thereby improving the quality of content preservation.

[0140] The content loss is a weighted combination of:

[0141] L content =β c ·L mse +γ c ·L a ,

[0142] where β c =0.6 and γ c= 0.4 are the weights of the mean squared error loss and the adversarial loss, respectively. These weights are chosen to balance the contribution of each loss to the content loss.

[0143] Content injection block

[0144] Experiments and observations show that the downsampling block is mainly useful for capturing and preserving low-level aspects of the content image (such as shapes and edges, see Figure 7 ) is very important.

[0145] They contribute less to style features but are crucial for content preservation. Therefore, the content embeddings derived from the content parser are injected into the downsampling block (see Figure 2 ).

[0146] Enhancement Strategy

[0147] Reversal operation for content preservation

[0148] During the sampling process, the input image is typically encoded into a latent space and noise is added. However, the presence of additional noise may cause content drift, thereby deviating from the intended target. ReNoise proposes to reverse the sampling process and generate a latent noise representation directly from the real image, eliminating the need for additional noise injection. Although InstantStyle proposes that image inversion may ignore subtle style differences in the image, this limitation does not hinder the content protection task of the present invention. The embodiment of the present invention does not rely solely on a single inversion process, but focuses on average prediction to enhance the approximation to the expected position in the forward diffusion progress, allowing the model to improve the accuracy of content reconstruction while minimizing computational costs. The basic formula for ReNoise inversion is as follows:

[0149]

[0150] This formula updates the noise ∈ t . represents the gradient relative to the noise, ρ t is a scaling factor, z t-1 is the potential representation of the previous step, φ t and ψ t are model parameters.

[0151] This iterative process helps improve the quality of image content reconstruction while preserving the distribution of the noisy latent representation.

[0152] reasoning

[0153] Since the cross-attention is decoupled, the image-conditional weights for content and style conditions can be adjusted independently during inference:

[0154] Z'=λ tAttn(Q,K,V)+λ c ·Attn(Q,K c ,V c )+λ s ·Attn(Q,K s ,V s ),

[0155] where λ c and λ s are the weight factors for content and style conditions, respectively. Z’ is the latent representation that combines the construction condition information to produce the final image.

[0156] Text condition weight factor λ t is set to 0.

[0157] On the contrary, if λ c =0 and λ s = 0, the model will revert to the original text-to-image diffusion model.

[0158] Dataset

[0159] Previous stylization methods often use the Laion-Aesthetics dataset to train style encoders. Laion-Aesthetics contains 92.3% natural images, and this dataset, with its high aesthetic scores, is particularly suitable for training content parsers. Therefore, we constructed a 100k art description dataset, "Palette." The 100,000 artworks in the "Palette" are primarily selected from WikiArt and LaionArt. Blip-2 is used to query the stylistic details of each artwork, thereby obtaining detailed textual descriptions of the artwork.

[0160] In this embodiment of the present invention, a style parser is designed, using a visual transformer to extract style features. A multi-step process generates a style embedding. During training, a combination of mean squared error loss, perceptual loss, and Gram loss is weighted and summed, while randomly discarding image conditions to achieve the effect of eliminating the need for classifier guidance. Given the differences in how different U-Net blocks capture style and content, precisely selected blocks are injected with style information, enhancing style expressiveness while avoiding tedious parameter tuning.

[0161] The content controller generates content embeddings through multi-layer integration and injects them into the next block of the U-Net using residual addition to stabilize the content structure. During training, the model is optimized using a weighted sum of mean squared error and adversarial loss. The Palette dataset, containing 100,000 artworks and textual interpretations, was constructed. Its high proportion of natural images and high-quality aesthetic ratings facilitate content parser training.

[0162] An enhancement strategy is proposed, introducing an inversion procedure during sampling and iteratively updating the noise according to the denoising formula, improving content reconstruction accuracy and stabilizing noise distribution characteristics. The Stable Diffusion Model XL version is used as the base model, and a pre-trained visual transformer is used as the image encoder. Image resolution, learning rate, and other training parameters are standardized. During the inference phase, the number of denoising steps and memory usage are flexibly adjusted according to actual needs, improving processing efficiency.

[0163] Other embodiments

[0164] The loss function weights in the style parser can be fine-tuned, such as moderately increasing the perceptual loss weight and slightly decreasing the Gram loss weight. This allows the generated images to retain the style characteristics while being closer to the user's desired style strength. Although the overall effect is slightly inferior to the optimal solution, the style control is more flexible and resource consumption is slightly increased.

[0165] In scenarios with specific styles or low content complexity, some adjacent U-Net blocks can share style or content injection points to simplify the model architecture and accelerate calculations. Although the accuracy of style-content decoupling is slightly reduced, the processing efficiency is improved, which is still better than traditional methods and can meet the needs of efficiency-sensitive scenarios.

[0166] The present invention has the following remarkable effects:

[0167] Precise decoupling of style and content: Style and content parsers are specifically constructed based on their essential differences and precisely integrated into different blocks of the U-Net, breaking the entanglement between style and content. This allows for dynamic and diverse style transfer without compromising the content. For example, in a qualitative comparison, compared to methods such as CS:GO, this method accurately transfers style while fully preserving the structure and details of content such as buildings and characters.

[0168] Comprehensively improve image quality: The style parser is trained collaboratively with multiple losses, the content parser is finely optimized with multiple losses, and the dataset is precisely adapted based on its characteristics, improving image quality and style similarity from multiple dimensions.

[0169] Deeply optimized user visual experience.

[0170] Experimental results

[0171] The quantitative evaluation results are shown in Table 1. Compared with the cutting-edge methods, the proposed method stands out in CSD and CAS indicators, demonstrating its excellent effectiveness in style and content control.

[0172] Table 1

[0173]

[0174]

[0175] As shown in Table 2, user studies show that our method leads the way in both style and content scoring and overall quality, achieving a delicate balance between quality, content similarity, and style similarity. When users filter based on visual appeal and content relevance, our method overwhelmingly favors it, demonstrating its profound improvement in the user visual experience.

[0176] Table 2

[0177]

[0178] In summary, the present invention proposes an image style transfer method based on a content and style parser. Compared with traditional technologies, the important innovative contributions of the embodiments of the present invention include:

[0179] The style parser uses a visual transformer to accurately extract style features. Its unique self-attention mechanism captures style associations between distant elements in an image, outperforming the local perception of traditional convolutional layers. When outputting style embeddings from specific layers, multiple loss functions are trained collaboratively, with varying weight balances. Random image conditional dropout enables flexible style control during inference, laying the foundation for precise style expression.

[0180] The content analyzer leverages its expertise in spatial information management to design a content controller that processes content images in layers. It embeds the fused content from multiple layers and injects the residuals into the U-Net downsampling block, leveraging the block's sensitivity to fundamental content features such as image shape and contour to stabilize the content. A weighted optimization approach is employed using both mean squared error and adversarial loss. The former corrects for prediction noise and improves generation quality, while the latter leverages a discriminator-based game to ensure content authenticity and legibility. These two approaches balance content distortion and style contamination to ensure content integrity.

[0181] Application scenarios of the present invention include but are not limited to:

[0182] In the field of video style transfer, image style transfer technology is extended to video streams. Video images are processed frame by frame, and the style is unified based on the video theme or user instructions. For example, in movie special effects production, realistic scenes can be transformed into oil paintings or ink paintings with a single click, or retro or sci-fi styles can be switched to suit the plot atmosphere. Leveraging the style-content management advantages of this invention, video content consistency and natural character movements are maintained, preventing jitter or content distortion caused by style transfer, thereby improving creative efficiency and artistic effect.

[0183] Virtual Reality (VR) / Augmented Reality (AR) Scene Construction: This invention enables style diversification and content adaptation when generating virtual objects and scenes in VR / AR environments. The style of virtual buildings and characters can be customized based on user immersion needs and interactive plots. For example, in historical VR experiences, this approach ensures that the style of virtual historical sites is relevant to the era, with authentic texture details. Furthermore, the rendering style can be smoothly switched based on the user's perspective. Through precise style conversion and content maintenance, user immersion and interactive realism are enhanced.

[0184] Digital art creation and design tools: Integration into drawing and design software empowers artists and designers. For example, illustrations can be quickly switched between styles to explore creative ideas, while poster design can intelligently match image styles to brand styles. Through efficient style transfer and intelligent content preservation, this approach stimulates creative inspiration, shortens design cycles, improves the quality of work, and revolutionizes the digital art creation process and efficiency.

[0185] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0186] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0187] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0188] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0189] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0190] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0191] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0192] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. Various media that can store program codes.

[0193] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0194] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0195] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0196] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0197] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for image style transfer based on content and style parser, characterized in that: The following steps are involved: S1. Building a Basic Model: Using the stable diffusion model as the foundational architecture, we learn and encode image content features through a denoising and de-noising process. S2. Build a style parser and extract style features: Use the Visual Transformer (ViT) as a style parser to extract style features from the reference image and generate style embeddings using a multi-layer perceptron. S3. Build a content parser and extract content features: Build a content parser that processes content images hierarchically to generate multiple levels of latent representations as content embeddings. This content embeddings are then integrated into the model via residual addition to stabilize the content architecture and preserve content features during style transfer. S4. Build a content controller and inject content: Process the content image through the content controller to capture the basic structure and minimize the interference of style features; The operations of the content controller specifically include: A tiling control strategy is adopted to preserve content information without affecting the style transfer process; Use ControlNet to manage spatial information within the U-Net architecture, capturing the basic structure of the content image and minimizing the interference of style features; The content fusion encoder integrates the content to generate multiple levels of latent representations to form content embedding; Integrate content embedding into U-Net via residual addition; Through the ControlNet mechanism, content embedding is accurately injected into the upsampling block of the diffusion model to achieve the separation and conversion of content and style; S5. Injecting style and content: Injecting style embeddings into the upsampling block of the model to enhance style expression, while injecting content embeddings into the downsampling block of the model to ensure content preservation; S6. Reasoning: During the reasoning phase, we leverage the built content and style parsers to decouple style from content, capturing and reproducing complex styles while maintaining the integrity of the content. In step S6, the inference includes a reversal operation for content preservation, specifically including: During the sampling process, the input image is encoded into a latent space, and the ReNoise technique reverses the sampling process to generate a latent noise representation directly from the real image, thus eliminating the need for additional noise injection; An iterative approach is used to update the noise, enhancing the approximation of the expected position in the forward diffusion progression by averaging the predictions.

2. The image style conversion method based on content and style parser according to claim 1, characterized in that: The processing of the style analyzer specifically includes: The image is divided into blocks and linearly embedded using the visual deformer ViT; Calculate query, key, and value matrices through the self-attention mechanism, update the input feature representation, and generate style features; By normalizing the output of the self-attention mechanism, the training process is stabilized and the model's generalization ability is improved; Style embedding is further extracted from the normalized features through a multi-layer perceptron (MLP).

3. The image style conversion method based on content and style parser according to claim 1 or 2, characterized in that: The training process of the style analyzer specifically includes: During training, we use the Gram loss to ensure style alignment by comparing the Gram matrices of features of the generated and target images. Keep the visual features of the generated image and the target image aligned through perceptual loss; The original style loss, perceptual loss and Gram loss are weighted summed to obtain the total style loss; Randomly drop image conditions during training, enabling classifier-free bootstrapping during inference.

4. The image style conversion method based on content and style parser according to any one of claims 1 to 2, characterized in that: The style parser includes the following style injection strategies: Leveraging the hierarchical nature of convolutional neural networks, we identify and specifically use upsampling blocks to capture and inject style information. A cross-attention mechanism is implemented to inject style information into the upsampling block guided by image conditions, thereby enhancing style expression.

5. The image style conversion method based on content and style parser according to any one of claims 1 to 2, characterized in that: The training process of the content analyzer specifically includes: Keep the parameters of the pre-trained diffusion model unchanged during the training phase; Use image-text pairs for training to achieve effective correspondence between content and text descriptions; Use mean squared error loss to measure and minimize the difference between the model's predicted noise and the actual noise; Use adversarial loss to train the discriminator to enhance its ability to distinguish between real and generated images; The content loss is formed by weighted combination of mean squared error loss and adversarial loss to balance the contribution of the two losses to content preservation quality.

6. The image style conversion method based on content and style parser according to any one of claims 1 to 2, characterized in that: The content embeddings derived from the content parser are injected into the downsampling block of the model.

7. The image style conversion method based on content and style parser according to any one of claims 1 to 2, characterized in that: Step S6 specifically includes: Leveraging the dissociative nature of the crisscross attention mechanism, we independently adjust the image conditional weights for content and style conditions during inference. Construct a latent representation of the final image by weightedly combining the attention results of content and style conditions; Set the text condition weight factor to zero; When the weight factors of both content and style conditions are zero, the model reverts to the original text-to-image diffusion model.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image style conversion method based on the content and style parser according to any one of claims 1 to 7 is implemented.

Images